Search

When a video sequence is compressed by the MPEG encoder with a low coding rate, various artifacts such as blocking artifacts are produced. To remove the artifacts, this paper proposes a video restoration approach with the 3-D redundant DCT, which utilizes spatiotemporal correlations of a moving image sequence more effectually than the 3-D orthogonal DCT. The approach recovers an image sequence by applying the shrinkage operation to 3-D DCT coefficients of the recovered sequence with the imposition of the MPEG-quantization constraint, an expansion of the JPEG-quantization constraint. Experimental simulations demonstrate that the approach successfully removes the compression artifacts selectively and improves subjective quality considerably.

An image taken under extremely low illumination is modeled as obeying the Poissonian-Gaussian distribution. This paper proposes a method to recover a higher-quality color moving-image sequence from Poissonian-Gaussian observations. The method performs virtual multiplex imaging, formed as a series of pixel binning and redundant subsampling, on the input sequence to increase its statistical reliability, denoises the multiple subsampled image sequences given by the redundant subsampling, and removes image blurs due to the pixel binning by super-resolution deburring to integrate the multiple denoised image sequences into an image sequence. Computer simulations show that the method successfully recovers a relatively-high-quality image sequence.

As a basic tool for deriving sparse representation of a color image from its atomic-decomposition with a redundant dictionary, the authors have recently proposed a new kind of shrinkage technique, viz. color shrinkage, which utilizes inter-channel color dependence directly in the three primary color space. Among various schemes of color shrinkage, this paper particularly presents the soft color-shrinkage and the hard color-shrinkage, natural extensions of the classic soft-shrinkage and the classic hard-shrinkage respectively, and shows their advantages over the existing shrinkage approaches where the classic shrinkage techniques are applied after a color transformation such as the opponent color transformation. Moreover, this paper presents the applications of our color-shrinkage schemes to color-image processing in the redundant tight-frame transform domain, and shows their superiority over the existing shrinkage approaches.

It is a very important and intriguing problem in digital image-processing to decompose an input image into intuitively convincible image-components such as a structure component and a texture component, which is an inherently nonlinear problem. Recently, several numerical schemes to solve the nonlinear image-decomposition problem have been proposed. The use of the nonlinear image-decomposition as a pre-process of several image-processing tasks will possibly pave the way to solve difficult problems posed by the classic approach of digital image-processing. Since the new image-processing approach via the nonlinear image-decomposition treats each separated component with a processing method suitable to it, the approach will successfully attain target items seemingly contrary to each other, for instance invisibility of ringing artifacts and sharpness of edges and textures, which have not attained simultaneously by the classic image-processing approach. This paper reviews quite recently developed state-of-the-art schemes of the nonlinear image-decomposition, and introduces some examples of the decomposition-and-processing approach.

We used optical low-pass filters with solid-state imaging devices to produce images. The actual optical low-pass filter was formed by combining two types of doubly refractive crystal devices. The filters cannot sharply cut-off the high frequency components and they reduce the frequency components near the Nyquist frequency. Thus, the projected images on the surface are blurred. This paper presents a demosaicking method that restores the blur caused by optical low-pass filters. Our sharpening approach to demosaicking employs a Landweber-type iterative algorithm. For our sharpening-demosaicking approach, the Bayer's pattern of the RGB primary color filter array is not applicable. Therefore, we studied another color-filter pattern, namely the WRB filter array, which is preferable to the RGB filter array, where W-filtering means that all the visible light passes through it. Our mathematical formulations for sharpening-demosaicking are similar to the least square problem, but there are many different least square solutions. To avoid ambiguity in the spatial frequency domain, we introduced a pass-band limitation that corresponds with the sub-sampling pattern of the mosaicking of color filters into the iterative algorithm.

We previously presented a method of selectively sharpening monochrome images. The algorithm in our method is based on the coupled nonlinear diffusion process composed of a nonlinear diffusion term, a fidelity term, and an overshooting term, and can sharpen blurred edges without increasing noise visibility. This paper extends our algorithm to selectively sharpening color images. We take into accounts two schemes to treat the three primary color components in finding out how to extend it. Through quantitative experiments, we evaluate selective sharpening performance achievable with the two schemes. Of the two schemes, which are based on the coupled nonlinear diffusion process, collectively treating the three primary color components achieves better performance than treating them independently, yet both two schemes can sharpen blurred color edges more selectively than the existing adaptive peaking method, which is a typical means of enhancing sharpness.

反復変換理論に基づく濃淡画像の符号化手法が数多く提案されている.その中で, A. E. Jacquinが提案した静止画像符号化では, 画像を異なる大きさのブロックに空間的に分割することで効率的な符号化を達成している.この概念を, 動画像符号化における時間方向の効率的な分割へと拡張した.

When an object in a scene is moving slowly and/or the camera is being moved, the observed image sequence contains rich information about both the moving object and the scene. In this case, it seems that we can enhance spatial resolution within a region of interest in the observed image frame by integrating pixels over multiple observed image frames. The interframe-integration task involves a method for estimating interframe pixel correspondence with real pixel accuracy. The principal existing block-matching algorithm can provide highly accurate correspondence estimates only in the case in which a rigid object undergoes translation in a direction parallel to the image plane, it cannot cope with more general types of motion such as rotation. Standard gradient-based algorithms cannot stably provide accurate correspondence estimates for real moving images. In this paper, we present a method for estimating interframe pixel correspondence with real pixel accuracy. The system uses iteratively warping quadrilateral patches covering the region of interest, and is an interframe integration method. Experimental simulations conducted on real moving image sequences demonstrate that the proposed method enhances spatial resolution considerably when a region of interest is limited to one planar surface of an object.

Although block-matching algorithms are often used to estimate local motion from moving images, they do not work well the analysis block contains multiple image regions moving in different directions. To solve this problem, we extend the basic concept of the existing block-matching algorithm. The proposed algorithm segments a given analysis block into multiple regions and gives coherent motion for each region. We formulate this segmentation and motion estimation problem into an optimum combination equation, and construct the algorithm. Through computer simulation, we demonstrate that extending the block-matching algorithm make it possible to estimate motions more accurately than is possible with existing block-matching algorithms.

We previously developed a signal-processing-based high-resulution image acquisition scheme using two cameras. This approach produces a high-resolution image by integrating two image taken with two cameras. The integration process has two stages : registration and reconstruction. When the two cameras have the same pixel aperture, however, obtaining spatially uniform resolution improvement requires severe limitations both in the arrangement of the cameras and in the geometrical configuration of the scene.To overcome this disadvantage, this paper introduces two methods : one for reducing noise and the other for improving resulution. The noise reduction method diminishes noise instead of improving resolution by the integration ; the resolution improvement method uses multiple cameras with different pixel apertures. Both methods always provide spatially uniform improvements. We also present an integration algorithm that iteratively performs the registration stage and the reconstruction stage alternately. Furthermore, we introduce a brightness retouch operation that is indispensable in using the alternately iterative algorithm.

We propose a new sparse-representation approach to super-resolution color-image interpolation. Whether an image to be interpolated actually undergoes blurs or not, the approach assumes that the image is blurred with a certain smoothing filter, and it first performs both deblurring and interpolation simultaneously so as to produce a provisional high-resolution image on the quasi blur-hypothesis, and then it blurs the provisional high-resolution image with the hypothesized blurring filter to produce a finally interpolated color-image. The quasi blur-hypothesis is introduced to enhance the spatially-regularization efficacy of the data-fidelity constraint in the variational problem of the super-resolution deblurring-and-interpolation. If a proper smoothing filter is chosen as the hypothesized blurring filter, the quasi blur-hypothesis approach can achieve state-of-the-art interpolation performance.

This paper extends the total-variation (TV) semi-norm of a scalar-valued image to a new color TV semi-norm of a color image, in the definition of which interdependence among the three primary color channels is taken into account. This paper defines a 12-D color-gradient operator that treats primary color signals and color-difference signals collectively; and then with the color-gradient operator this paper defines three kinds of color-TV semi-norms: isotropic, semi-isotropic, and anisotropic color TV semi-norms. By introducing dual definitions of the color-TV semi-norms, this paper reformulates color-TV variational denoising problems, and solves their dual problems with the projected gradient solver. Experimental simulations of color-image denoising demonstrate that the color TV semi-norms achieve selective noise-removal much more efficiently than the classic TV semi-norm, and show the merits and demerits among the three color TV semi-norms.

Towards the development of a very high definition (VHD) image acquisition system, previously we developed the signal processing based approach with multiple cameras. The approach produces an improved resolution image with sufficiently high signal-to-noise ratio by processing and integrating multiple images taken simultaneously with multiple cameras. Originally, in this approach, we used multiple cameras with the same pixel aperture, but in this case there are severe limitations both in the arrangement of multiple cameras and in the configuration of the scene, in order to guarantee the spatial uniformity of the resultant resolution. To overcome this difficulty completely, this work presents the utilization of multiple cameras with different pixel apertures, and develops a new, alternately iterative signal processing algorithm available in the different aperture case. Experimental simulations clearly show that the utilization of multiple different-aperture cameras prospects to be good and that the alternately iterative algorithm behaves satisfactorily.

To remove signal-dependent noise of a digital color camera, we propose a new denoising method with our hard color-shrinkage in the tight-frame grouplet transform domain. The classic hard-shrinkage works well for monochrome-image denoising. To utilize inter-channel color dependence, a noisy image undergoes the color transformation from the RGB to the luminance-and-chrominance color space, and the luminance and the chrominance components are separately denoised; but this approach cannot cope with actual signal-dependent noise. To utilize the noise's signal-dependencies, we have constructed the hard color-shrinkage where the interchannel color dependence is directly utilized in the RGB color space. The hard color-shrinkage alleviates denoising artifacts, and improves picture quality of denoised images.

Alter, Durand and Froment introduced the total-variation (TV) minimization approach to the artifact-free JPEG decoding, which is referred to as the ADF decoding method [1]. They formulated the decoding problem as the constrained TV restoration problem, in which the TV seminorm of its restored color image is minimized under the constraint that each DCT coefficient of the restored color image should be in the quantization interval of its corresponding DCT coefficient of the JPEG-compressed data. This paper proposes a new restoration approach to the JPEG decoding. Instead of the TV regularization, our new JPEG-decoding method employs a shrinkage operation in the redundant DCT domain, to mitigate degradations caused by the JPEG coding. Our new method not only can selectively suppress ringing artifacts near color edges, but also can efficiently eliminate blocking artifacts in originally smoothly-varying image regions, where the blocking artifacts are very noticeable. Through decoding simulations, we experimentally show that our new decoding method can reduce JPEG-coding artifacts more effectively than the ADF decoding method.

Recently, R. H. Chan, et al. proposed an image restoration method with the shift-invariant Haar wavelet transform. The method is suitable for image restoration in a digital color camera, but in a heavily noisy case produces artifacts originating from noise. In such a case, the restoration process should be split into three stages: color-interpolation, denoising, and deblurring; and along this line we present an approach to restore a high ISO-sensitivity image. Our approach firstly demosaics observed color data with the bilinear interpolation, which is robust against noise but causes blurs. Next, it applies our previously proposed color-shrinkage to each wavelet coefficient or each grouplet coefficient of the demosaicked image, to produce a denoised one. Finally, it applies to the denoised image a color-image restoration method corresponding to an extension of the method of Chan et al., and employs our previously proposed soft color-shrinkage. Our approach restores a high-quality image even in an ISO-6400 case.

Recently, R. H. Chan, T. F. Chan, L. Shen, and Z. Shen proposed an image-restoration method, referred to as the C 2 -S 2 method, in the shift-invariant Haar wavelet transform domain. The C 2 -S 2 method is suitable for image-restoration in a digital color camera and restores a sharp color image in a lightly noisy case; but in a heavily noisy case it produces colored artifacts originating from noise. In such a case, the image-restoration process should be split into a colorinterpolation stage, a denoising stage, and a deblurring stage; and along this line we present an approach to restore a high ISO-sensitivity color image. Our approach firstly de-mosaics observed color data with the bi-linear interpolation method, which is robust against observation noise but causes image blurs. Next, our approach applies our previouslyproposed spatially-adaptive soft color-shrinkage to each shift-invariant Haar wavelet coefficient of the demosaicked color-image, to produce a denoised color-image. Finally, our approach applies to the denoised color-image a colorimage restoration method that corresponds to an extension of the C 2 -S 2 method and employs our previously proposed soft color-shrinkage. The experimental simulations conducted on raw color data observed with a SLR digital color camera with 6400 ISO-sensitivity demonstrate that our approach restores a high-quality color image even in a high ISOsensitivity case, without producing noticeable artifacts.

The classic shrinkage works well for monochrome-image denoising. To utilize inter-channel color correlations, a noisy image undergoes the color-transformation from the RGB to the luminance-and-chrominance color space, and the luminance and the chrominance components are separately denoised. However, this approach cannot cope with signaldependent noise of a digital color camera. To utilize the noise's signal-dependencies, previously we have proposed the soft color-shrinkage where the inter-channel color correlations are directly utilized in the RGB color space. The soft color-shrinkage works well; but involves a large amount of computations. To alleviate the drawback, taking up the l0-l2 optimization problem whose solution yields the hard shrinkage, we introduce the l0 norms of color differences and the l0 norms of color sums into the model, and derive hard color-shrinkage as its solution. For each triplet of three primary colors, the hard color-shrinkage has 24 feasible solutions, and from among them selects the optimal feasible solution giving the minimal energy. We propose a method to control its shrinkage parameters spatially-adaptively according to both the local image statistics and the noise's signal-dependencies, and apply the spatially-adaptive hard color-shrinkage to removal of signal-dependent noise in a shift-invariant wavelet transform domain. The hard color-shrinkage performs mostly better than the soft color-shrinkage, from objective and subjective viewpoints.

A monochrome-image iterative deblurring method with the classical soft-shrinkage scheme in the shift-invariant Haar wavelet transform domain was recently proposed by R. H. Chan et al [1]. Extending the method, we present a new iterative sharpening-demosaicing method with the shift-invariant Haar wavelet transform and our previously proposed soft color-shrinkage [5] utilizing inter-channel color cross-correlations. Our new iterative method performs demosaicing and sharpening simultaneously in the wavelet transform domain. Simulation results demonstrate that our new method works as efficiently as our previously proposed sharpening-demosaicing method with the TV regularization in the spatial image-domain [2].

This paper extends the BV-L1 variational nonlinear image-decomposition approach, useful for image processing, to a genuine color-image decomposition approach. For utilizing inter-channel color cross-correlations, we introduce TV norms of color differences and TV norms of color sums into the BV-L1 energy functionals to be minimized, and then derive denoising-type decomposition-algorithm with an over-complete wavelet transform, by applying the Besov-norm approximation to the variational problem. Our method decomposes a noisy color image without producing undesirable low-frequency colored artifacts in its separated BV-component, and achieves desirable high-quality color-image decomposition, robust against colored random noise. We apply our decomposition method to color-image denoising.

This paper proposes a new digital-camera image-processing (IP) pipeline, in which observed raw color data is firstly demosaicked, and the demosaicked color image is decomposed into its two components, namely a structure component and a texture component, with our previously proposed BV-G color-image decomposition method, and then the two components are separately processed, and finally the two processed components are combined to synthesize an output color image. The BV-G color-image decomposition method utilizes inter-channel color cross-correlations, which alleviate bad effects of observation noise on quality of an output color image. Furthermore, this paper introduces a technique into the IP-pipeline, to adjust a trade-off between noise visibility and texture sharpness to user's taste.

To remove signal-dependent noise of a digital color camera, we present a new soft color-shrinkage scheme for color-image denoising in a wavelet transform domain. The classic soft-shrinkage scheme works well for monochrome-image denoising; to utilize inter-channel color cross-correlations, a noisy image undergoes the color-transformation from the RGB to the luminance-and-chrominance color space, and the luminance and the chrominance components are separately denoised. However, this color-denoising approach cannot cope with actual signal-dependent noise. To utilize the noise's signal-dependencies, we present an iterative soft color-shrinkage scheme where the inter-channel color cross-correlations are directly utilized in the RGB color space, and theoretically study its convergence property. Our color-shrinkage scheme alleviates denoising artifacts, and improves picture quality of denoised images.

This paper presents two new architectures of an image-processing (IP) pipeline for a digital color camera, which are referred to as a prior-decomposition approach and a posterior-decomposition approach. The prior-decomposition approach firstly decomposes each primary color channel of mosaicked raw color data as a product of its two components, a structure component and a texture component, with a proper multiplicative monochrome-image decomposition method such as the BV-G variational decomposition method. Each component is then demosaicked with its proper color-interpolation method. Next, white balancing, color enhancement and inverse gamma correction are applied to only the structure component. Finally, the two components are combined. On the other hand, the posteriordecomposition approach firstly interpolates mosaicked raw color data with a proper demosaicing method, and decomposes its demosaicked color image with our proposed multiplicative color-image decomposition method utilizing inter-channel color cross-correlations. The subsequent processing is performed in the same manner as in the priordecomposition approach. Our proposed two architectures produce a high-quality output full-color image, but somewhat differ in performance. We experimentally compare their performance, and discuss the merits and the demerits of them.

This paper extends the BV (Bounded Variation) - G and/or the BV-L1 variational nonlinear image-decomposition approaches, which are considered to be useful for image processing of a digital color camera, to genuine color-image decomposition approaches. For utilizing inter-channel color cross-correlations, this paper first introduces TV (Total Variation) norms of color differences and TV norms of color sums into the BV-G and/or BV-L1 energy functionals, and then derives denoising-type decomposition-algorithms with an over-complete wavelet transform, through applying the Besov-norm approximation to the variational problems. Our methods decompose a noisy color image without producing undesirable low-frequency colored artifacts in its separated BV-component, and they achieve desirable high-quality color-image decomposition, which is very robust against colored random noise.

The method of image aquisition is one of the principal problems for handling super high definition pictures. This paper proposes a new method for acquiring high resolution pictures by processing two images taken with two different cameras. The method integrates two images into a higher resolution image. Compared to an imaging device with equivalent resolution, it will be less sensitive to the shot noise which becomes more dominant at the pixel size is reduced. Our theoretical analysis clearly shows the improvement in the high frequencies and details. The aperture effect determines the upper bound in the performance using this method. When the aperture ratio is 100%, the resolution improvement is limited to twice the original sampling frequency. Additionally, using more than two cameras, SNR of the reconstructed image improves even in the case of 100% aperture ratio. This means that the pixel size can be further reduced when extra cameras are used.

In a digital camera, several factors cause signal-dependency of additive noise. Many denoising methods have been proposed, but many of them do not necessarily work well for actual signal-dependent noise. To solve the problem of removing signal-dependent noise of a digital color camera, this paper presents a denoising approach via nonlinear image-decomposition. As a pre-process, we employ the BV- L 1 nonlinear image-decomposition variational model. This variational model decomposes an input image into three components: a structural component corresponding to a cartoon image approximation collecting geometrical image features, a texture component corresponding to fine image textures, and a residual component. Each separated component is denoised with a denoising method suitable to it. For an image taken with a digital color camera under the condition of high ISO sensitivity, the BV- L 1 model removes its signal-dependent noise to a large extent from its separated structural component, in which geometrical image features are well preserved, but the structural component sometimes suffers color-smear artifacts. To remove those color-smear artifacts, we apply the sparse 3D transform-domain collaborative filtering to the separated structural component. On the other hand, the texture component and the residual component are rather contaminated with noise, and the effects of noise are selectively removed from them with our proposed color shrinkage denoising schemes utilizing inter-channel color crosscorrelations. Our method achieves efficient denoising and selectively removes signal-dependent noise of a digital color camera.

Previously we presented a TV-based super-resolution sharpening-demosaicing method. Our previous method makes it possible to restore frequency components higher than the Nyquist frequency, and to interpolate color signals effectively while preserving their sharp color edges, without producing ringing artifacts along the edges. However, since our previous method applies the TV regularization separately to each primary color channel, as side effects it sometimes produces false color artifacts and/or zipper artifacts along sharp color edges. To remedy the drawback, in addition to the TV regularization of each primary color signal, we introduce the TV regularization of color difference signals such as G-R, and that of color sum signals such as G+R, into the TV-based super-resolution sharpening-demosaicing method. Near sharp color edges, correct interpolation provides the smallest TV norms of color difference signals or the smallest TV norms of color sum signals. Unlike our previous method, our new method jointly interpolates the three primary color channels. We compare demosaicing performance of our new method with the state-of-the-art demosaicing methods. In the evaluations, we consider a noise-free case and a noisy case. For both cases our new method achieves the best performance, and for the noisy case our new method considerably outperforms the state-of-the-art methods.

This paper presents a new image-interpolation approach where one can adjust edge sharpness and texture intensity according to one’s taste. This approach is composed of the three stages. At the first stage, with the BV-G image-decomposition variational model, an image is represented as a product of its two components so that its separated structural component may correspond to a cartoon image-approximation and its separated texture components may collect almost all oscillatory variations representing textures, and the texture co mponent can be amplified or attenuated according to user’s taste. At the second stage, each separated component is interpolated with an interpolation method suitable to it. Since the structural component keeps sharp edges, its proper interpolation method is a TV-regularization super-resolution interpolation method that can restore frequency components higher than the Nyquist frequency and remove sample-hold blurs without producing ringing artifacts near edges. The texture component is an oscillatory function, and its proper interpolation method is a smoothne ss-regularization super-resolution interpolation method that can restore continuous variations and remove the blurs. At the final stage, the two interpolated components are combined. The approach enlarges images without not only blurring edges but also destroying textures, and removes blurs caused by the sample-hold and/or the optical low-pass filter without producing ringing artifacts. Keywords: Image decomposition, variational problem, interpolation, deblurring, super-resolution

Previously we presented a TV (total variation)-based sharpening-demosaicing method. Our previous method interpolates color signals effectively while preserving sharp color edges, without producing ringing artifacts along the edges. However, since our previous method applies the TV regularization separately to each primary color channel, it sometimes produces color artifacts along sharp color edges. To remedy the drawback, we introduce the TV regularization of color differences and color sums into our previous method. Unlike our previous method, our new method jointly interpolates the three primary color channels. We compare demosaicing performance of our new method with the state-of-the- art demosaicing methods. In the evaluations, we consider a noise- free case and a noisy case. For both cases our new method achieves the best performance, and for the noisy case our new method considerably outperforms the state-of-the-art methods.

This paper presents a new image-processing (IP) pipeline adopting an image-processing approach via nonlinear image-decomposition, for a digital color camera. This new IP pipeline is composed of four stages. At the first stage, with the multiplicative BV-G image-decomposition method, each primary color channel of observed raw color data mosaicked with the Bayer color filter array is decomposed as a product of two components. Its structural component corresponds to a cartoon signal-approximation and its texture component collects almost all oscillatory variations representing textures. At the second stage, each separated component is demosaicked with an interpolation method suitable to it. For demosaicing of the structural component, this paper employs our color-TV-regularization super-resolution deblurring-demosaicing method that can remove image blurs without producing ringing artifacts near edges. For interpolation of the texture component gathering most of observation noise, this paper employs a basic averaging-type demosaicing method that is robust against observation noise. At the third stage, white balancing, color enhancement and inverse gamma correction are applied to only the demosaicked structural component. At the final stage, the two processed components are combined. Our proposed IP pipeline succeeds in achieving color interpolation, denoising and deblurring, simultaneously.

To remove signal-dependent noise of a digital camera, we present a denoising approach via nonlinear image-decomposition. In the approach, at the first decomposition stage, multiplicative image-decomposition is performed, and a noisy image is represented as a product of its two components so that its structural component corresponding to a cartoon image approximation may not be corrupted by the noise and its texture component may collect almost all the noise. At the successive denoising stage, the structural component is used instead of the unknown true signal value, to adapt the soft-thresholding-type denoising manipulation of the texture component to the signal dependency of the noise. At the final image-synthesis stage, one combines the separated structure component with the denoised texture component to reproduce a denoised image. The approach selectively removes the signal-dependent noise without not only blurring sharp edges but also destroying visually important textures.

In a digital camera, several factors cause signal-dependency of additive noise. Many denoising methods have been proposed, but unfortunately most of them do not work well for the actual signal-dependent noise. To solve the problem of removing the signal-dependent noise of a digital camera, we present a denoising approach via the nonlinear imagedecomposition. In the nonlinear decomposition-and-denoising approach, at the first nonlinear image-decomposition stage, multiplicative image-decomposition is performed, and a noisy image is represented as a product of its two components so that its structural component corresponding to a cartoon approximation of the noisy image may not be corrupted by the noise and its texture component may collect almost all the noise. At the successive nonlinear denoising stage, intensity of the separated structural component is utilized instead of the unknown true signal value, to adapt the soft-thresholding-type denoising manipulation of the texture component to the signal dependency of the noise. At the final image-synthesis stage, the separated structure component is combined with the denoised texture component, and thus a sharpness-improved denoised image is reproduced. The nonlinear decomposition-and-denoising approach can selectively remove the signal-dependent noise of a digital camera without not only blurring sharp edges but also destroying visually important textures.

Time varying motion blur is sometimes brought by camera shake, and alternate blurred frames and sharp frames are included in video clips. This degradation is considered a kind of breathing distortion, which means oscillatory changes of blur. We propose a motion deblurring method which aims to suppress breathing distortions. A spatio-temporal regularization is introduced into the deconvolution approach to smooth the temporal change of blur.

In a digital camera, its output image is sometimes corrupted by additive noise heavily and its noisy image is often compressed with the JPEG encoder. When the coding rate of the JPEG encoder is not high enough, in a JPEG-decoded image there appear noticeable artifacts such as the blocking, the ringing, and the false color artifacts. In the high ISOsensitivity case, even if the coding rate is very high, camera's noise will produce noticeably annoying artifacts in a JPEG-decoded image. This paper presents a restoration-type decoding approach that recovers a quality-improved image from the JPEG-compressed data, while not only suppressing the occurrence of the coding artifacts particular to the JPEG compression but also removing the camera's noise to some extent. This decoding approach is a kind of superresolution image-restoration approach based on the TV (Total Variation) regularization; to reduce the ringing artifacts near sharp edges it selectively restores the DCT coefficients truncated by the JPEG compression, whereas in an originally smooth image region it flattens unnecessary signal variations to eliminate the blocking artifacts and the camera's noise. Extending the standard ROF (Rudin-Osher-Fetami) framework of the TV image restoration, in this paper we construct the super-resolution approach to the JPEG decoding. By introducing the JPEG-compressed data into the fidelity term of the energy functional and adopting a nonlinear cost function constrained by the JPEG-compressed data softly, we define a new energy functional whose minimization gives the super-resolution JPEG decoding.

A motion deblurring method with spatio-temporal regularization is presented. In conventional motion deblurring, the best image for each frame is restored individually. The aim of proposed method is that the all frames achieve the same level of smoothness, and an extended total variation is introduced to the deconvolution approach. Our method can be applied to a video clip acquired by handheld camera with camera shakes. Experimental results show that the strongly blurred images are deblurred sufficiently and the temporal change is suppressed.

The Retinex theory deals with separation of irradiance and reflectance from an image. Recently, Kimmel and others proposed a variational framework that unifies the previous Poisson-equation-type Retinex algorithms developed by Horn and others, and presented a separation algorithm with the time-evolution of a linear diffusion process. However, this separation algorithm cannot achieve physically rational separation, if true irradiance varies among color channels. To cope with this problem, we introduce a nonlinear diffusion process into the time-evolution. Moreover, we introduce an approach to treat all color channels collectively. Furthermore, we apply our separation algorithm to a high quality chroma key where a color of each pixel in the foreground frame are adaptively corrected through transformation of the separated irradiance.

The DCI (direct color-imaging) approach is promising as an acquisition scheme of color data with high spatial-resolution. The DCI sensor has three photo-sensing layers along its depth direction. Although each pixel has three color channels, their spectral sensitivities are overlapped with each other and the approach has a problem in color separation. To cope with this problem, we present a hybrid color-imaging approach between the DCI approach and the CFA (color-filter-array) approach. Our hybrid approach uses the DCI sensor, and pastes green and magenta color filters on pixels' surfaces according to a checkered pattern. The use of the green-magenta CFA improves color separation, but its sensed color channels have different spectra from the primary color-matching functions. Moreover, its sensed three color channels are sub-sampled according to the checkered pattern. To recover a primary tri-stimulus image with full spatial-resolution, not only color transformation of the sensed color channels but also their interpolation are needed. We present image recovery methods for the color transformation and the interpolation. Our hybrid color-imaging approach using the methods achieves higher spatial-resolution than the CFA approach, and shows the best color-reproduction among the three color-imaging approaches.

This paper presents an adaptable interpolation approach that can adjust edge sharpness and texture intensity to reconstruct a high quality image according to user's taste in picture quality. Our interpolation approach first resolves an input image I into its skeleton image U and its texture generator V and its residual image D such that I = U·V + D, and then interpolates each of the three components independently with a proper interpolation method suitable to each. The skeleton image is a bounded-variation function meaning a cartoon approximation of I, and interpolated with a super-resolution deblurring-oversampling method that interpolates sharp edges without producing ringing artifacts. The texture generator is an oscillatory function representing regular distinct textures, and interpolated with a standard linear interpolation algorithm. The residual image is a function with some randomness, and interpolated with a statistical re-sampling interpolation algorithm.

We previously presented a demosaicking method that simultaneously removes image blurs caused by an optical low-pass filter used in a digital color camera with the Bayer's RGB color filter array. Our prototypal sharpening-demosaicking method restored only spatial frequency components lower than the Nyquist frequency corresponding to the mosaicking pattern, but it often produced ringing artifacts near color edges. To overcome this difficulty, afterward we introduced the super-resolution into the prototypal method. We formulated the recovery problem in the DFT domain, and then introduced the super-resolution by the total-variation (TV) image regularization into the sharpening-demosaicking approach. The TV-based super-resolution effectively demosaiced sharp color images while preserving such image structures as intensity values are almost constant along edges, without producing ringing artifacts. However, the TV image regularization works as the smoothing, and it tends to suppress small intensity variations excessively. Hence, the TV-based super-resolution sharpening-demosaicking approach tends to crash texture details in texture image regions. To remedy the drawback, in this paper we introduce a spatially adaptive technique that controls the TV image regularization according to the saliency of color edges around a pixel.

As an imaging scheme with a single solid-state image sensor, the direct color-imaging approach is considered promising as an acquisition scheme of color data with high spatial-resolution. The sensor has three photo-sensing layers along its depth direction. Although each pixel has three color channels, their spectral sensitivities are overlapped with each other to a considerable extent, and the direct color-imaging approach is considered to have a problem in color separation. To cope with this problem, this paper presents a hybrid color-imaging approach between the direct color-imaging approach and the color-filter-array approach. Our hybrid approach uses a direct color-imaging sensor that has three photo-sensing layers, and pastes the green and the magenta color filters on pixels' surfaces of the direct color-imaging sensor according to a checkered pattern. The use of the checkered green-magenta color-filter-array improves color separation, but the sensed reddish and the sensed bluish color channels still have somewhat different color spectra from those of the red and the blue primary color channels. In addition to it, the sensed green color channel and the sensed reddish / bluish color channels are sub-sampled in a ratio of 2 to 1 according to the checkered pattern. To recover RGB primary color images with full spatial resolution, not only the color transformation of the reddish and the bluish color channels but also the interpolation of the sensed three color channels are needed. This paper presents a method for the color transformation and a method for the interpolation. Our methods achieve high spatial resolution better than the pure color-filter-array approach, and at the same time improve color separation to some extent.

The Retinex theory was first proposed by Land, and deals with separation of irradiance from reflectance in an observed image. The separation problem is an ill-posed problem. Land and others proposed various Retinex separation algorithms. Recently, Kimmel and others proposed a variational framework that unifies the previous Retinex algorithms such as the Poisson-equation-type Retinex algorithms developed by Horn and others, and presented a Retinex separation algorithm with the time-evolution of a linear diffusion process. However, the Kimmel's separation algorithm cannot achieve physically rational separation, if true irradiance varies among color channels. To cope with this problem, we introduce a nonlinear diffusion process into the time-evolution. Moreover, as to its extension to color images, we present two approaches to treat color channels: the independent approach to treat each color channel separately and the collective approach to treat all color channels collectively. The latter approach outperforms the former. Furthermore, we apply our separation algorithm to a high quality chroma key in which before combining a foreground frame and a background frame into an output image a color of each pixel in the foreground frame are spatially adaptively corrected through transformation of the separated irradiance. Experiments demonstrate superiority of our separation algorithm over the Kimmel's separation algorithm.

Previously we have presented a selective image sharpening method based on the coupled nonlinear diffusion process composed of a nonlinear diffusion term, a fidelity term and an isotropic peaking term, and it can sharpen only blurred edges without increasing the noise visibility. Our previously presented prototypal color-image sharpening methods based on the coupled nonlinear-diffusion process have been formulated on the linear color models, namely, the channel-bychannel model and the 3D vectorial model. Our prototypal methods can sharpen blurred color step edges, but they do not necessarily enhance contrasts of signal variations in complex texture image regions so well as in simple step-edge regions. To remedy the drawback, this paper extends our coupled nonlinear-diffusion color-image sharpening method to the nonlinear non-flat color model, namely, the chromaticity-brightness model, which is known to be closely relating to human color perception. We modify our time-evolution PDE's for the non-flat space of the chromaticity vector and present its digital implementations. Through experimental simulations, we compare our new color-image sharpening method based on the chromaticity-brightness model with our prototypal color-image sharpening methods based on the linear color models.

This chapter develops very high definition image acquisition system which is based on the signal-processing approach with multiple cameras. The approach produces an improved resolution image with sufficiently high signal-to-noise ratio by processing and integrating multiple images taken simultaneously with multiple cameras. Originally, in this approach, we used multiple cameras with the same pixel aperture, but in this case there needs to be severe limitations both in the arrangement of multiple cameras and in the configuration of the scene, in order to guarantee the spatial uniformity of the resultant resolution. To overcome this difficulty completely, this chapter presents the utilization of multiple cameras with different pixel apertures, and develops a new, alternately iterative signal-processing algorithm available in the different aperture case. Experimental simulations and results are also presented. These results show that the utilization of multiple different-aperture cameras prospects well and that the alternately iterative algorithm behaves satisfactorily.

In recent years, researchers in the advanced television system area have concentrated on high definition television(HDTV). Although HDTV has has twice the spatial resolution as compared to current broadcasting television, in comparison to 35-mm film which is considered to be a typical quality criterion in non-electronic visual media, HDTV quality is still insufficient. When the FTTH (Fiber To The Home) based telecommunication is realized in the future then it will provide us with large transmission capacity and there is still an open way to much higher difinition image communication with at least '2000x2000 pixel resolution which are expected to integrate all visual information including non-electronic materials such as 35-mm films for printing, motion pictures, etc,. Image acquisition method is one of the principal problems for handling very high resolution images. Although a CCD camera with two million pixels has already been developed for HDTV, it is still necessary to increase the resolution further for dealing with very high resolution images. Reducing the size of pixels is one way to increase resolution. However, the smaller the pixel size is, the amount of light available for each pixel is less and the picture quality is degraded since the existence of shot-noise (variation of input) is inevitable in principle. There needs to be a limitation in the pixel size reduction in order to keep shot noise invisible on a monitor. Current CCD technology has almost reached this bound. Thus, new schemes are required to increase the resolution even further. One promising approach to further improve resolution is to incorporate signal processing techniques into the imaging process. iFrom the signal processing point of view, we propose a new imaging scheme to acquire high resolution images by processing multiple pictures. Proposed method contains two different mechanisms to further increase resolution. In one mechanism, sampling sequences of two camera are integrated together after their relative discrepancies are estimated, and higher frequencies are reconstructed from this integrated double density sampling points. The other mechanism improves signal-to-noise-ratio by averaging the higher resolution images reconstructed by coupling different two images among multiple cameras. SJN improvement enables us to furt,her reduce pixel size in advance because resolution and S/N have trade-off relationship. Our preliminary experiment have clearly shown the improvements in the high frequencies and details.

The optical low-pass filter used for a digital camera is formed by combining two types of doubly refractive crystal device. Since the filter cannot sharply cut off high frequency components and it reduces frequency components under the Nyquist frequency, images projected on the imaging surface are blurred. This paper presents a demosaicking method that simultaneously removes image blurs caused by the optical low-pass filter. Our sharpening-demosaicking method employs the Landweber-type iterative algorithm. Since the Bayer's RGB color filter array is not necessarily proper for our method, we study another better color filter array, the WRB filter array where the W-filtering means that all the visible light passes through it. Our sharpening-demosaicking approach is formulated as the least square problem, but it has multiple solutions. To avoid the ambiguity, we introduce the frequency-band limitation corresponding to the mosaicking pattern of color filters, into the iterative algorithm.

We previously presented a method of selective image sharpening based on a coupled nonlinear diffusion process that was composed of a nonlinear diffusion term, a fidelity term and an isotropic peaking term. It could sharpen only blurred edges without increasing noise visibility. This paper extends our method to the removal of blurs in images due to image motion. Motion blur is not only shift-variant but also anisotropic. To adapt our method to motion de-blurring, we replaced our isotropic peaking term, with an anisotropic peaking term steered in direction of motion at each pixel. We then devised discrete calculus to adapt it to the direction of motion. Through experiments with a test sequence of moving images containing artificial motion-blurs, we quantitatively evaluated sharpening performance. Our new method with the anisotropic peaking term performed better than our prototypal method with the isotropic peaking term, and was robust against errors in the direction of motion.

Most digital color cameras use doubly refractive crystal devices as an optical low-pass filter. The filter reduces frequency components lower than the Nyquist frequency, and images are blurred. We previously presented a demosaicking method that simultaneously removes blurs caused by the optical low-pass filter. For the sharpening-demosaicking approach, the Bayer's RGB color filter array is not necessarily proper, and we studied another color-filter array, namely the WRB filter array, where the W-filtering means that all the visible light passes through it. Our prototypal sharpening-demosaicking method employed the iterative algorithm, and restored only spatial frequency components of color signals lower than the Nyquist frequency corresponding to the mosaicking pattern of the W filters. However, the same recovery problem is solved by a non-iterative method in the Discrete Fourier Transform domain. Moreover, our prototypal method often produced ringing artifacts near sharp color edges. To suppress those artifacts, we introduce the total-variation-based super-resolution into the sharpening-demosaicking approach. This super-resolution approach restores spatial frequency components higher than the Nyquist frequency from observed blurry spatial frequency components so that without producing ringing artifacts it can demosaic and sharpen images while preserving 1D image structures that intensity values are almost constant along the edge direction.

As an imaging scheme with a single solid-state image sensor, the direct color-imaging approach is considered promising as an acquisition scheme of multi-spectral color data with high spatial-resolution. The sensor has multiple photo-sensing layers more than two along its depth direction. Although each pixel has multiple color signals, their spectral sensitivities are overlapped with each other to a considerable extent. The overlapped color signals should be transformed to color signals such as RGB color signals and/or YMC color signals that are specified by an output device. We present a color transformation method for the direct color-imaging scheme. Our method tries to recover multi-spectral color data and then to transform the sensed color signals by utilizing the recovered multi-spectral color data. We formulate the problem as the inverse problem that multi-spectral color data with a large number of color channels are recovered from observed color signals with a smaller number of color channels, e.g. five channels, and solve it by the regularization technique that minimizes the energy functional composed of a color-fidelity energy term, a spectral-regularity energy term and so on. The color-fidelity energy term quantifies back-projection errors in the linear color transformation from the recovered multi-spectral color data to the sensed color signals, whereas the spectral-regularity energy term quantifies the smoothness property that a spectral reflectance at a color channel is similar to those at its neighboring channels. We simulate the direct color-imaging scheme and our color-transformation method. The results show that in the case of more than four photo-sensing layers our method restores multi-spectral reflectance acceptably.