Search

Depth image-based rendering (DIBR) techniques have recently drawn more attention in various 3D applications. In this paper, a real-time high-quality DIBR system that consists of disparity estimation and view synthesis is proposed. For disparity estimation, a local approach that focuses on depth discontinuities and disparity smoothness is presented to improve the disparity accuracy. For view synthesis, a method that contains view interpolation and extrapolation is proposed to render high-quality virtual views. Moreover, the system is designed with an optimized parallelism scheme to achieve a high throughput, and can be scaled up easily. It is implemented on an Altera Stratix IV FPGA at a processing speed of 45 frames per second for 1080p resolution. Evaluated on selected image sets of the Middlebury benchmark, the average error rate of the disparity maps is 6.02%; the average peak signal to noise ratio and structural similarity values of the virtual views are 30.07 dB and 0.9303, respectively. The experimental results indicate that the proposed DIBR system has the top-performing processing speed and its accuracy performance is among the best of state-of-the-art hardware implementations. [ABSTRACT FROM AUTHOR]

In this paper, we propose an ordinal deep learning approach for facial age estimation. Unlike conventional hand-crafted feature-based methods that require prior and expert knowledge, we propose an ordinal deep feature learning (ODFL) method to learn feature descriptors for face representation directly from raw pixels. Motivated by the fact that age labels are chronologically correlated and age estimation is an ordinal learning problem, our proposed ODFL enforces two criteria on the descriptors, which are learned at the top of the deep networks: 1) the topology-preserving ordinal relation is employed to exploit the order information in the learned feature space and 2) the age-difference cost information is leveraged to dynamically measure face pairs with different age value gaps. However, both the procedures of feature extraction and age estimation are learned independently in ODFL, which may lead to a sub-optimal problem. To address this, we further propose an end-to-end ordinal deep learning (ODL) framework, where the complementary information of both the procedures is exploited to reinforce our model. Extensive experimental results on five face aging datasets show that both our ODFL and ODL achieve superior performance in comparisons with most state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts. [ABSTRACT FROM PUBLISHER]

In this paper, a novel spectral reflectance estimation method from image pairs including near-infrared (NIR) components based on nonnegative matrix factorization (NMF) is presented. The proposed method enables estimation of spectral reflectance from only two kinds of input images: 1) an image including both visible light components and NIR components and 2) an image including only NIR components. These two images can be easily obtained using a general digital camera without an infrared-cut filter and one with a visible light-cut filter, respectively. Since RGB values of these images are obtained according to spectral sensitivity of the image sensor, the spectrum power distribution of the light source and the spectral reflectance, we have to solve the inverse problem for estimating the spectral reflectance. Therefore, our method approximates spectral reflectance by a linear combination of several bases obtained by applying NMF to a known spectral reflectance data set. Then estimation of the optimal solution to the above problem becomes feasible based on this approximation. In the proposed method, NMF is used for obtaining the bases used in this approximation from a characteristic that the spectral reflectance is a nonnegative component. Furthermore, the proposed method realizes simple approximation of the spectrum power distribution of the light source with direct and scattered light components. Therefore, estimation of spectral reflectance becomes feasible using the spectrum power distribution of the light source in our method. In the last part of this paper, we show some simulation results to verify the performance of the proposed method. The effectiveness of the proposed method is also shown using the method for several applications that are closely related to spectral reflectance estimation. Although our method is based on a simple scheme, it is the first method that realizes the estimation of the spectral reflectance and the spectrum power distribution of the light source from the above two kinds of images taken by general digital cameras and provides breakthroughs to several fundamental applications. [ABSTRACT FROM PUBLISHER]

Object recognition is widely used in vision computing for various applications. Traditional CPU and application specific integrated circuit for vision computing cannot provide high performance and enough flexibility, which limit the use of vision systems. In this paper, a hierarchical and parallel pipelined heterogeneous chip for object recognition is proposed to achieve high flexibility, high performance, and area efficiency. In addition, a reformulation of 3D position estimation is proposed. The method uses single precision to achieve the short computing time and accuracy requirement. The hardware resource is small. Application-specific components, such as connected component information extractor and information extraction accelerator, are designed for high performance. Reconfiguration processors and application-specific instruction set processor are introduced to improve flexibility. These components are connected to hierarchical parallel buses. The chip is fabricated in 180-nm CMOS technology and occupies 72.25 mm2 with 1.09M bits on-chip memory. It delivers 204 GOPS + 665M FLOPS operations. The results show that this hierarchical and parallel pipelined heterogeneous chip is suitable for embedded vision systems. [ABSTRACT FROM AUTHOR]

This paper presents a robust foreground detection method capable of adapting to different motion speeds in scenes. A key contribution of this paper is the background estimation using a proposed novel algorithm, neighbor-based intensity correction (NIC), that identifies and modifies the motion pixels from the difference of the background and the current frame. Concretely, the first frame is considered as an initial background that is updated with the pixel intensity from each new frame based on the examination of neighborhood pixels. These pixels are formed into windows generated from the background and the current frame to identify whether a pixel belongs to the background or the current frame. The intensity modification procedure is based on the comparison of the standard deviation values calculated from two pixel windows. The robustness of the current background is further measured using pixel steadiness as an additional condition for the updating process. Finally, the foreground is detected by the background subtraction scheme with an optimal threshold calculated by the Otsu method. This method is benchmarked on several well-known data sets in the object detection and tracking domain, such as CAVIAR 2004, AVSS 2007, PETS 2009, PETS 2014, and CDNET 2014. We also compare the accuracy of the proposed method with other state-of-the-art methods via standard quantitative metrics under different parameter configurations. In the experiments, NIC approach outperforms several advanced methods on depressing the detected foreground confusions due to light artifact, illumination change, and camera jitter in dynamic scenes. [ABSTRACT FROM PUBLISHER]

Binary image descriptors, which derive image feature description from the local image patches directly, are widely adopted in the mobile and embedded applications due to lower computational complexity and memory requirement. With the aim of improving the computation efficiency without degrading recognition performance, a lightweight binary robust descriptor is proposed based on the analysis of the state-of-the-art binary descriptors in this paper. A directional edge detection and optimized keypoint score function are developed to refine the keypoints. In addition, rotation invariance is achieved by executing circular symmetric-based descriptor generation and a coarse-grained orientation calculation method concurrently. The experimental results demonstrate that the proposed keypoint detector and binary descriptor achieve more than two times speedup and at least 23.6% improvement in processing speed with comparable performance, respectively. Furthermore, a very large scale integration architecture is also designed based on in-depth exploration of bit-level and task-level parallelism. Based on the postlayout simulation in a TSMC 65-nm CMOS process, the accelerator can achieve 135 frames/s on 1080p image while only consuming 87.5 mW at a 200-MHz operating frequency. [ABSTRACT FROM PUBLISHER]

Human action can be recognized from a single still image by modeling human–object interactions (HOIs), which infers the mutual spatial structure information between human and the manipulated object as well as their appearance. Existing approaches rely heavily on accurate detection of human and object and estimation of human pose; they are thus sensitive to large variations of human poses, occlusion, and unsatisfactory detection of small size objects. To overcome this limitation, a novel exemplar-based approach is proposed in this paper. Our approach learns a set of spatial pose–object interaction exemplars, which are probabilistic density functions describing spatially how a person is interacting with a manipulated object for different activities. Specifically, a new framework consisting of an exemplar-based HOI descriptor and an associated matching model is formulated for robust human action recognition in still images. In addition, the framework is extended to perform HOI recognition in videos, where the proposed exemplar representation is used for implicit frame selection to negate irrelevant or noisy frames by temporal structured HOI modeling. Extensive experiments are carried out on two image action datasets and two video action datasets. The results demonstrate the effectiveness of our proposed methods and show that our approach is able to achieve state-of-the-art performance, compared with several recently proposed competitors. [ABSTRACT FROM PUBLISHER]

Homography estimation is a fundamental problem in the field of computer vision. For estimating the homography between two images, one of the key issues is to match keypoints in the reference image to the keypoints in the moving image. To match keypoints in real time, a binary image descriptor, due to its low matching and storage costs, emerges as a more and more popular tool. Upon achieving the low costs, the binary descriptor sacrifices the discriminative power of using floating points. In this paper, we present BB-Homography, a new approach that fuses fast binary descriptor matching and bipartite graph for homography estimation. Starting with binary descriptor matching, BB-Homography uses bipartite graph matching (GM) algorithm to refine the matching results, which are finally passed over to estimate homography. On realizing the correlation between keypoint correspondence and homography estimation, BB-Homography iteratively performs the GM and the homography estimation such that they can refine each other at each iteration. In particular, based on spectral graph, a fast bipartite GM algorithm is developed for lowering the time cost of BB-Homography. BB-Homography is extensively evaluated on both public benchmarks and live-captured video streams that consistently shows that BB-Homography outperforms conventional methods for homography estimation. [ABSTRACT FROM PUBLISHER]

Visible light cameras commonly used for surveillance applications usually have a limited field of view (FoV). To acquire a broader FoV, distributed aperture systems (DASs) combine views from multiple cameras. Although some commercial or proprietary systems already exist, the open literature in this field reports solely the performance and/or hardware architecture of the respective systems, and omits the required details for a reimplementation. In this paper we present a low-cost, personal-computer-based 360° DAS, with a full description of the hardware architecture and the software implementation details. In particular, we describe in detail two problems, the offline estimation of the orientations of the cameras of the proposed DAS and the online synthesis of a virtual view in real time. For the first problem, we propose an area-based bundle adjuster by combining the forward additive Lucas–Kanade algorithm with a bundle adjustment strategy. The color virtual view of up to $2048\times 1024$ pixels is synthesized online at 39 frames/s. A large majority of the workload of the online phase is implemented on the graphic processing unit. The techniques and methods proposed in this paper are generic and independent on the arrangement of cameras. [ABSTRACT FROM PUBLISHER]

Detection of moving objects in videos is a crucial step toward successful surveillance and monitoring applications. A key component for such tasks is called background subtraction and tries to extract regions of interest from the image background for further processing or action. For this reason, its accuracy and real-time performance are of great significance. Although effective background subtraction methods have been proposed, only a few of them take into consideration the special characteristics of thermal imagery. In this paper, we propose a background subtraction scheme, which models the thermal responses of each pixel as a mixture of Gaussians with unknown number of components. Following a Bayesian approach, our method automatically estimates the mixture structure, while simultaneously it avoids over-/underfitting. The pixel density estimate is followed by an efficient and highly accurate updating mechanism, which permits our system to be automatically adapted to dynamically changing operation conditions. We propose a reference implementation of our method in reconfigurable hardware achieving both adequate performance and low-power consumption. Adopting a high-level synthesis design and demanding floating point arithmetic operations are mapped in reconfigurable hardware, demonstrating fast prototyping and on-field customization at the same time. [ABSTRACT FROM AUTHOR]

Due to the rapid growth of technology and the dropping cost of cameras, multiview imaging applications have attracted many researchers in recent years. Free viewpoint and 3D Televisions are among these interesting applications. One of the problems that should be solved to realize such applications is rendering. In this paper, we propose an optical flow-assisted adaptive patch-based view synthesis algorithm. This patch-based scheme reduces the size and number of holes during reconstruction. The size of patch is determined in response to edge information for better reconstruction, especially near the boundaries. In the first stage of the algorithm, disparity is obtained using optical flow estimation. Then, a reconstructed version of the left and right views is generated using our adaptive patch-based algorithm. The mismatches between each view and its reconstructed version are obtained in the mismatch detection steps. This stage results in two masks as outputs, which help with the refinement of disparities and the selection of the best patches for final synthesis. Finally, the remaining holes are filled using our simple hole-filling scheme and the refined disparities. The objective and subjective performances of the proposed algorithm are compared with recent methods. The results show that the proposed algorithm achieves an improvement of 2.14 dB on average. [ABSTRACT FROM AUTHOR]

Eye movements play a very significant role in human–computer interaction (HCI) as they are natural and fast, and contain important cues for human cognitive state and visual attention. Over the last two decades, many techniques have been proposed to accurately estimate the gaze. Among these, video-based remote eye trackers have attracted much interest, since they enable nonintrusive gaze estimation. To achieve high estimation accuracies for remote systems, user calibration is inevitable in order to compensate for the estimation bias caused by person-specific eye parameters. Although several explicit and implicit user calibration methods have been proposed to ease the calibration burden, the procedure is still cumbersome and needs further improvement. In this paper, we present a comprehensive analysis of regression-based user calibration techniques. We propose a novel weighted least squares regression-based user calibration method together with a real-time cross-ratio based gaze estimation framework. The proposed system enables to obtain high estimation accuracy with minimum user effort, which leads to user-friendly HCI applications. Experimental results conducted on both simulations and user experiments show that our framework achieves a significant performance improvement over the state-of-the-art user calibration methods when only a few points are available for the calibration. [ABSTRACT FROM PUBLISHER]

Image dehazing is a crucial image processing step for outdoor vision systems. However, images recovered through conventional image dehazing methods that use either haze-relevant priors or heuristic cues to estimate transmission maps may not lead to sufficiently accurate haze removal from single images. The most commonly observed effects are darkened and brightened artifacts on some areas of the recovered images, which cause considerable loss of fidelity, brightness, and sharpness. This paper develops a variational image dehazing method on the basis of a color-transfer image dehazing model that is superior to conventional image dehazing methods. By creating a color-transfer image dehazing model to remove haze obscuration and acquire information regarding the coefficients of the model by using the devised convolutional neural network-based deep framework as a supervised learning strategy, an image fidelity, brightness, and sharpness can be effectively restored. The experimental results verify through quantitative and qualitative evaluations of either synthesized or real haze images, and the proposed method outperforms existing single image dehazing methods. [ABSTRACT FROM AUTHOR]

With the advances in commercial light field cameras, light field image processing has attracted considerable attention from researchers. In this paper, we propose a novel method for generating stereoscopic images from a light field image pair with flexible control over the convergence of the virtual stereo cameras. We have developed a light field image-capturing prototype that consists of two horizontally arranged light field cameras (i.e., with their optical axes being parallel to each other). When using our proposed device for image/video capture, stereo photographers can concentrate on how to capture the desired visual experience without being frequently disturbed by having to manipulate the stereo camera parameters, i.e., the convergence angle of a stereo camera. During postprocessing, our method estimates accurate disparity maps for the light field image pair and then generates the target stereoscopic images that satisfy the desired stereo camera convergence requirements by adopting a novel view synthesis method for light field images. We have conducted extensive experiments to demonstrate the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

Resampling detection is one of the most important topics in image forensics, and the most widely used method in resampling detection is spectral analysis. Since JPEG is the most widely used image format, it is reasonable that the resampling operation is processed on JPEG images. JPEG block artifacts bring severe interference to spectrum-based methods and degrade the detection performance. In addition, the spectral characteristics of the downscaling scenarios are very weak. The detection of downscaling still presents a considerable challenge to forensic applications. In this paper, we propose a method to estimate the downscaling factors of pre-JPEG compressed images in the presence of image downscaling after JPEG compressions. We first analyze the spectrum of scaled images and give an exact formulation of how the scaling factors influence the appearance of periodic artifacts. The expected positions of the characteristic resampling peaks are analytically derived. For the downscaling scenario, the shifted JPEG block artifacts produce periodic peaks, which cause misdetection in the characteristic peak. We find that the interval between the adjacent extrema of difference images obeys the geometric distribution and the distribution has periodic peaks for JPEG images. Hence, we adopt the difference image extremum interval histogram and combine the spectral method to obtain the final estimation. The experimental results demonstrate that the proposed detection method outperforms some state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Pedestrian detection is an important yet challenging task. In this paper, a pedestrian codetection framework for binocular stereo sequences is proposed. Binocular vision and consecutive frames can provide more information than a single image. That information can be used to improve detection performance by reducing the number of candidate detection windows with low confidence or enhancing the high-confidence candidates. To design the framework, we follow the intuition that a pedestrian has consistent appearance when observed from the same or different viewpoints. First, a baseline detector is used in both stereo images with a conservative threshold in order to expand the set of detection candidates. Before the detection process, an assisted stixel world model is computed for both left and right frames. Thus, the search range of the detector is greatly reduced with the help of the stixel world. Second, adjacency-constrained patch matching is proposed to build correspondence between two candidates in both intra and inter sequences of binocular vision. Finally, we establish a mechanism to update the score of the detection aided by the corresponding candidates. The experimental results show that our framework significantly improves the performance of the evaluated baseline pedestrian approaches. [ABSTRACT FROM AUTHOR]

High-quality and cost-efficient image upscaling design is very important for many real-time video processing applications, especially when the display panel resolution reaches ultrahigh definition. Compared with New Edge-Directed Interpolation (NEDI) based implicit edge directional upscaling, explicit methods require less computational resource and more easily reach real-time performance, especially when the required image definition and upscaling ratio are very high. Nevertheless, the investigation of applications of explicit methods in video processing systems remains largely missing arguably because it is commonly believed that explicit edge-directed interpolation tends to introduce unexpected artifacts because of inaccurate detection and hence its image quality is relatively poor. This paper proposes an explicit edge-directed adaptive interpolation method that leverages more sophisticated edge detection and orientation estimation algorithms to avoid misinterpolation, thereby providing similar or even better image quality than those with implicit methods. Targeting the real-time 4K video display system, the proposed edge-directed image upscaling algorithm is further implemented with an efficient very-large-scale integration (VLSI) architecture. The experimental results demonstrate that the proposed interpolation algorithm outperforms previous explicit and implicit edge-directed methods in both objective and subjective tests. The presented VLSI implementation further demonstrates that the maximum output video sequence of the proposed interpolation method can reach 4\mathrm k\times 2\textk @60 Hz with a reasonable hardware cost. [ABSTRACT FROM AUTHOR]

We develop a framework for estimating the noise level of a natural image using two important statistics: 1) high kurtosis and 2) scale invariance in transform domain. By exploring the said priors of natural image statistics in 2-D discrete cosine transform (DCT) domain, we reveal the limitations of these statistics for images with highly directional edges or large smooth areas. Then, we derive a novel two-step estimation scheme for noise variance: 1) in preliminary estimation, an integration of wavelet and nondirectional DCT transform is used to alleviate the influence of image’s structures and 2) a noise-injection rectification is further devised to deal with the noise-free image contents. A simulation and comparative study demonstrates that this algorithm reliably infers noise variance and its robustness over wide ranges of visual content and noise levels, while outperforming some relevant methods. This paper can significantly improve the performance of existing denoising techniques that require the noise variance as a critical parameter. [ABSTRACT FROM PUBLISHER]

The autoregressive (AR) model has been widely used in signal processing for its effective estimation, especially in image processing. Many dedicated $2\times $ interpolation algorithms adopt the AR model to describe the strong correlation between low-resolution (LR) pixels and high-resolution (HR) pixels. However, these AR model-based methods closely depend on the fixed relative position between LR pixels and HR pixels that are nonexistent in the general scale interpolation. In this paper, we present an adaptive general scale interpolation algorithm that is capable of arbitrary scaling factors considering the nonstationarity of natural images. Different from other dedicated $2\times $ interpolation methods, the proposed AR terms are modeled by pixels with their adjacent unknown HR neighbors. To compensate for the information loss caused by mismatches of AR models, we consider a weighting scheme suitable for general scale situations based on the pixel similarity to increase accuracy of the estimation. Comprehensive experiments demonstrate the effectiveness of the proposed method on general scaling factors. The maximum gain of peak signal-to-noise ratio is 2.07 dB compared with segment adaptive gradient angle in $1.5 \times $ enlargements. To evaluate the performance in resolution adaptive video coding, we have also tested our method on Joint Scalable Video Model codec and obtained better subjective quality and rate-distortion performance. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a novel energy optimization framework to accurately estimate surface normal and reflectance of an object from an input image sequence. Input images are captured from a fixed viewpoint under varying lighting conditions. In the proposed approach we combine photometric stereo and Retinex constraints into our energy function. To formulate inter-image constraints, shading information is added to the Lambertian model to account for shadows. For intra-image constraints, we moderate the strength of shading smoothness according to shadow mask and normal variations. By minimizing this energy function we are able to recover accurate surface normals and reflectance. Experimental results show that our approach yields more realistic normal map and accurate albedo map than the state-of-the-art uncalibrated photometric stereo algorithms. As for intrinsic image decomposition, results on the real and synthetic scenes show that the proposed approach outperforms previous ones. [ABSTRACT FROM PUBLISHER]

This paper presents a maximum a posteriori (MAP) framework to incorporate a spatial consistency prior model for depth reconstruction in the shape-from-focus (SFF) process. Existing SFF techniques, which reconstruct a dense 3-D depth from multifocus image frames, usually have poor performance over low-contrast regions and usually need a large number of frames to achieve satisfactory results. To overcome these problems, a new depth reconstruction process is proposed to estimate the depth values by solving an MAP estimation problem with the inclusion of a spatial consistency model. This consistency model assumes that within a local region, the depth value of each pixel can be roughly predicted by an affine transformation of the image features at that pixel. A local learning process is proposed to construct the consistency model directly from the multifocus image sequence. By adopting this model, the depth values can be inferred in a more robust way, especially over low-contrast regions. In addition, to improve the computational efficiency, a cell-based version of the MAP framework is proposed. Experimental results demonstrate the effective improvement in accuracy and robustness as compared with existing approaches over real and synthesized image data. In addition, experimental results also demonstrate that the proposed method can achieve quite impressive performance, even with only the use of a few image frames. [ABSTRACT FROM AUTHOR]

The visibility of outdoor images captured in inclement weather is often degraded due to the presence of haze, fog, sandstorms, and so on. Poor visibility caused by atmospheric phenomena in turn causes failure in computer vision applications, such as outdoor object recognition systems, obstacle detection systems, video surveillance systems, and intelligent transportation systems. In order to solve this problem, visibility restoration (VR) techniques have been developed and play an important role in many computer vision applications that operate in various weather conditions. However, removing haze from a single image with a complex structure and color distortion is a difficult task for VR techniques. This paper proposes a novel VR method that uses a combination of three major modules: 1) a depth estimation (DE) module; 2) a color analysis (CA) module; and 3) a VR module. The proposed DE module takes advantage of the median filter technique and adopts our adaptive gamma correction technique. By doing so, halo effects can be avoided in images with complex structures, and effective transmission map estimation can be achieved. The proposed CA module is based on the gray world assumption and analyzes the color characteristics of the input hazy image. Subsequently, the VR module uses the adjusted transmission map and the color-correlated information to repair the color distortion in variable scenes captured during inclement weather conditions. The experimental results demonstrate that our proposed method provides superior haze removal in comparison with the previous state-of-the-art method through qualitative and quantitative evaluations of different scenes captured during various weather conditions. [ABSTRACT FROM AUTHOR]

This paper proposes a new framework for joint image registration and high-resolution (HR) image reconstruction from multiple low-resolution (LR) observations with zooming motion. Conventional super-resolution (SR) methods typically formulate the SR problem as a two-stage process, namely, image registration followed by HR reconstruction. An important step in image SR is the effective estimation of motion parameters. However, the registration algorithms in these two-stage processes experience various degrees of errors. This could degrade the quality of subsequent HR reconstruction. In view of this, this paper presents a new approach that performs joint image registration and SR reconstruction. The proposed iterative SR framework enables the HR image and motion parameters to be estimated simultaneously and progressively. This could increase the potential SR improvement as more accurate estimates of motion parameters could be obtained iteratively. Experimental results show that the proposed method is effective in performing image registration and SR for simulated and real-life images and videos. [ABSTRACT FROM AUTHOR]

Belief propagation based algorithms perform best in disparity estimation but suffer from high computational complexity and storage, especially in message passing. This paper proposes an efficient architecture design with three techniques to solve the problems. For the memory storage, we propose the spinning-message and the sliding-bipartite node plane that can reduce memory cost to 1.2% for image-scale algorithms and 23.4% for block-scale algorithms, when compared to the traditional approach. For the logic complexity, we propose a buffer-free processing element architecture that has 3.6 times hardware efficiency of the previous work. The three proposed techniques could be applied to various belief propagation based algorithms to save significant hardware cost as well as approach real-time speed. [ABSTRACT FROM PUBLISHER]

We present a high-quality temporal frame interpolation (TFI) method that employs piecewise-smooth motion and handles disoccluded regions using the observation that motion discontinuities travel with the foreground object. We derive a “motion discontinuity” likelihood map from the divergence of a motion field between the input frames. Motion which is modeled at the reference frame is mapped to the target frame using a cellular-affine mapping strategy—a process during which regions of disocclusion are readily observed. This information is then used to guide the occlusion-aware, bidirectional FI process. Furthermore, we propose two computationally inexpensive texture optimizations that selectively improve the quality of the interpolated frames in regions around moving objects. The scheme produces very high-quality interpolated frames and outperforms current high-quality state-of-the-art TFI schemes by 2–2.5 dB; the method works with a very low-complexity motion estimation scheme and runs orders of magnitudes faster than its competitors. [ABSTRACT FROM AUTHOR]

This letter presents an approach for automatically estimating the spatial bound of the blur kernel in a motion-blurred image based on the statistics of multilevel image gradients. We observe that blur has a significant impact on the histogram of oriented gradients (HOGs) at higher levels of an image pyramid, but has much less of an impact at coarser levels. Based on this fact, we estimate the spatial bound of the unknown blur kernel using a learning-based approach. We first learn a generic pyramid HOG model from natural sharp images, then given an HOG pyramid of a blurry image, we predict the corresponding model of its latent sharp image. Finally, we learn another model to predict the spatial kernel bound from the difference between the observed and the predicted HOG pyramids. Experimental results show that the proposed method can estimate accurate blur kernel sizes, enabling existing blind deconvolution methods to achieve best possible results. [ABSTRACT FROM PUBLISHER]

Optical flow estimation is a fundamental task in computer vision and image processing. Due to the difficulty in obtaining the ground truth of flow field, unsupervised learning approaches attract more and more research interests in recent years. However, despite of their good generalization capability, unsupervised optical flow methods suffer in the scenarios with large displacement, small objects, and occlusions. In this work, we propose a novel optical flow network based on decoder with multi-scale kernels. Different from previous U-Net like or pyramidal methods, we design our network based on RAFT architecture that with a 4D correlation layer and recurrent decoder. More importantly, we incorporate three novel ideas with regard to the input, information processing and output of the update units improve the performance. Firstly, we utilize various motion-related information as input to the update units. Secondly, we propose a module of multi-scale update unit. Thirdly, for the final flow up-sampling procedure, we propose an image-guided up-sampling loss to guide the learning of up-sampling masks. Our model is trained by the occlusion-aware photometric loss, edge-aware smoothness loss, self-supervised loss, and image-guided up-sampling loss. Experimental results demonstrate that our model achieves the state-of-the-art performance on both Sintel and KITTI and outperforms other unsupervised optical flow methods remarkably. [ABSTRACT FROM AUTHOR]

A well-known problem in stereoscopic videos is visual fatigue. However, conventional depth adjustment methods provide little guidance in deciding the amount of depth control or determining the condition for depth control. We propose a depth adjustment method based on binocular fusion characteristics, where the fusion time is used as the parameter for adjustment. Visual comfort enhancement is implemented with the horizontal image shift approach for depth adjustment. The speeded-up robust feature is used to estimate maximum disparity and disparity distribution, while a face detection algorithm is used to estimate the viewing distance with a single web camera. Binocular fusion characteristics are acquired in advance by measuring the time required for fusion under various conditions, including foreground disparity, background disparity, and focal distance for random-dot stereograms. Finally, a subjective evaluation is conducted in fixed and free-to-move viewing conditions, and the results show that comfortable videos were generated based on the proposed depth adjustment method. [ABSTRACT FROM PUBLISHER]

We present a novel approach to free-viewpoint video. Our main contribution is the formulation of a hybrid approach between image morphing and depth-image based rendering. When rendering the scene from novel viewpoints, we use both dense pixel correspondences between image pairs as well as an underlying, view-dependent geometrical model. Our novel reconstruction scheme iteratively refines geometric and correspondence information. By combining the strengths of both depth and correspondence estimation, our approach enables free-viewpoint video also for challenging scenes as well as for recordings that may violate typical constraints in multiview reconstruction. For example, our method is robust against inaccurate camera calibration, asynchronous capture, and imprecise depth reconstruction. Rendering results for different scenes and applications demonstrate the versatility and robustness of our approach. [ABSTRACT FROM PUBLISHER]

In just noticeable difference (JND) models, evaluation of contrast masking (CM) is a crucial step. More specifically, CM due to edge masking (EM) and texture masking (TM) needs to be distinguished due to the entropy masking property of the human visual system. However, TM is not estimated accurately in the existing JND models since they fail to distinguish TM from EM. In this letter, we propose an enhanced pixel domain JND model with a new algorithm for CM estimation. In our model, total-variation based image decomposition is used to decompose an image into structural image (i.e., cartoon like, piecewise smooth regions with sharp edges) and textural image for estimation of EM and TM, respectively. Compared with the existing models, the proposed one shows its advantages brought by the better EM and TM estimation. It has been also applied to noise shaping and visual distortion gauge, and favorable results are demonstrated by experiments on different images. [ABSTRACT FROM PUBLISHER]