Search

Applying image-based processing methods to original videos on a framewise level breaks the temporal consistency between consecutive frames. Traditional video temporal consistency methods reconstruct an original frame containing flickers from corresponding nonflickering frames, but the inaccurate correspondence realized by optical flow restricts their practical use. In this article, we propose a temporally broad learning system (TBLS), an approach that enforces temporal consistency between frames. We establish the TBLS as a flat network comprising the input data, consisting of an original frame in an original video, a corresponding frame in the temporally inconsistent video on which the image-based technique was applied, and an output frame of the last original frame, as mapped features in feature nodes. Then, we refine extracted features by enhancing the mapped features as enhancement nodes with randomly generated weights. We then connect all extracted features to the output layer with a target weight vector. With the target weight vector, we can minimize the temporal information loss between consecutive frames and the video fidelity loss in the output videos. Finally, we remove the temporal inconsistency in the processed video and output a temporally consistent video. Besides, we propose an alternative incremental learning algorithm based on the increment of the mapped feature nodes, enhancement nodes, or input data to improve learning accuracy by a broad expansion. We demonstrate the superiority of our proposed TBLS by conducting extensive experiments.

The development of technologies that can generate Deepfake videos is expanding rapidly. These videos are easily synthesized without leaving obvious traces of manipulation. Though forensically detection in high-definition video datasets has achieved remarkable results, the forensics of compressed videos is worth further exploring. In fact, compressed videos are common in social networks, such as videos from Instagram, Wechat, and Tiktok. Therefore, how to identify compressed Deepfake videos becomes a fundamental issue. In this paper, we propose a two-stream method by analyzing the frame-level and temporality-level of compressed Deepfake videos. Since the video compression brings lots of redundant information to frames, the proposed frame-level stream gradually prunes the network to prevent the model from fitting the compression noise. Aiming at the problem that the temporal consistency in Deepfake videos might be ignored, we apply a temporality-level stream to extract temporal correlation features. When combined with scores from the two streams, our proposed method performs better than the state-of-the-art methods in compressed Deepfake videos detection.

Most existing studies performed video denoising by using multiple adjacent noisy frames to recover one clean frame; however, despite achieving relatively good quality for each individual frame, these approaches may result in visual flickering when the denoised frames are considered in sequence. In this paper, instead of separately restoring each clean frame, we propose a multiframe-to-multiframe (MM) denoising scheme that simultaneously recovers multiple clean frames from consecutive noisy frames. The proposed MM denoising scheme uses a training strategy that optimizes the denoised video from both the spatial and temporal dimensions, enabling better temporal consistency in the denoised video. Furthermore, we present an MM network (MMNet), which adopts a spatiotemporal convolutional architecture that considers both the interframe similarity and single-frame characteristics. Benefiting from the underlying parallel mechanism of the MM denoising scheme, MMNet achieves a highly competitive denoising efficiency. Extensive analyses and experiments demonstrate that MMNet outperforms the state-of-the-art video denoising methods, yielding temporal consistency improvements of at least 13.3% and running more than 2 times faster than the other methods.

Current approaches of human body skeleton extraction mainly suffer from following problems: insufficient temporal and spatial continuity, unrobust to background, ambient noise, etc. This paper proposes a three-dimensional human body skeleton extraction method from consecutive meshes. We extract the consistent skeletons from consecutive surfaces based on shape segmentation and skeleton sequences; then, we present a spatiotemporal skeleton optimization model to adjust the skeleton sequences. Experiments on multiview images captured from a light field device demonstrate that our method captures more complete and accurate skeletons compared to state-of-the-art methods.

Video stitching enables creation of videos with a wide field of view from videos captured from ordinary, probably moderately displaced, cameras. In this paper, a novel video stitching algorithm based on multi-view feature detection and registration using region-based statistics has been proposed. While stitching the videos captured using moving cameras, the motion in those cameras will affect the alignment of video frames. Movements in the cameras and in the objects will affect the temporal consistency while stitching the videos. Temporally reliable feature points are considered to maintain both spatial and temporal consistency using multi-view spatiotemporal feature points in this paper. Experiments have been carried out on different data sets, and the results showed the better and effective performance of the proposed approach.

Temporal consistency information plays an important role in video segmentation. Some previous automatic annotation methods inevitably caused the temporal consistency information loss of sequential frames when processing sparse labeled frames. Thus, as a supplement, interpolation method was proposed to compensate motion flows between two frames via interpolating high-precision frames and corresponding labels, but its quality is remained to be improved. We propose a new interpolation module to smooth the large motion between frames and preserve temporal consistency. First, we use an optical flow estimator to directly estimate the optical flow, and then refine the interpolation frame by combining the features of the original frames. We evaluate our method on the MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset. Experimental results demonstrate our method can well preserve the temporal consistency information and outperform the state-of-the-art method in segmentation accuracy.

We propose a method for the unsupervised reconstruction of a temporally-coherent sequence of surfaces from a sequence of time-evolving point clouds, yielding dense, semantically meaningful correspondences between all keyframes. We represent the reconstructed surface as an atlas, using a neural network. Using canonical correspondences defined via the atlas, we encourage the reconstruction to be as isometric as possible across frames, leading to semantically-meaningful reconstruction. Through experiments and comparisons, we empirically show that our method achieves results that exceed that state of the art in the accuracy of unsupervised correspondences and accuracy of surface reconstruction., Comment: ICCV 2021

We present a novel method to learn temporally consistent 3D reconstruction of clothed people from a monocular video. Recent methods for 3D human reconstruction from monocular video using volumetric, implicit or parametric human shape models, produce per frame reconstructions giving temporally inconsistent output and limited performance when applied to video. In this paper, we introduce an approach to learn temporally consistent features for textured reconstruction of clothed 3D human sequences from monocular video by proposing two advances: a novel temporal consistency loss function; and hybrid representation learning for implicit 3D reconstruction from 2D images and coarse 3D geometry. The proposed advances improve the temporal consistency and accuracy of both the 3D reconstruction and texture prediction from a monocular video. Comprehensive comparative performance evaluation on images of people demonstrates that the proposed method significantly outperforms the state-of-the-art learning-based single image 3D human shape estimation approaches achieving significant improvement of reconstruction accuracy, completeness, quality and temporal consistency., Comment: To appear in Dynavis Workshop, CVPR 2021

Video super-resolution, which utilizes the relevant information of several low-resolution frames to generate high-resolution images, is a challenging task. One possible solution called sliding window method tries to divide the generation of high-resolution video sequences into independent sub-tasks, and only adjacent low-resolution images are used to estimate the high-resolution version of the central low-resolution image. Another popular method named recurrent algorithm proposes to utilize not only the low-resolution images but also the generated high-resolution images of previous frames to generate the high-resolution image. However, both methods have some unavoidable disadvantages. The former one usually leads to bad temporal consistency and requires higher computational cost while the latter method always can not make full use of information contained by optical flow or any other calculated features. Thus more investigations need to be done to explore the balance between these two methods. In this work, a bidirectional frame recurrent video super-resolution method is proposed. To be specific, a reverse training is proposed that the generated high-resolution frame is also utilized to help estimate the high-resolution version of the former frame. With the contribution of reverse training and the forward training, the idea of bidirectional recurrent method not only guarantees the temporal consistency but also make full use of the adjacent information due to the bidirectional training operation while the computational cost is acceptable. Experimental results demonstrate that the bidirectional super-resolution framework gives remarkable performance that it solves the time-related problems when the generated high-resolution image is impressive compared with recurrent-based video super-resolution method.

Recently, developing temporally consistent video-based processing techniques has drawn increasing attention due to the defective extend-ability of existing image-based processing algorithms (e.g., filtering, enhancement, colorization, etc). Generally, applying these image-based algorithms independently to each video frame typically leads to temporal flickering due to the global instability of these algorithms. In this paper, we consider enforcing temporal consistency in a video as a temporal denoising problem that removing the flickering effect in given unstable pre-processed frames. Specifically, we propose a novel model termed Temporal Denoising Mask Synthesis Network (TDMS-Net) that jointly predicts the motion mask, soft optical flow and the refining mask to synthesize the temporal consistent frames. The temporal consistency is learned from the original video and the learned temporal features are applied to reprocess the output frames that are agnostic (blind) to specific image-based processing algorithms. Experimental results on two datasets for 16 different applications demonstrate that the proposed TDMS-Net significantly outperforms two state-of-the-art blind temporal consistency approaches.

Superpixel segmentation has been proved its effectiveness as a preprocessing step for many applications. Similarly, the over-segmentation with temporal consistency of a video can be useful for quite a few researches. In this paper, a novel supervoxel segmentation algorithm based on lazy random walk is proposed. In the proposed framework, a superpixel segmentation is first applied to each frame and the produced superpixels are called special-pixels. Then, we propose a generalized spatio-temporal adjacency matrix, which keeps the information of special-pixels with neighboring relationship, for the generation of supervoxels using lazy random walk approach. In addition, the center relocation and the splitting strategy of supervoxels are proposed to improve the quality of the supervoxels.

Recent progress in video style transfer has shown promising results which contain less flickering effects. However, existing algorithms mainly trade off generality for efficiency, i.e., constructing one network per style example, and often work for short video clips only. In this work, we propose a video multi-style transfer (VMST) framework which enables fast and multi-style video transfer within one single network. Specifically, we design a multi-instance normalization block (MIN-Block) to learn different style examples and two ConvLSTM modules to encourage the temporal consistency. The proposed algorithm is demonstrated to be able to generate temporally-consistent video transfer results in different styles while keeping each stylized frame visually pleasing. Extensive experimental results show that the proposed method performs favorably against single-style models and some post-processing techniques that alleviate the flickering issue. We achieve as many as 120 stylization effects in a single model and show results on long-term videos that consist of thousands of frames.

Despite the recent success of single image-based 3D human pose and shape estimation methods, recovering temporally consistent and smooth 3D human motion from a video is still challenging. Several video-based methods have been proposed; however, they fail to resolve the single image-based methods' temporal inconsistency issue due to a strong dependency on a static feature of the current frame. In this regard, we present a temporally consistent mesh recovery system (TCMR). It effectively focuses on the past and future frames' temporal information without being dominated by the current static feature. Our TCMR significantly outperforms previous video-based methods in temporal consistency with better per-frame 3D pose and shape accuracy. We also release the codes. For the demo video, see https://youtu.be/WB3nTnSQDII. For the codes, see https://github.com/hongsukchoi/TCMR_RELEASE., Comment: Accepted to CVPR 2021, 10 pages

The application of image processing techniques to individual frames of video often results in temporal inconsistency. Conventional approaches used for preserving the temporal consistency in videos have shortcomings as they are used for only particular jobs. Our work presents a multipurpose video temporal consistency preservation method that utilizes an adaptive simple linear iterative clustering (SLIC) algorithm. First, we locate the inter-frame correspondent pixels through the SIFT Flow and use them to find the respective regions. Then, we apply a multiframe matching statistical method to get the spatially or temporally correspondent frames. Besides, we devise a least-squares energy-based flickering-removing objective function by taking into account the inter-frame temporal consistency and inter-region spatial consistency jointly. The obtained results demonstrate the potential of the proposed method.

Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of the current frame, which we call the guidance image. We further propose a novel neural network architecture to take advantage of the information stored in the guidance images. Extensive experimental results on several challenging datasets verify the effectiveness of our approach in achieving world consistency—the output video is consistent within the entire rendered 3D world.

Semantic segmentation is a fundamental task in indoor scene understanding. Most previous supervised approaches rely on densely annotated image data sets. Due to the limited amount of images with segmentation labels, the performance of existing networks is greatly limited. In this paper, we exploit temporal correlation in video frames to improve the performance and robustness of segmentation networks. Two effective learning strategies are proposed to propagate the information from a few labeled frames to their immediate neighbor frames. First, we scale up training dataset for supervised semantic segmentation networks by generating pseudo ground-truth for neighboring frames from a labeled frame using filtered homography transformation. Furthermore, we introduce a self-supervised loss function to ensure temporal consistency between the segmentation results of adjacent frames. The experimental results demonstrate that our proposed method outperforms state-of-the-art techniques for semantic segmentation on NYU-Depth V2 dataset.

Lane and road marker segmentation is crucial in autonomous driving, and many related methods have been proposed in this field. However, most of them are based on single-frame prediction, which causes unstable results between frames. Some semantic multi-frame segmentation methods produce error accumulation and are not fast enough. Therefore, we propose a deep learning algorithm that takes into account the continuity information of adjacent image frames, including image sequence processing and an end-to-end trainable multi-input single-output network to jointly process the segmentation of lanes and road markers. In order to emphasize the location of the target with high probability in the adjacent frames and to refine the segmentation result of the current frame, we explicitly consider the time consistency between frames, expand the segmentation region of the previous frame, and use the optical flow of the adjacent frames to reverse the past prediction, then use it as an additional input of the network in training and reasoning, thereby improving the network’s attention to the target area of the past frame. We segmented lanes and road markers on the Baidu Apolloscape lanemark segmentation dataset and CULane dataset, and present benchmarks for different networks. The experimental results show that this method accelerates the segmentation speed of video lanes and road markers by 2.5 times, increases accuracy by 1.4%, and reduces temporal consistency by only 2.2% at most.

This study targets 2D articulated human pose estimation (i.e. localisation of body limbs) in stereo videos. Although in recent years depth-based devices (e.g. Microsoft Kinect) have gained popularity, as they perform very well in controlled indoor environments (e.g. living rooms, operating theatres or gyms), they suffer clear problems in outdoor scenarios and, therefore, human pose estimation is still an interesting unsolved problem. The authors propose here a novel approach that is able to localise upper-body keypoints (i.e. shoulders, elbows, and wrists) in temporal sequences of stereo image pairs. The authors' method starts by locating and segmenting people in the image pairs by using disparity and appearance information. Then, a set of candidate body poses is computed for each view independently. Finally, temporal and stereo consistency is applied to estimate a final 2D pose. The authors' validate their model on three challenging datasets: `stereo human pose estimation dataset', `poses in the wild' and `INRIA 3DMovie'. The experimental results show that the authors' model not only establishes new state-of-the-art results on stereo sequences, but also brings improvements in monocular sequences.

The two distinctive visual experiences of binocular display, with and without stereoscopic glasses, have been recently utilized for visual sharing between the colorblind and the normal-vision audiences. However, all existing methods only work for still images, and lack of temporal consistency for video application. In this paper, we propose the first synthesis method for colorblind-sharable videos that possess the temporal consistency for both visual experiences of colorblind and normal-vision, and retains all other crucial characteristics for visual sharing with colorblind. We formulate this challenging multi-constraint problem as a global optimization and minimize an objective function consisting of temporal term, color preservation term, color distinguishability term, and binocular fusibility term. Qualitative and quantitative experiments are conducted to evaluate the effectiveness of the proposed method comparing to existing methods.

Spatiotemporal image fusion is considered as a promising way to provide Earth observations with both high spatial resolution and frequent coverage, and recently, learning-based solutions have been receiving broad attention. However, these algorithms treating spatiotemporal fusion as a single image super-resolution problem, generally suffers from the significant spatial information loss in coarse images, due to the large upscaling factors in real applications. To address this issue, in this paper, we exploit temporal information in fine image sequences and solve the spatiotemporal fusion problem with a two-stream convolutional neural network called StfNet . The novelty of this paper is twofold. First, considering the temporal dependence among image sequences, we incorporate the fine image acquired at the neighboring date to super-resolve the coarse image at the prediction date. In this way, our network predicts a fine image not only from the structural similarity between coarse and fine image pairs but also by exploiting abundant texture information in the available neighboring fine images. Second, instead of estimating each output fine image independently, we consider the temporal relations among time-series images and formulate a temporal constraint. This temporal constraint aiming to guarantee the uniqueness of the fusion result and encourages temporal consistent predictions in learning and thus leads to more realistic final results. We evaluate the performance of the StfNet using two actual data sets of Landsat-Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and both visual and quantitative evaluations demonstrate that our algorithm achieves state-of-the-art performance.

This paper proposes a method for detecting forged objects in videos that include dynamic scenes such as dynamic background or non-stationary scenes. In order to adapt to dynamic scenes, we combine Convolutional Neural Network and Recurrent Neural Network. This enables us to consider spatio-temporal consistency of videos. We also construct new video forgery databases for object modification as well as object removal. Our proposed method using Convolutional Long Short-Term Memory achieved Area-Under-Curve (AUC) 0.977 and Equal-Error-Rate (EER) 0.061 on the object addition database. We also achieved AUC 0.872 and EER 0.219 on the object modification database.

This paper proposes a new video retargeting method that utilizes the cropping and bilinear scaling techniques. Specifically, the cropping positions for a given image are determined by using results of the logo and saliency detection, and the strength of the bilinear scaling is determined depending on the size of cropping. As a result, the proposed method effectively reduces distortion of visually important contents and maintains a temporal consistency. Experimental results showed that the proposed method provided the highest preference compared with the existing methods in the paired-comparison-based user study evaluation.

In this paper we develop a vision-based rain intensity measurement method for dynamic scenes. The method first measures the area density of rain by analyzing temporal changes in pixel values in the video input. The area density, represented as a binary rain map, is then mapped to a rain intensity value using linear regression. To ensure temporal consistency of scene content across frames in dynamic scenes, we applied superpixel-based content alignment. Potential false detections in the binary rain map are removed using directional morphological opening. Experiments show that both superpixel-based content alignment and morphological opening are important for good rain map generation and rain intensity estimation

The foreground extraction of video is derived from frame difference and background subtraction. These methods have always relied on the temporal consistency of successive video frames. If the video consistency is suddenly destroyed, such as an object being occluded, some frames are lost, or the camera is shaken, etc., the results of foreground extraction may be significantly degraded. In this paper, our extends One-Shot Video Object Segmentation by adding two branchs for Positioning foreground targets and propagating foreground semantic information. These two branches are parallel to the existing foreground segmentation branches, which facilitates efficient iterative and fine-tuning of the foreground segmentation branches. This foreground extraction method is different from the previous background subtraction and frame difference methods. Our method does not rely on the time information and only needs to give the manual annotation of one frame, which can extract all the specific foreground in the video sequence. Then, the coloring can get complete foreground information for the separated target. Experiments show that the foreground extraction algorithm applied in this paper has better robustness in dynamic background, camera shake, intermittent motion of objects, low contrast, and so on.

Many Defocus blur estimation methods have been proposed in recent years but, when applied to video sequences in a frame-by-frame manner, they typically exhibit temporal inconsistencies or flickering. This paper presents a temporal coherence scheme that can be coupled to any existing defocus blur estimation for still images, aiming to produce spatiotemporally coherent defocus blur map videos. The proposed method is based on the design of a Kalman Filter which is applied on a patch level. Experimental results show that the proposed method can smooth out undesirable temporal fluctuations whilst still being able to preserve the abrupt local appearance changes due to motion, occlusions or dis-occlusions.

This paper presents a fully automated approach to classifying deformable and non-deformable moving objects in a video surveillance scene. We estimate an object's motion using Marzat optical-flow algorithm. We filter the motion vectors and attempt to find the transformation that represents the correct mapping between the two positions. The Fundamental transformation is estimated using the Normalized Eight-Point Algorithm. We introduce a new type of graph to set the thresholds between deformable and non-deformable motion. Furthermore, we use temporal consistency to classify deformable and non-deformable objects. For experiments, we used a varied corpus of real surveillance videos. Our proposed approach for motion classification achieved a precision rate of 92 percent.

Segmenting moving objects in a video sequence has been a challenging problem and critical to outdoor robotic navigation. While recent literature has laid focus on regularizing object labels over a sequence of frames, exploiting the spatio-temporal features for motion segmentation has been scarce. Particularly in real world dynamic scenes, existing approaches fail to exploit temporal consistency in segmenting moving objects with large camera motion.

We propose a novel approach to the real-time landing site detection and assessment in unconstrained man-made environments using passive sensors. Because this task must be performed in a few seconds or less, existing methods are often limited to simple local intensity and edge variation cues. By contrast, we show how to efficiently take into account the potential sites' global shape, which is a critical cue in man-made scenes. Our method relies on a new segmentation algorithm and shape regularity measure to look for polygonal regions in video sequences. In this way, we enforce both temporal consistency and geometric regularity, resulting in very reliable and consistent detections. We demonstrate our approach for the detection of landable sites such as rural fields, building rooftops and runways from color and infrared monocular sequences significantly outperforming the state-of-the-art.

In this work, an online 3D reconstruction algorithm is proposed which attempts to solve the structure from motion problem for occluded and degenerate data. To deal with occlusion the temporal consistency of data within a limited window is used to compute local reconstructions. These local reconstructions are transformed and merged to obtain an estimation of the 3D object shape. The algorithm is shown to accurately reconstruct a rotating and translating artificial sphere and a rotating toy dinosaur from a video. The proposed algorithm (WIFAME) provides a versatile framework to deal with missing data in the structure from motion problem.

This paper proposes a novel hierarchical approach to improve the accuracy of the classification of normal-vs-abnormal frames in white-light colonoscopy videos. The existing approaches label each frame independently, without considering the temporal consistency between adjacent frames. Temporal consistency, however, can improve the classification accuracy in the presence of unclear/uncertain images. We propose to leverage temporal consistency between adjacent frames for colonoscopy video frame classification using a novel hierarchical classifier. Comparative experiments with five challenging full colonoscopy videos show that the proposed approach considerably improves the mean class normal/abnormal classification accuracy compared to the approaches where the frames are classified independently.

Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera “seeing” behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

In this study, an efficient, robust algorithm for automatic target detection and tracking is introduced. Procedure starts with a detection phase. Proposed method uses two alternatives for the detection phase, namely maximally stable extremal regions detector and Canny edge detector. After detection, regions of interest are evaluated and eliminated according to their compactness and effective saliency. The detection process is repeated for a predetermined number of pyramid levels where each level processes a downsampled version of input image to achieve scale invariance. Then, temporal consistency for detections from all scales is evaluated and target likelihood map is constructed using kernel density estimation in order to merge all target hypotheses. Finally, outstanding targets are selected from target likelihood map and tracking is achieved by minimizing spatial distance between the selected targets in consecutive frames.

Predicting human attention for video is requires exploiting temporal knowledge included in the video. We propose a novel hierarchical spatial-temporal saliency model for video based on the center-surround framework using both static features and temporal features. Saliency cues are analyzed through a hierarchical segmentation model, and fused across multiple levels, yielding the spatial-temporal saliency map. An adaptive temporal window using motion information is also developed to combine saliency values of consecutive frames in order to keep temporal consistency across frames. Performance evaluation on several popular benchmark datasets validates that our method outperforms existing state-of-the-arts.

We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.

International audience; Extending image processing techniques to videos is a non-trivial task; applying processing independently to each video frame often leads to temporal inconsistencies, and explicitly encoding temporal consistency requires algorithmic changes. We describe a more general approach to temporal consistency. We propose a gradient-domain technique that is blind to the particular image processing algorithm. Our technique takes a series of processed frames that suffers from flickering and generates a temporally-consistent video sequence. The core of our solution is to infer the temporal regularity from the original unprocessed video, and use it as a temporal consistency guide to stabilize the processed sequence. We formally characterize the frequency properties of our technique, and demonstrate , in practice, its ability to stabilize a wide range of popular image processing techniques including enhancement and stylization of color and tone, intrinsic images, and depth estimation.

This paper presents novel results on temporally consistent depth estimation with the use of noise removal from video. Basing on the prior works, a novel, more advanced noise reduction algorithm is proposed. It uses motion compensation and additional refinement for elimination of artifacts. Also, exhaustive results, both subjective and objective, are presented for commonly known MPEG multiview test sequence set. Various means for depth maps temporal consistency measurement are tested and, among them, bitrate-based approach is identified to be the most adequate. It is shown that the proposed noise reduction technique allows for a significant improvement of temporal-consistency of the depth, which was demonstrated with 30% bitrate reduction of depth MVC coding.

Based on the development of image registration, sequence registration can be done by computing the transformations between consecutive frames. To take into account the accumulated error, global registration method is usually employed as a global error minimizing approach. However, in real surveillance applications, the visible sequence and infrared sequence may be taken at different times, or from different viewpoints, and may have different dynamic contents. Therefore, global registration is only an approximate estimation for two sequences, resulting in inferior local contents. In this paper we present a novel integrated global-to-local framework that addresses the problems of dynamic infrared and visible image sequence registration. We propose to maximize the sum of the mutual information of two sequences for the global homography estimation. Then, frame-to-frame registration is performed to estimate the per-frame local homography. Finally, a smoothing strategy is adopted to smooth the local homographies in the temporal domain to enforce temporal consistency. We evaluate our proposed framework by comparing it to the state-of-the art sequence registration algorithm. Our method achieves improved performance on the public benchmark dataset.

Holography is an imaging technique to reconstruct wavefront information of the light scattered by real objects or a scene, allowing an observer to perceive three-dimensional 3D images with the unassisted eye. Such 3D holographic images result from reproducing the intensity and phase of light by diffraction. This paper presents a temporal consistency enhancement for digital holographic video. The proposed temporal consistency enhancement method improves compression efficiency and visual quality by reducing the flickering artifact.

In this paper, we propose a new video matting method based on sampling. By detecting the movement of foreground and background objects from video, we define the local transformation which transfer the small areas between different frames in the video. By the local transformation, foreground and the occluded background object can be sampled from neighbor frames. Benefited from temporal consistency, we can get high quality alpha matte. Experiment results show that our method achieves ideal results in the scenes where old method get poor results.

We propose a framework for temporally consistent video completion. To this end we generalize the exemplar-based inpainting method of Criminisi et al. [7] to video inpainting. Specifically we address two important issues: Firstly, we propose a color and optical flow inpainting to ensure temporal consistency of inpainting even for complex motion of foreground and background. Secondly, rather than requiring the user to hand-label the inpainting region in every single image, we propose a flow-based propagation of user scribbles from the first to subsequent video frames which drastically reduces the user input. Experimental comparisons to state-of-the-art video completion methods demonstrate the benefits of the proposed approach.

International audience; In this paper we address the problem of detecting reliably a subset of pedestrian targets (heads) in a high-density crowd exhibiting extreme clutter and homogeneity, with the purpose of obtaining tracking initializations. We investigate the solution provided by discriminative learning where we require that the detections in the image space be localized over most of the target area and temporally stable. The results of our tests show that discriminative learning strategies provide valuable cues about the target localization which may be combined with other complementary strategies in order to bootstrap tracking algorithms in these challenging environments.

In this paper, we tackle the problem of efficiently segmenting objects in weakly labeled videos. Internet videos (e.g., YouTube) are often associated with a semantic tag describing the main object within the video. However, this tag does not provide any spatial or temporal information about the object within the video. So these videos are weakly labeled. We propose a novel and efficient approach to localize the object of interest within the video and perform pixel-level segmentation. Given a video with an object tag, our proposed method automatically localizes the object and segments it from the background in each frame of the video. Our method combines object appearance modeling and temporal consistency among frames in a principled framework. Our method does not require user inputs or object detectors, so it can be potentially applied to videos of any object categories. We evaluate our method on a dataset consisting of more than 100 video shots of 10 different object categories. Our experimental results show that our method outperforms other baseline approaches.

Vision-based environment perception is particularly challenging in bad weather. Under such conditions, even most powerful stereo algorithms suffer from highly correlated, ”blob”-like noise, that is hard to model. In this paper we focus on extending an existing stereo-based scene representation – the Stixel World – to allow its application even under problematic conditions. To this end, we estimate the probability of existence for each detected obstacle. Results show that the amount of false detections can be reduced significantly by demanding temporal consistency of the representation and by analyzing cues that represent the geometry of typical obstacles.

Video editing and object recognition are two significant fields in computer vi- sion: the first has remarkably assisted digital production and post-production tasks of a digital video footage; the second is considered fundamental to image classification or image based search in large databases (e.g. the web). In this thesis, we address two problems, namely we present a novel formulation that tackles video editing tasks and we develop a mechanism that allows to generate more robust descriptors for objects in an image. Concerning the first problem, this thesis proposes two variational models to perform temporally coherent video editing. These models are applied to change an object’s (rigid or non-rigid) texture throughout a given video sequence. One model is based on propagating color information from a given frame (or be- tween two given frames) along the motion trajectories of the video; while the other is based on propagating gradient domain information. The models we present in this thesis require minimal user intervention and they automatically accommodate for illumination changes in the scene. Concerning the second problem, this thesis addresses the problem of affine invariance in object recognition. We introduce a way to generate geometric affine invariant quantities that are used in the construction of feature descrip- tors. We show that when these quantities are used they do indeed achieve a more robust recognition than the state of the art descriptors. i, La edición de vídeo y el reconocimiento de objetos son dos áreas fundamentales en el campo de la visión por computador: la primera es de gran utilidad en los procesos de producción y post-producción digital de vídeo; la segunda es esencial para la clasificación o búsqueda de imágenes en grandes bases de datos (por ejemplo, en la web). En esta tesis se acometen ambos problemas, en concreto, se presenta una nueva formulación que aborda las tareas de edición de vídeo y se desarrolla un mecanismo que permite generar descriptores más robustos para los objetos de la imagen. Con respecto al primer problema, en esta tesis se proponen dos modelos variacionales para llevar a cabo la edición de vídeo de forma coherente en el tiempo. Estos modelos se aplican para cambiar la textura de un objeto (rígido o no) a lo largo de una secuencia de vídeo dada. Uno de los modelos está basado en la propagación de la información de color desde un determinado cuadro de la secuencia de vídeo (o entre dos cuadros dados) a lo largo de las trayectorias de movimiento del vídeo. El otro modelo está basado en la propagación de la información en el dominio del gradiente. Ambos modelos requieren una intervención mínima por parte del usuario y se ajustan de manera automática a los cambios de iluminación de la escena. Con respecto al segundo problema, esta tesis aborda el problema de la invariancia afín en el reconocimiento de objetos. Se introduce un nuevo método para generar cantidades geométricas afines que se utilizan en la generación de descriptores de características. También se demuestra que el uso de dichas cantidades proporciona mayor robustez al reconocimiento que los descriptores existentes actualmente en el estado del arte.

This paper proposes a system to automatically segment the left ventricle in cardiac MR cine images. Individual frames are segmented using a shortest path algorithm and temporal consistency is enforced through the backward and forward deformation fields of an inverse consistent deformable registration. In addition, a segmentation of the mitral valve plane is obtained from long axis images. This algorithm was applied to 95 datasets as part of the STACOM'11 4D LV Segmentation Challenge. We analyze the results and evaluate the strengths and weaknesses of our system.

In free viewpoint video system based on multiview video plus depth, inconsistency with depth video need to be eliminated to ensure high-quality virtual view generation and compression performance. The preprocessing method proposed can compensate both spatial and temporal depth information inaccuracy by using Bayesian probability model and Rival penalized competitive learning in Self-Organizing Maps. Firstly, each gray value in depth video is assigned to specific class after clustering. Then gradient filter is utilized in smoothing. Experiments show that the proposed algorithm reduced the bit rate ranging 7.97%–46.83% while ensuring quality of generated virtual viewpoint.

The demand for high quality depth maps from stereo and multi-camera videos increases constantly. The main application for these depth maps is rendering new perspectives of the captured scene by means of Depth Image Based Rendering (DIBR). Accurate depth maps are the linchpin of DIBR. On the basis of a four-camera set-up, we show that combining hybrid recursive matching with motion estimation, cross -bilateral post-processing and mutual depth map fusion produces spatio-temporal consistent depth maps appropriate for artifact-free view synthesis.

This paper proposes a novel method for estimating temporally consistent disparity maps. The proposed approach utilizes a coherence function that represents the temporal consistency between the disparity maps of consecutive frames. The coherence function is computed according to a motion probability between consecutive frames based on the assumption that a disparity varies smoothly over time. The temporal consistency between consecutive disparity maps is enforced based on the coherence function. In order to cope with scene changes, the similarity of consecutive frames is also considered in the coherence function. Experimental results show that the proposed method yields satisfactory disparity maps and synthesized views on several challenging stereo videos, including clips with fast object and camera motions as well as scene change.

This paper presents a new approach to the problem of camera localization with non-overlapping camera views, particularly relevant for video surveillance systems. We show how to recast localization as quasi-convex optimization under the L-Infinity norm. Thereby we add the problem of reconstructing camera centers and 3D points for non-overlapping cameras with known internal parameters and known rotations to the class of known geometric problems solvable with Second Order Cone Programming. The 3D points are never seen by more than one camera, which makes the localization problem ill-posed. Therefore, the proposed approach employs temporal consistency of the 3D points to supply the missing constraints. Our formulation allows a global optimal solution to be found with a clear physical meaning of the cost function being minimized.

International audience; This paper presents a real-time stereo image sequences matching approach dedicated to intelligent vehicles applications. The main idea of the paper consists in integrating temporal information into the matching scheme. The estimation of the disparity map of an actual frame exploits the disparity map estimated for its preceding frame. An association between the two frames is searched, i.e. temporal integration. The disparity range is inferred for the actual frame based on both the association and the disparity map of the preceding frame. Dynamic programming technique is considered for matching the image features. As a similarity measure, a variance-based cost function is used. The proposed approach is tested on virtual and real stereo image sequences and the results are satisfactory. The method is fast and able to provide about 20 millions disparity maps per second on a HP Pavilion dv6700 2.1 GHZ.