Search

The development of technologies that can generate Deepfake videos is expanding rapidly. These videos are easily synthesized without leaving obvious traces of manipulation. Though forensically detection in high-definition video datasets has achieved remarkable results, the forensics of compressed videos is worth further exploring. In fact, compressed videos are common in social networks, such as videos from Instagram, Wechat, and Tiktok. Therefore, how to identify compressed Deepfake videos becomes a fundamental issue. In this paper, we propose a two-stream method by analyzing the frame-level and temporality-level of compressed Deepfake videos. Since the video compression brings lots of redundant information to frames, the proposed frame-level stream gradually prunes the network to prevent the model from fitting the compression noise. Aiming at the problem that the temporal consistency in Deepfake videos might be ignored, we apply a temporality-level stream to extract temporal correlation features. When combined with scores from the two streams, our proposed method performs better than the state-of-the-art methods in compressed Deepfake videos detection.

Most existing studies performed video denoising by using multiple adjacent noisy frames to recover one clean frame; however, despite achieving relatively good quality for each individual frame, these approaches may result in visual flickering when the denoised frames are considered in sequence. In this paper, instead of separately restoring each clean frame, we propose a multiframe-to-multiframe (MM) denoising scheme that simultaneously recovers multiple clean frames from consecutive noisy frames. The proposed MM denoising scheme uses a training strategy that optimizes the denoised video from both the spatial and temporal dimensions, enabling better temporal consistency in the denoised video. Furthermore, we present an MM network (MMNet), which adopts a spatiotemporal convolutional architecture that considers both the interframe similarity and single-frame characteristics. Benefiting from the underlying parallel mechanism of the MM denoising scheme, MMNet achieves a highly competitive denoising efficiency. Extensive analyses and experiments demonstrate that MMNet outperforms the state-of-the-art video denoising methods, yielding temporal consistency improvements of at least 13.3% and running more than 2 times faster than the other methods.

With the rapid development of photography, we can easily record the 3D world by taking photos and videos. In traditional images and videos, the viewer observes the scene from fixed viewpoints and cannot navigate the scene or edit the 2D observation afterward. Thus, visual content editing and synthesis become an essential task in computer vision. However, achieving high-quality visual synthesis often requires a complex and expensive multi-camera setup. This is not practical for daily use because most people only have one cellphone camera. But a single camera, on the contrary, could not provide enough multi-view constraints to synthesize consistent visual content. Therefore, in this thesis, I address this challenging single-camera visual synthesis problem by leveraging different regularizations. I study three consistent synthesis problems: time-consistent synthesis, view-consistent synthesis, and view-time-consistent synthesis. I show how we can take cellphone-captured monocular images and videos as input to model the scene and consistently synthesize new content for an immersive viewing experience. Doctor of Philosophy With the rapid development of photography, we can easily record the 3D world by taking photos and videos. More recently, we have incredible cameras on cell phones, which enable us to take pro-level photos and videos. Those powerful cellphones even have advanced computational photography features build-in. However, these features focus on faithfully recording the world during capturing. We can only watch the photo and video as it is, but not navigate the scene, edit the 2D observation, or synthesize content afterward. Thus, visual content editing and synthesis become an essential task in computer vision. We know that achieving high-quality visual synthesis often requires a complex and expensive multi-camera setup. This is not practical for daily use because most people only have one cellphone camera. But a single camera, on the contrary, is not enough to synthesize consistent visual content. Therefore, in this thesis, I address this challenging single-camera visual synthesis problem by leveraging different regularizations. I study three consistent synthesis problems: time-consistent synthesis, view-consistent synthesis, and view-time-consistent synthesis. I show how we can take cellphone-captured monocular images and videos as input to model the scene and consistently synthesize new content for an immersive viewing experience.

Video stitching enables creation of videos with a wide field of view from videos captured from ordinary, probably moderately displaced, cameras. In this paper, a novel video stitching algorithm based on multi-view feature detection and registration using region-based statistics has been proposed. While stitching the videos captured using moving cameras, the motion in those cameras will affect the alignment of video frames. Movements in the cameras and in the objects will affect the temporal consistency while stitching the videos. Temporally reliable feature points are considered to maintain both spatial and temporal consistency using multi-view spatiotemporal feature points in this paper. Experiments have been carried out on different data sets, and the results showed the better and effective performance of the proposed approach.

Multiple object tracking (MOT) is a critical module for enabling autonomous vehicles to achieve safe planing and navigation in cluttered environments. In tracking-by-detection systems, there are inevitably many false positives and misses among learning-based input detections. The challenge for MOT is to combine these detections into tracks, and filter them based on their uncertainties, states, and temporal consistency to achieve accurate and persistent tracks. In this paper, we propose to solve the 3D MOT problem for autonomous driving applications using a random finite set-based (RFS) Multiple Measurement Models filter (RFS-M3). In partiuclar, we propose multiple measurement models for a Poisson multi-Bernoulli mixture (PMBM) filter in support of different application scenarios. Our RFS-M3 filter can naturally model these uncertainties accurately and elegantly. We combine the learning-based detections with our RFS-M3 tracker through incorporating the detection confidence score into the PMBM prediction and update step. The superior experimental results of our RFS-M3 tracker on Waymo, Argoverse and nuSceness datasets illustrate that our RFS-M3 tracker outperforms state-of-the-art deep learning-based and traditional filter-based approaches. To the best of our knowledge, this represents a first successful attempt for employing an RFS-based approach in conjunction with 3D learning-based amodal detections for 3D MOT applications with comprehensive validation using challenging datasets made available by industry leaders.

The model drift problem is an inevitable problem for online visual tracking. Model drift can amplify the false tracking result over time and lead to failed tracking finally. Background-aware correlation filter (BACF) tracker obtains the training samples from negative background patches instead of shifted foreground patches and can mitigate the drifting to some extent. However, in complex scenes, such as occlusion, deformation, the inappropriate update of BACF may lead to model drift. We propose an anti-drift background-aware correlation filter via introducing the temporal consistency constraint into BACF. That is, we combine the excellent ability of BACF to distinguish the foreground and background with the ability of the temporal consistency constraint to stabilize model changes, and improve anti-drift performance of BACF. To improve computation efficient, we offer a fast algorithm via alternative Direction Method of Multipliers (ADMM) in the frequency domain. Besides, we design a simple yet effective adaptive feature channel selection method, which can further improve the success rate and precision of the tracker in complex scenes. Our proposed tracker with hand-crafted features achieves a gain of 2.2%, 3.3%, and 5.3% in AUC success rate on OTB-2013, OTB-2015, and Temple Color-128 dataset, respectively. Our proposed tracker with deep features achieves AUC success rate of 69.1% on OTB-2013 dataset. Besides, our proposed tracker with hand-crafted features achieves a gain of 3.94% in EAO on VOT-2018 dataset. Moreover, the proposed tracker performs favorably against several representative state-of-the-art methods regarding precision and success rate.

Temporal consistency information plays an important role in video segmentation. Some previous automatic annotation methods inevitably caused the temporal consistency information loss of sequential frames when processing sparse labeled frames. Thus, as a supplement, interpolation method was proposed to compensate motion flows between two frames via interpolating high-precision frames and corresponding labels, but its quality is remained to be improved. We propose a new interpolation module to smooth the large motion between frames and preserve temporal consistency. First, we use an optical flow estimator to directly estimate the optical flow, and then refine the interpolation frame by combining the features of the original frames. We evaluate our method on the MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset. Experimental results demonstrate our method can well preserve the temporal consistency information and outperform the state-of-the-art method in segmentation accuracy.

We present a novel method to learn temporally consistent 3D reconstruction of clothed people from a monocular video. Recent methods for 3D human reconstruction from monocular video using volumetric, implicit or parametric human shape models, produce per frame reconstructions giving temporally inconsistent output and limited performance when applied to video. In this paper, we introduce an approach to learn temporally consistent features for textured reconstruction of clothed 3D human sequences from monocular video by proposing two advances: a novel temporal consistency loss function; and hybrid representation learning for implicit 3D reconstruction from 2D images and coarse 3D geometry. The proposed advances improve the temporal consistency and accuracy of both the 3D reconstruction and texture prediction from a monocular video. Comprehensive comparative performance evaluation on images of people demonstrates that the proposed method significantly outperforms the state-of-the-art learning-based single image 3D human shape estimation approaches achieving significant improvement of reconstruction accuracy, completeness, quality and temporal consistency., Comment: To appear in Dynavis Workshop, CVPR 2021

Recent progress in video style transfer has shown promising results which contain less flickering effects. However, existing algorithms mainly trade off generality for efficiency, i.e., constructing one network per style example, and often work for short video clips only. In this work, we propose a video multi-style transfer (VMST) framework which enables fast and multi-style video transfer within one single network. Specifically, we design a multi-instance normalization block (MIN-Block) to learn different style examples and two ConvLSTM modules to encourage the temporal consistency. The proposed algorithm is demonstrated to be able to generate temporally-consistent video transfer results in different styles while keeping each stylized frame visually pleasing. Extensive experimental results show that the proposed method performs favorably against single-style models and some post-processing techniques that alleviate the flickering issue. We achieve as many as 120 stylization effects in a single model and show results on long-term videos that consist of thousands of frames.

Despite the recent success of single image-based 3D human pose and shape estimation methods, recovering temporally consistent and smooth 3D human motion from a video is still challenging. Several video-based methods have been proposed; however, they fail to resolve the single image-based methods' temporal inconsistency issue due to a strong dependency on a static feature of the current frame. In this regard, we present a temporally consistent mesh recovery system (TCMR). It effectively focuses on the past and future frames' temporal information without being dominated by the current static feature. Our TCMR significantly outperforms previous video-based methods in temporal consistency with better per-frame 3D pose and shape accuracy. We also release the codes. For the demo video, see https://youtu.be/WB3nTnSQDII. For the codes, see https://github.com/hongsukchoi/TCMR_RELEASE., Comment: Accepted to CVPR 2021, 10 pages

As a rapidly growing medium, volumetric video is gaining attention beyond academia, reaching industry and creative communities alike. This brings new challenges to reduce the barrier to entry from a technical and economical point of view. We present a system for robustly and autonomously performing temporally coherent tracking for volumetric sequences, specifically targeting those from sparse setups or with noisy output. Our system will detect and recover missing pertinent geometry across highly incoherent sequences as well as provide users the option of propagating drastic topology edits. In this way, affordable multi-view setups can leverage temporal consistency to reduce processing and compression overheads while also generating more aesthetically pleasing volumetric sequences.

Head detection in real-world videos is a classical research problem in computer vision. Head detection in videos is challenging than in a single image due to many nuisances that are commonly observed in natural videos, including arbitrary poses, appearances, and scales. Generally, head detection is treated as a particular case of object detection in a single image. However, the performance of object detectors deteriorates in unconstrained videos. In this paper, we propose a temporal consistency model (TCM) to enhance the performance of a generic object detector by integrating spatial-temporal information that exists among subsequent frames of a particular video. Generally, our model takes detection from a generic detector as input and improves mean average precision (mAP) by recovering missed detection and suppressing false positives. We compare and evaluate the proposed framework on four challenging datasets, i.e., HollywoodHeads, Casablanca, BOSS, and PAMELA. Experimental evaluation shows that the performance is improved by employing the proposed TCM model. We demonstrate both qualitatively and quantitatively that our proposed framework obtains significant improvements over other methods. Copyright © 2020 Sultan Daud Khan et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of the current frame, which we call the guidance image. We further propose a novel neural network architecture to take advantage of the information stored in the guidance images. Extensive experimental results on several challenging datasets verify the effectiveness of our approach in achieving world consistency—the output video is consistent within the entire rendered 3D world.

3D hand pose estimation based on RGB images has been studied for a long time. Most of the studies, however, have performed frame-by-frame estimation based on independent static images. In this paper, we attempt to not only consider the appearance of a hand but incorporate the temporal movement information of a hand in motion into the learning framework, which leads to the necessity of a large-scale dataset with sequential RGB hand images. We propose a novel method that generates a synthetic dataset that mimics natural human hand movements by re-engineering annotations of an extant static hand pose dataset into pose-flows. With the generated dataset, we train a newly proposed recurrent framework, exploiting visuo-temporal features from sequential synthetic hand images and emphasizing smoothness of estimations with temporal consistency constraints. Our novel training strategy of detaching the recurrent layer of the framework during domain finetuning from synthetic to real allows preservation of the visuo-temporal features learned from sequential synthetic hand images. Hand poses that are sequentially estimated consequently produce natural and smooth hand movements which lead to more robust estimations. Utilizing temporal information for 3D hand pose estimation significantly enhances general pose estimations by outperforming state-of-the-art methods in our experiments on hand pose estimation benchmarks.

In this paper, we propose a novel hierarchical scheme for detection and tracking of vehicles using a vehicle-mounted camera in nighttime under urban environment, where a vehicle can be represented by a pair of taillights and various types of lights are commonplace. The proposed scheme, therefore, mainly focuses on devising robust detection and pairing of taillights in spite of their inherent diversity and continuous transformation in appearance. Thus the appearance symmetry, which many conventional methods rely on, for paring is not guaranteed to be available all the times. Each of the three layers in the scheme is devised to identify a vehicle from individual lights and clutters detected in a hierarchical manner. Robust detection of a pair of taillights, which can be regarded as a vehicle, is sought by successive groupings of the components in a layer and checking not only the intra-layer but the inter-layer relations between them. A structural Kalman filter is employed to maintain the temporal consistency in the motion of the components and their relations as well. Exploiting such relational information increases accuracy in tracking of individual components by reducing effects from fluctuation in positions and shapes, and eventually compensating possible failures in detection of them. As a result, the proposed scheme achieves enhancement in detection and tracking of vehicles in nighttime as proven by experiments on videos including crowded urban traffic scenes.

Lane and road marker segmentation is crucial in autonomous driving, and many related methods have been proposed in this field. However, most of them are based on single-frame prediction, which causes unstable results between frames. Some semantic multi-frame segmentation methods produce error accumulation and are not fast enough. Therefore, we propose a deep learning algorithm that takes into account the continuity information of adjacent image frames, including image sequence processing and an end-to-end trainable multi-input single-output network to jointly process the segmentation of lanes and road markers. In order to emphasize the location of the target with high probability in the adjacent frames and to refine the segmentation result of the current frame, we explicitly consider the time consistency between frames, expand the segmentation region of the previous frame, and use the optical flow of the adjacent frames to reverse the past prediction, then use it as an additional input of the network in training and reasoning, thereby improving the network’s attention to the target area of the past frame. We segmented lanes and road markers on the Baidu Apolloscape lanemark segmentation dataset and CULane dataset, and present benchmarks for different networks. The experimental results show that this method accelerates the segmentation speed of video lanes and road markers by 2.5 times, increases accuracy by 1.4%, and reduces temporal consistency by only 2.2% at most.

This study targets 2D articulated human pose estimation (i.e. localisation of body limbs) in stereo videos. Although in recent years depth-based devices (e.g. Microsoft Kinect) have gained popularity, as they perform very well in controlled indoor environments (e.g. living rooms, operating theatres or gyms), they suffer clear problems in outdoor scenarios and, therefore, human pose estimation is still an interesting unsolved problem. The authors propose here a novel approach that is able to localise upper-body keypoints (i.e. shoulders, elbows, and wrists) in temporal sequences of stereo image pairs. The authors' method starts by locating and segmenting people in the image pairs by using disparity and appearance information. Then, a set of candidate body poses is computed for each view independently. Finally, temporal and stereo consistency is applied to estimate a final 2D pose. The authors' validate their model on three challenging datasets: `stereo human pose estimation dataset', `poses in the wild' and `INRIA 3DMovie'. The experimental results show that the authors' model not only establishes new state-of-the-art results on stereo sequences, but also brings improvements in monocular sequences.

The two distinctive visual experiences of binocular display, with and without stereoscopic glasses, have been recently utilized for visual sharing between the colorblind and the normal-vision audiences. However, all existing methods only work for still images, and lack of temporal consistency for video application. In this paper, we propose the first synthesis method for colorblind-sharable videos that possess the temporal consistency for both visual experiences of colorblind and normal-vision, and retains all other crucial characteristics for visual sharing with colorblind. We formulate this challenging multi-constraint problem as a global optimization and minimize an objective function consisting of temporal term, color preservation term, color distinguishability term, and binocular fusibility term. Qualitative and quantitative experiments are conducted to evaluate the effectiveness of the proposed method comparing to existing methods.

This paper reports experimental research aimed to provide seamlessly moving illusory tactile sensations across a large area of the body using a few vibration actuators. The human vibrotactile sensitivity differences among the body sites are calibrated using empirically-measured psychophysical magnitude functions. We present a new phantom sensation rendering method that uses the Gaussian function emphasizing the spatial continuity of perceived movement and the temporal consistency of perceived intensity. We demonstrate that our rendering method outperforms previous methods for phantom sensations via a perceptual experiment. Our method is tailored to eliciting the perception of illusory tactile sensation moving in a long distance, from the thigh to the upper back.

This paper proposes a method for detecting forged objects in videos that include dynamic scenes such as dynamic background or non-stationary scenes. In order to adapt to dynamic scenes, we combine Convolutional Neural Network and Recurrent Neural Network. This enables us to consider spatio-temporal consistency of videos. We also construct new video forgery databases for object modification as well as object removal. Our proposed method using Convolutional Long Short-Term Memory achieved Area-Under-Curve (AUC) 0.977 and Equal-Error-Rate (EER) 0.061 on the object addition database. We also achieved AUC 0.872 and EER 0.219 on the object modification database.

This paper proposes a new video retargeting method that utilizes the cropping and bilinear scaling techniques. Specifically, the cropping positions for a given image are determined by using results of the logo and saliency detection, and the strength of the bilinear scaling is determined depending on the size of cropping. As a result, the proposed method effectively reduces distortion of visually important contents and maintains a temporal consistency. Experimental results showed that the proposed method provided the highest preference compared with the existing methods in the paired-comparison-based user study evaluation.

Supervoxels have been widely used as a preprocessing step to exploit object boundaries to improve the performance of video processing tasks. However, most of the traditional supervoxel algorithms do not perform well in regions with complex textures or weak boundaries. These methods may generate supervoxels with overlapping boundaries. In this paper, we present the novel video supervoxel generation algorithm using partially absorbing random walks to get more accurate supervoxels in these regions. Our spatial–temporal framework is introduced by making full use of the appearance and motion cues, which effectively exploits the temporal consistency in video sequence. Moreover, we build a novel Laplacian optimization structure using two adjacent frames to make our approach more efficient. Experimental results demonstrated that our method achieved better performance than the state-of-the-art supervoxel algorithms.

The foreground extraction of video is derived from frame difference and background subtraction. These methods have always relied on the temporal consistency of successive video frames. If the video consistency is suddenly destroyed, such as an object being occluded, some frames are lost, or the camera is shaken, etc., the results of foreground extraction may be significantly degraded. In this paper, our extends One-Shot Video Object Segmentation by adding two branchs for Positioning foreground targets and propagating foreground semantic information. These two branches are parallel to the existing foreground segmentation branches, which facilitates efficient iterative and fine-tuning of the foreground segmentation branches. This foreground extraction method is different from the previous background subtraction and frame difference methods. Our method does not rely on the time information and only needs to give the manual annotation of one frame, which can extract all the specific foreground in the video sequence. Then, the coloring can get complete foreground information for the separated target. Experiments show that the foreground extraction algorithm applied in this paper has better robustness in dynamic background, camera shake, intermittent motion of objects, low contrast, and so on.

Many Defocus blur estimation methods have been proposed in recent years but, when applied to video sequences in a frame-by-frame manner, they typically exhibit temporal inconsistencies or flickering. This paper presents a temporal coherence scheme that can be coupled to any existing defocus blur estimation for still images, aiming to produce spatiotemporally coherent defocus blur map videos. The proposed method is based on the design of a Kalman Filter which is applied on a patch level. Experimental results show that the proposed method can smooth out undesirable temporal fluctuations whilst still being able to preserve the abrupt local appearance changes due to motion, occlusions or dis-occlusions.

This paper presents a fully automated approach to classifying deformable and non-deformable moving objects in a video surveillance scene. We estimate an object's motion using Marzat optical-flow algorithm. We filter the motion vectors and attempt to find the transformation that represents the correct mapping between the two positions. The Fundamental transformation is estimated using the Normalized Eight-Point Algorithm. We introduce a new type of graph to set the thresholds between deformable and non-deformable motion. Furthermore, we use temporal consistency to classify deformable and non-deformable objects. For experiments, we used a varied corpus of real surveillance videos. Our proposed approach for motion classification achieved a precision rate of 92 percent.

Segmenting moving objects in a video sequence has been a challenging problem and critical to outdoor robotic navigation. While recent literature has laid focus on regularizing object labels over a sequence of frames, exploiting the spatio-temporal features for motion segmentation has been scarce. Particularly in real world dynamic scenes, existing approaches fail to exploit temporal consistency in segmenting moving objects with large camera motion.

Historical monuments are considered as one of the key aspects of our cultural heritage. Unfortunately, due to a variety of factors, the monuments get damaged. The need for preservation of cultural heritage has desiderated research on digitally repairing the photographs of damaged monuments. One may think of digitally undoing the damage to the monuments by inpainting, a process to fill-in missing regions in an image. For images of historic monuments, in particular, there is a consensus to fill-up the defaced regions and cracks so that one can view these in their undamaged form. Thus, we are not talking about image restoration, but about object completion by digitally repairing defaced regions/cracks that the physical objects have. In this chapter, we discuss techniques for automatically detecting the damaged facial regions and cracks in photographs of monuments. Unlike the usual practice of manually selecting the mask for inpainting, the regions to be inpainted are automatically selected and inpainting is done using the existing algorithm. Thus, the process of digital repair using inpainting is completely automated. We also extend our work on crack detection to perform auto-inpainting in videos by making use of scale invariant feature transform (SIFT) and homography. Finally, we provide a temporal consistency measure to quantify the quality of the inpainted video.

Disparity estimation for still images has attracted much interest and acquired many promising results. However, simply applying these methods to produce a disparity sequence may suffer from the undesirable flickering artifacts. These errors not only distinctly decrease the visible quality of the synthesized video, but also significantly reduce the coding efficiency of the disparity sequence. In this paper, a novel temporal consistency enhancement algorithm based on Guided Filter and Temporal Gradient (GFTG) is proposed. The flickering artifacts and noises are effectively removed and the edges of objects are well preserved. Both quantitative and qualitative evaluations show that the spatio-temporal consistency has been highly improved by utilizing our approach.

This paper examines the problem of detecting various types of animals in video sequence taken in the wild. Animal detection is useful in prevention of animal- vehicle accidents and will increase human and wildlife safety. We propose a fully automated method for detecting large animals before they enter the road and warn the driver through audio and visual signals, this also helps in saving crops in farm from animals. The proposed framework aims to automatically extract foreground objects of interest without any user interaction or the use of any training data (i.e., not limited to any particular type of object).To separate foreground and background regions within and across video frames, the proposed method utilizes visual and motion saliency information extracted from the input video. A conditional random field is applied to effectively combine the saliency induced features, which allows us to deal with unknown pose and scale variations of the foreground object (and its articulated parts). Based on the ability to preserve both spatial continuity and temporal consistency in the proposed VOE framework, experiments on a variety of videos verify that our method is able to produce quantitatively and qualitatively satisfactory VOE results. Neural network is used to find the animal type.

To achieve the goal of frontal vehicle detection in night-driving condition, we propose an effective method to detect the red taillights of vehicles. The challenge is that the taillight images captured with automatic exposure typically are overexposed, which makes red color segmentation often erroneous. Instead of customizing the camera hardware to tackle this problem, we combine morphological and logical operations to extract the overexposed region in taillights, which leads to a much more reliable taillight detection scheme. Then, we develop a robust pairing process that clusters two taillight candidates into a pair that represents a vehicle. Several criteria are considered in the pairing process, including the similarities of area, shape, and height of a pair of lights. In addition, we include the temporal consistency criterion; that is, a pair of taillights should be continually detected for a certain duration of time. An energy function is used to aggregate these criteria together. Our experiments show that both the missing and false detection rates are lower than 1.5%.

A robust video classification method based on spatial-temporal low rank representation learning is proposed in this paper. The proposed algorithm utilizes the spatial manifold structure and the temporal continuity of video data. On one hand, the global spatial subspace structure will be exploited by an improved version of low rank representation, which will be more robust to occlusions. On the other hand, the temporal information of the videos will be penalized by column sparse constraint in representation space. Compared with traditional methods, the proposed method in this paper is more suitable to pursue the global structure of the data space while maintain the temporal consistency of the video, and the performance of the proposed method is more robust when the video is contaminated by occlusions and outliers. An efficient solution based on linearized alternating direction method with adaptive penalty (LADMAP) is built for proposed method. Finally, we evaluate the performance of the proposed algorithm on two standard database, i.e., HONDA and YaleB. The experimental results show that our method outperforms several state-of-art algorithms.

We propose a novel approach to the real-time landing site detection and assessment in unconstrained man-made environments using passive sensors. Because this task must be performed in a few seconds or less, existing methods are often limited to simple local intensity and edge variation cues. By contrast, we show how to efficiently take into account the potential sites' global shape, which is a critical cue in man-made scenes. Our method relies on a new segmentation algorithm and shape regularity measure to look for polygonal regions in video sequences. In this way, we enforce both temporal consistency and geometric regularity, resulting in very reliable and consistent detections. We demonstrate our approach for the detection of landable sites such as rural fields, building rooftops and runways from color and infrared monocular sequences significantly outperforming the state-of-the-art.

Curvilinear networks are prevalent in nature and span many different scales, ranging from micron-scale neural structures in the brain to petameter-scale dark-matter arbors binding massive galaxy clusters. Reliably reconstructing them in an automated fashion is of great value in many different scientific domains. However, it remains an open Computer Vision problem. In this thesis we focus on automatically delineating curvilinear tree structures in images of the same object of interest taken at different time instants. Unlike virtually all of the existing methods approaching the task of tree structures delineation we process all the images at once. This is useful in the more ambiguous regions and allows to reason for the tree structure that fits best to all the acquired data. We propose two methods that utilize this principle of temporal consistency to achieve results of higher quality compared to single time instant methods. The first, simpler method starts by building an overcomplete graph representation of the final solution in all time instants while simultaneously obtaining correspondences between image features across time. We then define an objective function with a temporal consistency prior and reconstruct the structures in all images at once by solving a mathematical optimization. The role of the prior is to encourage solutions where for two consecutive time instants corresponding candidate edges are either both retained or both rejected from the final solution. The second multiple time instant method uses the same overcomplete graph principle but handles the temporal consistency in a more robust way. Instead of focusing on the very local consistency of single edges of the overcomplete graph we propose a method for describing topological relationships. This favors solutions whose connectivity is consistent over time. We show that by making the temporal consistency more global we achieve additional robustness to errors in the initial features matching step, which is shared by both the approaches. In the end, this yields superior performance. Furthermore, an added benefit of both our approaches is the ability to automatically detect places where significant changes have occurred over time, which is challenging when considering large amounts of data. We also propose a simple single time instant method for delineating tree structures. It computes a Minimum Spanning Arborescence of an initial overcomplete graph and proceeds to optimally prune spurious branches. This yields results of lower but still competitive quality compared to the mathematical optimization based methods, while keeping low computational complexity. Our methods can applied to both 2D and 3D data. We demonstrate their performance in 3D on microscopy volumes of mouse brain and rat brain. We also test them in 2D on time-lapse images of a growing runner bean and aerial images of a road network.

This paper proposes a novel hierarchical approach to improve the accuracy of the classification of normal-vs-abnormal frames in white-light colonoscopy videos. The existing approaches label each frame independently, without considering the temporal consistency between adjacent frames. Temporal consistency, however, can improve the classification accuracy in the presence of unclear/uncertain images. We propose to leverage temporal consistency between adjacent frames for colonoscopy video frame classification using a novel hierarchical classifier. Comparative experiments with five challenging full colonoscopy videos show that the proposed approach considerably improves the mean class normal/abnormal classification accuracy compared to the approaches where the frames are classified independently.

Depth-image-based rendering (DIBR) is a commonly used method for synthesizing additional views using video-plus-depth (V+D) format. A critical issue with DIBR-based view synthesis is the lack of information behind foreground objects. This lack is manifested as disocclusions, holes, next to the foreground objects in rendered virtual views as a consequence of the virtual camera “seeing” behind the foreground object. The disocclusions are larger in the extrapolation case, i.e. the single camera case. Texture synthesis methods (inpainting methods) aim to fill these disocclusions by producing plausible texture content. However, virtual views inevitably exhibit both spatial and temporal inconsistencies at the filled disocclusion areas, depending on the scene content. In this paper, we propose a layered depth image (LDI) approach that improves the spatio-temporal consistency. In the process of LDI generation, depth information is used to classify the foreground and background in order to form a static scene sprite from a set of neighboring frames. Occlusions in the LDI are then identified and filled using inpainting, such that no disocclusions appear when the LDI data is rendered to a virtual view. In addition to the depth information, optical flow is computed to extract the stationary parts of the scene and to classify the occlusions in the inpainting process. Experimental results demonstrate that spatio-temporal inconsistencies are significantly reduced using the proposed method. Furthermore, subjective and objective qualities are improved compared to state-of-the-art reference methods.

In this study, an efficient, robust algorithm for automatic target detection and tracking is introduced. Procedure starts with a detection phase. Proposed method uses two alternatives for the detection phase, namely maximally stable extremal regions detector and Canny edge detector. After detection, regions of interest are evaluated and eliminated according to their compactness and effective saliency. The detection process is repeated for a predetermined number of pyramid levels where each level processes a downsampled version of input image to achieve scale invariance. Then, temporal consistency for detections from all scales is evaluated and target likelihood map is constructed using kernel density estimation in order to merge all target hypotheses. Finally, outstanding targets are selected from target likelihood map and tracking is achieved by minimizing spatial distance between the selected targets in consecutive frames.

Predicting human attention for video is requires exploiting temporal knowledge included in the video. We propose a novel hierarchical spatial-temporal saliency model for video based on the center-surround framework using both static features and temporal features. Saliency cues are analyzed through a hierarchical segmentation model, and fused across multiple levels, yielding the spatial-temporal saliency map. An adaptive temporal window using motion information is also developed to combine saliency values of consecutive frames in order to keep temporal consistency across frames. Performance evaluation on several popular benchmark datasets validates that our method outperforms existing state-of-the-arts.

We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.

International audience; Extending image processing techniques to videos is a non-trivial task; applying processing independently to each video frame often leads to temporal inconsistencies, and explicitly encoding temporal consistency requires algorithmic changes. We describe a more general approach to temporal consistency. We propose a gradient-domain technique that is blind to the particular image processing algorithm. Our technique takes a series of processed frames that suffers from flickering and generates a temporally-consistent video sequence. The core of our solution is to infer the temporal regularity from the original unprocessed video, and use it as a temporal consistency guide to stabilize the processed sequence. We formally characterize the frequency properties of our technique, and demonstrate , in practice, its ability to stabilize a wide range of popular image processing techniques including enhancement and stylization of color and tone, intrinsic images, and depth estimation.

Robust egomotion estimation is a key prerequisite for making a robot truly autonomous. In previous work, a multimodel extension of random sample consensus (RANSAC) was introduced to deal with environments with rapid changes by incorporating moving object information. A multiscale matching algorithm was also proposed to resolve the issue of imperfect segmentation. In this paper, we present a novel specialization of RANSAC that extends the previous work. A unified framework is introduced to achieve simultaneously egomotion estimation, multiscale segmentation, and moving object detection in the RANSAC paradigm. The motivation of this work is to provide a robust real-time solution to the problem of egomotion estimation, segmentation, and moving object detection in highly dynamic environments. The idea is to augment the discriminative power of spatial and temporal appearances of objects by the spatiotemporal consistency. The objective is twofold. First, split mismerged segments and distinguish nonstationary objects from stationary objects by the spatial consistency. Second, merge oversegmented segments and differentiate moving objects from outlying objects by the temporal consistency. Moving objects of considerably different sizes, from pedestrians to trucks, can be properly segmented and correctly detected. We also show that the performance of egomotion estimation can be further improved by taking into account both stationary and moving object information. Our approach is extensively evaluated on challenging data sets and compared to the state of the art. The experiments also show that our approach serves as a general framework that works well with various planar range data. © 2011 Wiley Periodicals, Inc. © 2011 Wiley Periodicals, Inc.

A general hierarchical particle filtering framework for designing an optimal proposal distribution is proposed. The essential idea is to augment a second filter's estimate into the proposal distribution design. Based on this framework,a novel algorithm for robust and efficient visual tracking is given. The algorithm using hierarchical particle filtering procedures at each level and can efficiently extract high-confidence regions through video frames by exploiting the temporal consistency of region confidences. Comparisons with the mean shift tracker and a generic particle filter shows the advantages and limitations of the new approach.