Search

International audience; When trying to independently apply image-trained algorithms to successive frames in videos, noxious flickering tends to appear. State-of-the-art post-processing techniques that aim at fostering temporal consistency, generate other temporal artifacts and visually alter the style of videos. We propose a postprocessing model, agnostic to the transformation applied to videos (eg style transfer, image manipulation using GANs, etc.), in the form of a recurrent neural network. Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation, as well as a novel style preserving perceptual loss. The former improves long-term temporal consistency learning, while the latter fosters style preservation. We evaluate our model on the DAVIS and this http URL datasets and show that our approach offers state-of-the-art results concerning flicker removal, and better keeps the overall style of the videos than previous approaches.

Measuring video image quality is difficult and very often not precise. There are many factors that can affect the results and their interpretation. The advantages of subjective testing are in obtaining valid results for conventional and compressed television systems and the possibility of evaluating scalar MOS over a wide range of still and motion video image applications. Their disadvantages are in a wide variety of possible methods and tests to be considered, the high number of observers required and in high time demands. There are many objective testing approaches. It can be stated that the algorithms with video image feature analysis correlate better with subjective results than just simple pixel-based methods. A combination of different measurements and features gives the best results and correlation between subjective and objective scores but it is hardly technology independent.

Low-light videos are usually accompanied with acquisition noise, motion blur and some other specific distortions, which makes it hard to compress by the video coding technologies and generates less satisfying compressed videos. In order to enhance both the subjective and objective quality of compressed dark videos, we propose a novel learning based post-processing scheme for the most recent VVC. Specifically, we adopt a multi-scale residue learning structure, named Dark Video Restoration Convolutional Neural Network (DVRCNN), as an additional out-loop post-processing method. To avoid the over-smooth effect by MSE metric, SSIM and texture loss are also added to the final loss function. Luma and chroma components are decomposed then fed to two corresponding models separately. Compared with VVC baseline on the six sequences given in ICME 2020 Grand Challenge, our approach significantly reduces the BD-rate by 36.08% and achieves a fair promotion on both objective and subjective quality, especially for the low bit-rate compressed sequence with severe distortion. Validation results show that the proposed model generates well to continuous scenes and variable bitrates.

The epidemic growth of internet services has created a broad attention on digital content authentication and copyright protection. Heading the issue we are going to propose a DCT based novel approach on invisible video watermarking along with an improved video key-frame extraction algorithm. In this article first the key-frames (video bookmarks) of the video sequence are being extracted. In order to identify the video bookmarks, a novel method is proposed based on boundary luminosity analysis that is proven efficient regarding fast camera movement and moving objects in the video frames. A pair of secret keys will be applied during the watermark embedding phase. Our framework embeds watermarks in only those video frames which have been identified as key-frames of that video stream because, the key-frames are mostly kept as it is in different compression algorithms. The low frequency DCT coefficients of key-frames are chosen for watermarking, that again offers robustness to the compression algorithms. Instead of embedding watermark bits directly, a new method is proposed- scaled average. The watermark is perceptually invisible in the video key-frames according to human visual system. At last uncompressed video is re-reconstructed with all frames of the video in appropriate order of appearance. The watermark extraction is blind i.e., except the keys nothing is needed for watermark extraction.

This paper proposes a pseudo-dolly-in video generation method that reproduces motion parallax by applying image reconstruction processing to multi-view videos. Since dolly-in video is taken by moving a camera forward to reproduce motion parallax, we can present a sense of immersion. However, at a sporting event in a large-scale space, moving a camera is difficult. Our research generates dolly-in video from multi-view images captured by fixed cameras. By applying the Image-Based Modeling technique, dolly-in video can be generated. Unfortunately, the video quality is often damaged by the 3D estimation error. On the other hand, Bullet-Time realizes high-quality video observation. However, moving the virtual-viewpoint from the capturing positions is difficult. To solve these problems, we propose a method to generate a pseudo-dolly-in image by installing 3D estimation and image reconstruction techniques into Bullet-Time and show its effectiveness by applying it to multi-view videos captured at an actual soccer stadium. In the experiment, we compared the proposed method with digital zoom images and with the dolly-in video generated from the Image-Based Modeling and Rendering method., Published in: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) Date of Conference: 9-13 Oct. 2017 Conference Location: Nantes, France, 9780769563275

The depth-image-based rendering is a key technique for 3D video and free viewpoint video synthesis. One of the critical problems in current synthesis methods is that the background (BG) occluded by the foreground objects might be exposed in the new view, and some holes are produced in the synthesized video. However, most of the traditional hole-filling approaches may bring some blurry effect or artifacts in the virtual view. In this paper, a foreground removal approach for hole filling is proposed, in which the foreground objects are removed from both the 2D video and its corresponding depth map, and then a BG video and its depth map are generated before the 3D warping and used to eliminate the holes in the synthesized video. Moreover, a BG extension method is applied in the reference view to prevent the large holes occurring along the border areas in the virtual view. Our analysis and experimental results have indicated that the proposed approach has better performance compared with the other methods in terms of the quality of synthesized video, computational complexity, and running time in multiview synthesis or multiframe synthesis.

The exponential growth in the number of surveillance videos makes the search and retrieval of their contents an extensive, time-consuming, and tedious task. Video abstraction is a general solution to alleviate this problem by generating a short and concise version of the original video. The existing abstraction approaches have commonly relied on global characteristics of visual content and neglected the local details of video frames. This paper presents an enhanced video abstraction approach called Density-based Surveillance video abstraction (DbSva) to generate a static short-length video. The novelty of DbSva is (a) to integrate the advantages of both the global and local features of video contents by fusion and (b) to employ the DENsity-based CLUstEring algorithm (DENCLUE) to significantly improve the quality of abstract videos. Utilizing fusion and the DENCLUE algorithm resulted in the extraction of more informative parts of the videos and increased the robustness of the proposed approach to handle large-scale and noisy videos with no further tuning of the input parameters. A number of qualitative and quantitative experiments support the effectiveness of the proposed approach in generating higher-quality abstract videos compared to the other approaches.

A new image enhancement algorithm employing guided filtering is proposed in this work for enhancement of solar images and videos, so that users can easily figure out important fine structures imbedded in the recorded images/movies for solar observation. The proposed algorithm can efficiently remove image noises, including Gaussian and impulse noises. Meanwhile, it can further highlight fibrous structures on/beyond the solar disk. These fibrous structures can clearly demonstrate the progress of solar flare, prominence coronal mass emission, magnetic field, and so on. The experimental results prove that the proposed algorithm gives significant enhancement of visual quality of solar images beyond original input and several classical image en-hancement algorithms, thus facilitating easier determi-nation of interesting solar burst activities from recorded images/movies.

Video analytics plays an important role in a wide variety of defense-, monitoring- and surveillance-related systems for air and ground environments. In this context, multispectral video processing is attracting increased interest in recent years, due in part to technological advances in video capture. Compared with monochromatic video, multispectral video offers better spectral resolution, and different bands of multispectral video streams can enhance video analytics capabilities in different ways. For example, the infrared bands can provide better separation of shadows from objects, and improved spatial resolution in scenes that are impaired by fog or haze [16].

View synthesis prediction (VSP) is a crucial coding tool for improving compression efficiency in the next generation 3D video systems. However, VSP is susceptible to catastrophic error propagation when multi-view video plus depth (MVD) data are transmitted over lossy networks. This paper aims at accurately modeling the transmission errors propagated in the inter-view direction caused by VSP. Toward this end, we first study how channel errors gradually propagate along the VSP-based inter-view prediction path. Then, a new recursive model is formulated to estimate the expected end-to-end distortion caused by those channel losses. For the proposed model, the compound impact of the transmission distortions of both the texture video and depth map on the quality of the synthetic reference view is mathematically analyzed. Especially, the expected view synthesis distortion due to depth errors is characterized in the frequency domain using a new approach, which combines the energy densities of the reconstructed texture image and the channel errors. The proposed model also explicitly considers the disparity rounding operation invoked for the sub-pixel precision rendering of the synthesized reference view. Experimental results are presented to demonstrate that the proposed analytic model is capable of effectively modeling the channel-induced distortion for MVD-based 3D video transmission.

A new algorithm and its hardware architecture are presented to up-scale high-definition (HD) and full-HD video streams to 4-K ultra-HD video streams in real time. The Lagrange interpolation is employed, as it provides high estimation accuracy and hardware-friendly properties. To enhance the accuracy further, the pixels at the edge regions are specially processed by employing an image-sharpening technique. Experimental results show that the proposed architecture provides the best visual quality at the cost of reasonable hardware resources.

Offline or deferred solutions are frequently employed for high quality and reliable results in current video stabilization. However, neither of these solutions can be used for strict real-time applications. In this paper, we propose a practical and robust algorithm for real-time video stabilization. To achieve this, a novel and efficient motion model based on inter-frame homography estimation is proposed to represent the video motion. An important feature of the proposed motion model is that it updates at each frame input to reduce the accumulation errors caused by parallax or scene changes. We also propose a novel Kalman filter for the motion smoothing and a unique mosaic algorithm for the video completion. The proposed Kalman filter and mosaic algorithm enable the development of a practical real-time video stabilizer that not only produces steady video but also retains the full resolution of the original video. We verify the proposed algorithm through a broad range of video sequences that demonstrate that the proposed algorithm is computationally efficient while being able to robustly stabilize videos with various challenges.

We present a video vectorization method that generates a video in vector representation from an input video in raster representation. A vector-based video representation offers the benefits of vector graphics, such as compactness and scalability. The vector video we generate is represented by a simplified tetrahedral control mesh over the spatial-temporal video volume, with color attributes defined at the mesh vertices. We present novel techniques for simplification and subdivision of a tetrahedral mesh to achieve high simplification ratio while preserving features and ensuring color fidelity. From an input raster video, our method is capable of generating a compact video in vector representation that allows a faithful reconstruction with low reconstruction errors.

Here we present results of research into multi-agent approach to solving the problem of constructing a wireless Ad Hoc network to transmit video stream during optical navigation. When obtaining data from video cameras in real time, there is a problem of realization of effective coordination of video data. To improve the algorithms of capturing an object, it is necessary to improve the architecture of devices that utilize network metadata for the analysis of scenes from video surveillance. As a result of exploring the ways of solving the specified problems, we proposed a method for active coordination of video cameras based on the multi-agent approach (MAC-on-MAA). The application of this method increases efficiency of network cameras through their reconfiguration. We examined the problem of determining an adequate level of illuminance normalization of a video frame based on analysis of the modules throughput of multi-agent system for encoding transmitted video data. It was revealed that the time of analysis of video stream from a network decreases when ignoring the decoding of certain domains of graphic elements in the received video frames. The devised MAC-on-MAA method allows effective updating of routing tables in the reactive protocol AODV. We obtained analytic expressions for the assessment of quantitative characteristics of energy consumption by mobile agents. Protection of communication channels of video surveillance network is improved through the integration of steganocontainers in the free bits of RGB characteristics of video frames pixels.

One of the essential pre-processing steps of semantic video analysis is the video shot boundary detection (SBD). It is the primary step to segment the sequence of video frames into shots. Many SBD systems using supervised learning have been proposed for years; however, the training process still remains its principal limitation. In this paper, a multi-modal visual features-based SBD framework is employed that aims to analyze the behaviors of visual representation in terms of the discontinuity signal. We adopt a candidate segment selection that performs without the threshold calculation but uses the cumulative moving average of the discontinuity signal to identify the position of shot boundaries and neglect the non-boundary video frames. The transition detection is structurally performed to distinguish candidate segment into a cut transition and a gradual transition, including fade in/out and logo occurrence. Experimental results are evaluated using the golf video clips and the TREC2001 documentary video data set. Results show that the proposed SBD framework can achieve good accuracy in both types of video data set compared with other proposed SBD methods.

In free viewpoint video system, color video and the associated depth video are utilized to synthesize arbitrary virtual viewpoint. Hence, high quality of depth video is a necessity for virtual view rendering. However, depth video estimated by depth estimate software is inconsistent and inaccurate which decreases the quality of virtual views. To solve the problem, a depth video processing algorithm is proposed in this paper. Firstly, depth video is divided into five clusters adaptively by Fuzzy C-means clustering method. Meanwhile, the edges of depth video are detected and expanded into 8×8 block-wise. Secondly, for pixels in non-edge regions of depth video, a cluster dependent filtering method is adopted according to the feature of each cluster. Finally, corners in the corresponding texture video are detected. For pixels in edge regions of depth video, a corner-aware filtering method is used. Experimental results show that the proposed algorithm enhances the depth video and significantly improves the quality of the virtual views. The peak signal to noise ratio of virtual views rendered by using the proposed algorithm is 0.43 dB higher than that of the virtual views rendered by using the original depth video. The proposed algorithm also outperforms Martin's algorithm in terms of virtual view rendering performance.

Full-reference (FR) video quality estimators (QEs) resize either the distorted input video or the reference video to compute the quality when these videos have different spatial resolutions. This resizing operation causes several limitations. Multiscale Image Quality Estimator (MIQE) overcomes those limitations for images, but it does not consider the temporal characteristics of video. In this paper, we develop an FR video QE that integrates MIQE with the motion information to estimate the quality of the distorted video without resampling the reference or the test videos. We also perform subjective tests to compare the proposed algorithm with the existing QEs. In these tests, the reference and the input videos are displayed at their native resolutions. The test results show that the proposed algorithm outperforms other QEs when the reference video and the input video have different spatial resolutions. We have also evaluated the performance of the approach using the Scalable Video Database.

Content-based video retrieval and video synopsis are generally considered as two different areas. In this paper, we present an efficient approach for video retrieval based on the perceptual synopsis database of the videos. Video synopsis encapsulates an overview of a shot in a single frame. This is the first time video synopsis is used for video indexing providing the user an intuitive link for accessing actions in the video. We propose an enhanced synopsis called meta synopsis for the video database index, which will contain all essential information for retrieval. Various information such as background of a scene, motion trajectory of the foreground objects, color, texture, and mutual information in the synopsis database will empower us to retrieve relevant video content from huge video databases. Experiments were conducted on the OVP, BBC Motion Gallery, TRECVID data set, and other videos. Instead of using key frames as the query frames, the method accepts any arbitrary query frames. The experimental results illustrate that our proposed method can accurately identify a pertinent video from huge video databases.

With the advance of coding and network technology, the multiview video (MVV) system has received growing attention in the 3-D TV market since it can provide consumers with more realistic scenes from different viewpoints. In the MVV system, each video sequence consists of one color-map video and one depth video to be encoded and the depth-image-based rendering technique is often used to create/synthesize virtual views for different viewpoints. To attain the same quality of virtual views, encoding depth videos in a fast and bitrate-saving manner is crucial. Based on the depth no-synthesis-error model developed by Zhao et al. , this paper proposes a new intra-/inter-prediction scheme first and then a fast quadtree structure determination scheme for encoding depth videos in 3-D High Efficiency Video Coding (3D-HEVC). The first proposed scheme has bitrate-saving merit because for each coding unit, each depth value is modified to approach the predicted intra-/inter-depth value as closely as possible, but without causing any synthesis errors in the resultant virtual views. The second proposed scheme has the merit of low computational cost because it can terminate the quadtree structure determination for coding tree units as early as possible. Based on four typical test video sequences, the empirical results demonstrate that on average, our proposed encoding method for depth videos has 16.3% bitrate and 13.8% encoding-time improvement ratios when compared with the traditional method in 3D-HEVC test model, while preserving the quality of the rendered virtual views. In addition, our proposed method outperforms that of Lee et al. in terms of the required bitrate and encoding time.

Video content stored in Video Event Data Recorders (VEDRs) are used as important evidence when certain events such as vehicle collisions occur. However, with sophisticated video editing software, assailants can easily manipulate video records to their advantage without leaving visible clues. Therefore, the integrity of video content recorded through VEDRs cannot be guaranteed, and the number of related forensic issues increases. Existing video integrity detection methods use the statistical properties of the pixels within each frame of the video. However, these methods require ample time, because they check frames individually. Moreover, the frame can easily be replaced and forged using the appropriate public software. To solve this problem, we propose an integrity checking mechanism using the structure of ordered fields in a video file, because existing video editing software does not allow users to access or modify field structures. In addition, because our proposed method involves checking the header information of video content only once, much less detection time is required compared with existing methods that examine the entire frames. We store an ordered file structure of video content as a signature in the database using a customized automated tool. The signature appears according to the video editing software. Then, the suspected video content is compared to a set of signatures. If the file structure matches with a signature, we recognize a manipulated video file by its corresponding editing software. We tested five types of video editing software that cover 99% of the video editing software market share. Furthermore, we arranged 305,981 saving options for all five video editing suites. As a result, we obtained 100% detection accuracy using stored signatures, without false positives, in a collection of 305,981 video files. The principle of this method can be applied to other video formats.

Modern day datasets continue to increase in both size and diversity. One example of such ‘big data’ is video data. Within the medical arena, more disciplines are using video as a diagnostic tool. Given the large amount of data stored within a video image, it is one of most time consuming types of data to process and analyse. Therefore, it is desirable to have automated techniques to extract, process and analyse data from video images. While many methods have been developed for extracting and processing video data, statistical modelling to analyse the outputted data has rarely been employed. We develop a method to take a video sequence of periodic nature, extract the RGB data and model the changes occurring across the contiguous images. We employ harmonic regression to model periodicity with autoregressive terms accounting for the error process associated with the time series nature of the data. A linear spline is included to account for movement between frames. We apply this model to video sequences of retinal vessel pulsation, which is the pulsatile component of blood flow. Slope and amplitude are calculated for the curves generated from the application of the harmonic model, providing clinical insight into the location of obstruction within the retinal vessels. The method can be applied to individual vessels, or to smaller segments such as 2 × 2 pixels which can then be interpreted easily as a heat map.

Rate control in video compression adjusts the encoding parameters to reach a certain target bitrate for the encoded video. State-of-the-art rate controllers for hybrid video coding typically employ content-dependent video bitrate models and video quality metrics (VQMs). To capture the content characteristics, temporal and spatial video activity measures are determined from the raw video using computationally complex algorithms that require access to the uncompressed source video. In automotive deployments, however, full access to the uncompressed source video and the internal functions of video encoders is typically not possible. As a remedy, in this paper, we present a low-complexity approach to estimate the temporal activity (TA) and spatial activity (SA) measures for videos that are captured by a front-facing camera of a vehicle, based on the context information of the vehicle. To this end, we exploit information about the dynamics of the vehicle and other vehicles in the field-of-view of the front-facing camera. We apply the estimated TA and SA values to a video bitrate model and an objective VQM and use these models to solve the rate control problem to determine the optimal encoding settings for given bitrate constraints. The proposed low-complexity solution offers a similar accuracy in achieving rate constraints and similar perceptual quality characteristics as a solution that uses the computed TA and SA values, with the advantage that no access to the uncompressed source video stream or the internal functions of the video encoder is required.

Convolutional neural networks (CNN) are a special type of deep neural networks (DNN). They have so far been successfully applied to image super-resolution (SR) as well as other image restoration tasks. In this paper, we consider the problem of video super-resolution. We propose a CNN that is trained on both the spatial and the temporal dimensions of videos to enhance their spatial resolution. Consecutive frames are motion compensated and used as input to a CNN that provides super-resolved video frames as output. We investigate different options of combining the video frames within one CNN architecture. While large image databases are available to train deep neural networks, it is more challenging to create a large video database of sufficient quality to train neural nets for video restoration. We show that by using images to pretrain our model, a relatively small video database is sufficient for the training of our model to achieve and even improve upon the current state-of-the-art. We compare our proposed approach to current video as well as image SR algorithms.

In 3-D video (3DV) applications, depth-image-based rendering (DIBR) has been widely employed to synthesize virtual views. However, this approach is performed in a frame-based way, meaning each whole frame is dealt with and the characteristics of different regions in the frame are ignored. As a result, redundant pixels in some regions are abused during the subsequent warping and blending stage. This paper proposes a region-aware 3-D warping approach for DIBR in which warped frames are reasonably divided beforehand so that only the indispensable regions are used. With the proposed scheme, it is possible to avoid noneffective and repeated pixels during the warping stage. In addition, the blending process is also saved. The experimental results show that compared to the state-of-the-art VSRS3.5 and VSRS-1D-fast algorithms, our approach can achieve significant computation savings without sacrificing synthesis quality.

Underexposed video enhancement aims at revealing hidden details that are barely noticeable in LDR video frames with noise. Previous work typically relies on a single heuristic tone mapping curve to expand the dynamic range, which inevitably leads to uneven exposure and visual artifacts. In this paper, we present a novel approach for underexposed video enhancement using an efficient perception-driven progressive fusion. For an input underexposed video, we first remap each video frame using a series of tentative tone mapping curves to generate an multi-exposure image sequence that contains different exposed versions of the original video frame. Guided by some visual perception quality measures encoding the desirable exposed appearance, we locate all the best exposed regions from multi-exposure image sequences and then integrate them into a well-exposed video in a temporally consistent manner. Finally, we further perform an effective texture-preserving spatio-temporal filtering on this well-exposed video to obtain a high-quality noise-free result. Experimental results have shown that the enhanced video exhibits uniform exposure, brings out noticeable details, preserves temporal coherence, and avoids visual artifacts. Besides, we demonstrate applications of our approach to a set of problems including video dehazing, video denoising and HDR video reconstruction.

Color-plus-depth 3D video is booming up with the growing popularity of depth-sensing camera. This imposes new challenges on video resizing since the information embedded in the depth and color video should be taken into consideration simultaneously. In this paper, we propose an energy-constrained video retargeting approach for color-plus-depth 3D video. We first build a 3D video visual attention (3DV-VA) computation model by combining gradient, saliency, motion and depth information. Then, an energy function is generated using spatial, temporal and 3DV-VA constraints. Finally, seam selection and carving operation is conducted to generate a retargeted 3D video. Experimental results on two types of 3D video sequences shows that the proposed approach produces better seam carving results than the related existing methods.

In this paper, we extend the free-energy principle to video quality assessment (VQA) by incorporating with the recent psychophysical study on human visual speed perception (HVSP). A novel video quality metric, namely the free-energy principle inspired video quality metric (FePVQ), is therefore developed and applied to perceptual video coding optimization. The free-energy principle suggests that the human visual system (HVS) can actively predict “orderly” information and avoid “disorderly” information for image perception. Basically, “orderly” is associated with the skeletons and edges of objects, and “disorderly” mostly concerns textures in images. Based on this principle, an image is separated into orderly and disorderly regions, and processed differently in image quality assessment. For videos, visual attention, or fixation, is associated with the objects with significant motion according to HVSP, resulting in a motion strength factor in the FePVQ so that the free-energy principle is extended into spatio-temporal domain for VQA. In addition, we investigate the application of the FePVQ in perceptual rate distortion optimization (RDO). For this purpose, the FePVQ is realized with low computational cost by using the relative total variation model and the block-wise motion vectors of video coding to simulate the free-energy principle and the HVSP, respectively. The experimental results indicate that the proposed FePVQ is highly consistent with the HVS perception. The linear correlation coefficient and Spearman's rank-order correlation coefficient are up to 0.8324 and 0.8281 on the LIVE video database. Better perceptual quality of encoded video sequences is achieved by FePVQ-motivated RDO in video coding.

To manipulate large video contents, effective video indexing and retrieval are required. A large number of video indexing and retrieval algorithms have been presented for frame-wise user query or video content query whereas a relatively few video sequence matching algorithms have been proposed for video sequence query. In this paper, we propose an efficient algorithm that extracts key frames using color histograms and matches the video sequences using edge features. To effectively match video sequences with a low computational load, we make use of the key frames extracted by the cumulative measure and the distance between key frames, and compare two sets of key frames using the modified Hausdorff distance. Experimental results with real sequence show that the proposed video sequence matching algorithm using edge features yields the higher accuracy and performance than conventional methods such as histogram difference, Euclidean metric, Battachaya distance, and directed divergence methods.

This paper investigates a novel learning-based camera motion characterization scheme along with its application to the video stabilization problem. The proposed characterization scheme represents the compressed domain block motion vectors (MVs) using polar angle and magnitude histograms. Discriminative features from these two histograms are extracted and fed to a supervised learning-based hierarchical classifier for recognizing the six camera motion patterns. A comparative analysis with an existing scheme is carried out to support and validate the proposed characterization scheme. The proposed scheme works at the frame level by classifying the inter-frame camera motion patterns. This scheme is extended to classify the video segments and a novel application to video stabilization is investigated. An experimental analysis of a number of test sequences captured using a handheld video camera shows that by characterizing the smooth and jittery motions, selective video stabilization could be carried out only on those video segments that have been degraded. This approach of selective video stabilization saves considerable amount of computational time compared with running the stabilization algorithm on the entire video sequence, as proposed in the literature. The proposed strategy of using a classification scheme prior to applying the video stabilization routine offers a new paradigm to the conventional video stabilization problem. Experimental validation carried out using exhaustive search motion estimation obtained block MVs, and H.264/Advanced Video Coding-obtained MVs shows that by using the idea of selective video stabilization, up to 62% reduction in processing time can be achieved compared with video stabilization approaches wherein the entire video sequence is processed.

A comprehensive scheme of temporal profile of surveillance video is proposed.It provides the sampling selection, analyzes properties of shape and motion of the temporal profile, develop the algorithms to detect dynamic targets, and measure their motion direction.Finally, the visualization of targets with multiple temporal slices with transparency is provided.Further, robust object detection based on background updating, and real time data transmission of is given.A comparison of our method with other video indexing methods is also given. This work converts the surveillance video to a temporal domain image called temporal profile that is scrollable and scalable for quick searching of long surveillance video by human operators. Such a profile is sampled with linear pixel lines located at critical locations in the video frames. It has precise time stamp on the target passing events through those locations in the field of view, shows target shapes for identification, and facilitates the target search in long videos. In this paper, we first study the projection and shape properties of dynamic scenes in the temporal profile so as to set sampling lines. Then, we design methods to capture target motion and preserve target shapes for target recognition in the temporal profile. It also provides the uniformed resolution of large crowds passing through so that it is powerful in target counting and flow measuring. We also align multiple sampling lines to visualize the spatial information missed in a single line temporal profile. Finally, we achieve real time adaptive background removal and robust target extraction to ensure long-term surveillance. Compared to the original video or the shortened video, this temporal profile reduced data by one dimension while keeping the majority of information for further video investigation. As an intermediate indexing image, the profile image can be transmitted via network much faster than video for online video searching task by multiple operators. Because the temporal profile can abstract passing targets with efficient computation, an even more compact digest of the surveillance video can be created.

Reconstructing a full-resolution stereoscopic video from an asymmetric stereoscopic video is a challenging task. The existing approaches require depth information, which imposes an additional challenge in data acquisition. In this paper, we propose a novel scheme that is capable of obtaining super-resolution and depth estimation simultaneously from an asymmetric stereoscopic video. The proposed scheme models the video super-resolution and stereo matching with a unified energy function. Then, we apply an alternating optimization method to minimize this energy function, which can be implemented with a two-step algorithm. In the first step we calculate the initial depth map by using a region-based cooperative optimization technique while considering the temporal consistency in video. In the second step we resolve the super-resolution problem under the guidance of the depth information. It is effective because each step benefits from the additional improvement over the previous step. We iteratively update the two steps until stable depth and super-resolution results are obtained. We have conducted a series of experiments on public stereoscopic video sequences to evaluate the performance of the proposed method. Both objective indexes and subjective visual comparisons verify that the proposed scheme can achieve satisfactory super-resolution results and high-quality depth map simultaneously. In particular, the subjective evaluation experiments on a 3-D monitor show that this scheme outperforms others and achieves the best visual sharpness.

This paper proposes a novel representation method which is called Depth and residual images based representation (DRIBR) for virtual image rendering in 3D video. The proposed DRIBR is composed of three parts: a single view image from one particular viewpoint, its corresponding depth map as well as a residual image. The residual image is pre-calculated from the combination of the above two components plus an image taken from the other viewpoint. The proposed DRIBR achieves low data redundancy, and resolves the occlusion problem which is one of the most annoying artifacts during the rendering of virtual image in traditional Depth image based rendering (DIBR). This paper addresses details of DRIBR and virtual image rendering algorithms based on the proposed format. We test the DRIBR on various stereo datasets and the experimental results demonstrate that our method can achieve better performance on both subjective and objective quality assessment.

Cloud gaming allows games to be rendered on the cloud server and allows the rendered videos to be encoded and streamed in real time to the player’s devices. Compared with other video streaming applications, cloud gaming offers a unique opportunity to enhance the video encoding process by exploiting rendering information. In this paper, we propose two techniques to improve cloud gaming video encoding, aiming at enhancing the perceived video quality and reducing the computational complexity, respectively. First, we develop a rendering-based prioritized encoding technique to improve the perceived game video quality according to network bandwidth constraints. We first propose a technique to generate a macroblock (MB)-level saliency map for every game video frame using rendering information. Furthermore, based on such a saliency map, a prioritized rate allocation scheme is proposed to dynamically adjust the value of quantization parameter of each MB. The experimental results indicate that the perceptual quality can be greatly improved using the proposed technique. We also develop a rendering-based encoding acceleration technique that utilizes rendering information to reduce the computational complexity of video encoding. This technique mainly consists of two parts. First, we propose a method to directly calculate the motion vectors (MVs) without employing the compute intensive motion search procedure. Second, based on the computed MVs, we propose a fast mode selection algorithm to reduce the number of candidate modes of each MB. The experimental results show that the proposed technique can achieve more than 42% saving in encoding time with very limited degradation in video quality.

The current research concerns the problem of video stabilization “in a point”, which aims to stabilize all video frames according to one chosen reference frame to produce a new video, as by a static camera. Similar task importance relates providing static background in the video sequence that can be usable for correct measurements in the frames when studying dynamic objects in the video. For this aim we propose an efficient combined approach, called “3×3OF9×9”. It fuses our the previous development for fast and rigid 2D video stabilization [2] with the well-known Optical Flow approach, applied by parts via Otsu segmentation, for eliminating the influence of moving objects in the video. The obtained results are compared with those, produced by the commercial software Warp Stabilizer of Adobe-After-Effects CS6.

The Visual Computer, 22 (9-11), ISSN:0178-2789, ISSN:1432-2315

Currently, there is a rapid development in the techniques of the automated image based modelling (IBM), especially in advanced structure-from-motion (SFM) and dense image matching methods, and camera technology. One possibility is to use video imaging to create 3D reality based models of cultural heritage architectures and monuments. Practically, video imaging is much easier to apply when compared to still image shooting in IBM techniques because the latter needs a thorough planning and proficiency. However, one is faced with mainly three problems when video image sequences are used for highly detailed modelling and dimensional survey of cultural heritage objects. These problems are: the low resolution of video images, the need to process a large number of short baseline video images and blur effects due to camera shake on a significant number of images. In this research, the feasibility of using video images for efficient 3D modelling is investigated. A method is developed to find the minimal significant number of video images in terms of object coverage and blur effect. This reduction in video images is convenient to decrease the processing time and to create a reliable textured 3D model compared with models produced by still imaging. Two experiments for modelling a building and a monument are tested using a video image resolution of 1920×1080 pixels. Internal and external validations of the produced models are applied to find out the final predicted accuracy and the model level of details. Related to the object complexity and video imaging resolution, the tests show an achievable average accuracy between 1 – 5 cm when using video imaging, which is suitable for visualization, virtual museums and low detailed documentation.

This chapter presents an overview of the acquisition, processing, and rendering pipeline in a multiview video system, targeting the unique feature of rendering virtual views, not captured by the input camera feeds. This is called Virtual View Synthesis and supports Free Navigation similar to The Matrix bullet time effect that was popularized in the late 1990s. A substantial part of the chapter is devoted in explaining how to estimate the respective camera parameters and their relative positions in the system, as well as how to estimate/measure depth, which is an essential information in order to obtain smooth virtual view transitions with Depth Image-Based Rendering.

We formulate a just noticeable rendering distortion (JNRD) model.Spatial and temporal correlation of depth video is enhanced based on JNRD model.Bit rate of the depth video is greatly saved while quality of virtual view is maintained. Spatial and temporal inconsistency of depth video deteriorates encoding efficiency in three dimensional video systems. A depth video processing algorithm based on human perception is presented. Firstly, a just noticeable rendering distortion (JNRD) model is formulated by combining the analyses of the influence of depth distortion on virtual view rendering with human visual perception characteristics. Then, depth video is processed based on the JNRD model from two aspects, spatial and temporal correlation enhancement. During the process of spatial correlation enhancement, depth video is segmented into edge, foreground, and background regions, and smoothened by Gaussian and mean filters. The operations of the temporal correlation enhancement include temporal-spatial transpose (TST), temporal smoothing filter and inverse TST. Finally, encoding and virtual view rendering experiments are conducted to evaluate the proposed algorithm. Experimental results show that the proposed algorithm can greatly reduce the bit rate while it maintains the quality of virtual view.

Nowadays, broadcasters deliver ultra-high resolution video to their consumers. This live video is sent to a set-top box for display on a television. However, if one or more users in the home want to view the same video on their personal mobile devices with a lower display resolution and limited processing power, decoding the original ultra-high resolution video would result in stuttering and quickly drain the battery life on these devices. To enable a satisfactory consumer experience, the resolution of the video stream should be adapted to the target mobile device at the set-top box. The aim of this paper is to investigate the performance of different machine learning strategies to arbitrary downsize video pre-encoded with the high efficiency video coding standard (HEVC). These machine learning techniques exploit correlation between input and output coding information to predict the splitting behavior of HEVC coding units. Several machine learning algorithms are optimized. Additionally, both online and offline training strategies are tested. Of the tested algorithms, online-trained random forests achieve the best compression-efficiency with a bit rate increase of 5.4% and an average complexity reduction of 70%1.

Mobile video quality assessment plays an essential role in multimedia systems and services. In the case of scalable video coding, which enables dynamic adaptation based on terminal capabilities and heterogeneous network, variable resolution is one of the most prominent types of video distortions. In this paper, we propose a new hybrid spatial and temporal distortion metric for evaluating video streaming quality with variable spatio-temporal resolution. The key idea is to project video sequence into feature domain and calculate the distortion of content information from the projected principal component matrix and its eigenvectors. This metric can measures the degree of content information degradation especially in spatio-temporal resolution scalable video. The performance of the proposed metric is evaluated and compared to some state-of-the-art quality evaluation metrics in the literature. Our results show that the proposed metric achieves good correlations with the subjective evaluations of the EPFL scale video database.

Advanced technologies in consumer electronics products have enabled individual users to record, share, and view videos on mobile devices. With the volume of videos increasing tremendously on the Internet, fast and accurate video search has attracted much research attention. A good similarity measure is a key component in a video retrieval system. Most of the existing solutions only rely on either the low-level visual features or the surrounding textual annotations. Those approaches often suffer from low recall as they are highly susceptible to changes in viewpoint, illumination, and noisy tags. By leveraging geo- metadata , more reliable and precise search results can be obtained. However, two issues remain challenging: (1) how to quantify the spatial relevance of videos with the visual similarity to generate a pertinent ranking of results according to users’ needs, and (2) how to design a compact video representation that supports efficient indexing for fast video retrieval. In this study, we propose a novel video description which consists of (a) determining the geographic coverage of a video based on the camera’s field-of-view and a pre-constructed geo-codebook, and (b) fusing video spatial relevance and region-aware visual similarities to achieve a robust video similarity measure. Toward a better encoding of a video’s geo-coverage, we construct a geo-codebook by semantically segmenting a map into a collection of coherent regions. To evaluate the proposed technique we developed a video retrieval prototype. Experiments show that our proposed method improves the mean average precision by $4.6\% \sim 10.5\% $ , compared with existing approaches.

The data is exploding day by day in digital technology. Now a day’s multimedia data is also handled by the database, multimedia data contains data like images, text and video. The video processing plays a tremendous role in the multimedia but all the videos are not same, it can exists number of settings and different number of formats. By This video processing system the video is processed for enhancement, analysis, dividing the channels and binarization by using different image processing techniques. In this system different color system like YCBR, HSL, and RGB color systems are considered for processing any type of video. For this system, the input video can be from a stored file or continuous stream of video sequences from the web camera (or) any type of camera by this video processing system we can improve the quality of the video and we can also apply some special effects to the video by applying various image processing techniques and filters. The enhancement techniques considered in his system are filtering with correlation and convolution, adaptive smoothing, conservative smoothing and median filtering. The analysis techniques like edge detection, histogram and statistical analysis are considered for this system. Binarization methods implemented in this system are Custom Threshold, Order Dither. The Color filters like converting RGB to Grayscale, Grayscale to RGB ,Sepia, invert, rotate, Custom Color filter, Euclidean color filter, channel filter, red, green, blue, cyan, magenta and yellow, they are so many other filters are also implemented in this system.

Many compelling video processing effects can be achieved if per-pixel depth information and 3D camera calibrations are known. However, the success of such methods is highly dependent on the accuracy of this "scene-space" information. We present a novel, sampling-based framework for processing video that enables high-quality scene-space video effects in the presence of inevitable errors in depth and camera pose estimation. Instead of trying to improve the explicit 3D scene representation, the key idea of our method is to exploit the high redundancy of approximate scene information that arises due to most scene points being visible multiple times across many frames of video. Based on this observation, we propose a novel pixel gathering and filtering approach. The gathering step is general and collects pixel samples in scene-space, while the filtering step is application-specific and computes a desired output video from the gathered sample sets. Our approach is easily parallelizable and has been implemented on GPU, allowing us to take full advantage of large volumes of video data and facilitating practical runtimes on HD video using a standard desktop computer. Our generic scene-space formulation is able to comprehensively describe a multitude of video processing applications such as denoising, deblurring, super resolution, object removal, computational shutter functions, and other scene-space camera effects. We present results for various casually captured, hand-held, moving, compressed, monocular videos depicting challenging scenes recorded in uncontrolled environments.

In this paper, we propose a novel 2D-to-3D video conversion method for 3D entertainment applications. 3D entertainment is getting more and more popular and can be found in many contexts, such as TV and home gaming equipment. 3D image sensors are a new method to produce stereoscopic video content conveniently and at a low cost, and can thus meet the urgent demand for 3D videos in the 3D entertaiment market. Generally, 2D image sensor and 2D-to-3D conversion chip can compose a 3D image sensor. Our study presents a novel 2D-to-3D video conversion algorithm which can be adopted in a 3D image sensor. In our algorithm, a depth map is generated by combining global depth gradient and local depth refinement for each frame of 2D video input. Global depth gradient is computed according to image type while local depth refinement is related to color information. As input 2D video content consists of a number of video shots, the proposed algorithm reuses the global depth gradient of frames within the same video shot to generate time-coherent depth maps. The experimental results prove that this novel method can adapt to different image types, reduce computational complexity and improve the temporal smoothness of generated 3D video.

Since rapid growth of IT technologies, the use of multimedia data such as image and videos are explosively increasing. It is an important aspect of not only for users but also researchers. Duplicate images and videos are rapidly increasing and it causes difficulties in retrieval and management as well. It also causes copyright problems. In this paper, we discus prior duplicate video detection techniques and overcome previous research problems using block histogram and dynamic matching approach duplicate video detection method. We improved excessive abstract of previous block mean-value based feature extraction method to be robust in various video transformations. Also, we created feature vector of timely histogram by unit of blocks to reflect video features. We proposed dynamic matching algorithm to match videos which is suitable for large-scale video data. To evaluate our proposal, we used VIREO video datasets which is provided by Hong Kong City University and Carnegie Mellon University and MUSCLE-VCD-2007 dataset which is provided by INRIA. Our method showed 90 % of accuracy on duplicate video detection. Our proposed method showed robustness especially in various video transformations. Also, we tested video clustering test to prove our method and dynamic matching method showed 5 times fast compare to existing method which is suitable for real-time and large-scale video detection process.