Search

International audience; When trying to independently apply image-trained algorithms to successive frames in videos, noxious flickering tends to appear. State-of-the-art post-processing techniques that aim at fostering temporal consistency, generate other temporal artifacts and visually alter the style of videos. We propose a postprocessing model, agnostic to the transformation applied to videos (eg style transfer, image manipulation using GANs, etc.), in the form of a recurrent neural network. Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation, as well as a novel style preserving perceptual loss. The former improves long-term temporal consistency learning, while the latter fosters style preservation. We evaluate our model on the DAVIS and this http URL datasets and show that our approach offers state-of-the-art results concerning flicker removal, and better keeps the overall style of the videos than previous approaches.

Low-light videos are usually accompanied with acquisition noise, motion blur and some other specific distortions, which makes it hard to compress by the video coding technologies and generates less satisfying compressed videos. In order to enhance both the subjective and objective quality of compressed dark videos, we propose a novel learning based post-processing scheme for the most recent VVC. Specifically, we adopt a multi-scale residue learning structure, named Dark Video Restoration Convolutional Neural Network (DVRCNN), as an additional out-loop post-processing method. To avoid the over-smooth effect by MSE metric, SSIM and texture loss are also added to the final loss function. Luma and chroma components are decomposed then fed to two corresponding models separately. Compared with VVC baseline on the six sequences given in ICME 2020 Grand Challenge, our approach significantly reduces the BD-rate by 36.08% and achieves a fair promotion on both objective and subjective quality, especially for the low bit-rate compressed sequence with severe distortion. Validation results show that the proposed model generates well to continuous scenes and variable bitrates.

The illegal distribution of a digital movie is a common and significant threat to the film industry. With the advent of high-speed broadband Internet access, a pirated copy of a digital video can now be easily distributed to a global audience. A possible means of limiting this type of digital theft is digital video watermarking whereby additional information, called a watermark, is embedded in the host video. This watermark can be extracted at the decoder and used to determine whether the video content is watermarked. This paper presents a review of the digital video watermarking techniques in which their applications, challenges, and important properties are discussed, and categorizes them based on the domain in which they embed the watermark. It then provides an overview of a few emerging innovative solutions using watermarks. Protecting a 3D video by watermarking is an emerging area of research. The relevant 3D video watermarking techniques in the literature are classified based on the image-based representations of a 3D video in stereoscopic, depth-image-based rendering, and multi-view video watermarking. We discuss each technique, and then present a survey of the literature. Finally, we provide a summary of this paper and propose some future research directions.

The epidemic growth of internet services has created a broad attention on digital content authentication and copyright protection. Heading the issue we are going to propose a DCT based novel approach on invisible video watermarking along with an improved video key-frame extraction algorithm. In this article first the key-frames (video bookmarks) of the video sequence are being extracted. In order to identify the video bookmarks, a novel method is proposed based on boundary luminosity analysis that is proven efficient regarding fast camera movement and moving objects in the video frames. A pair of secret keys will be applied during the watermark embedding phase. Our framework embeds watermarks in only those video frames which have been identified as key-frames of that video stream because, the key-frames are mostly kept as it is in different compression algorithms. The low frequency DCT coefficients of key-frames are chosen for watermarking, that again offers robustness to the compression algorithms. Instead of embedding watermark bits directly, a new method is proposed- scaled average. The watermark is perceptually invisible in the video key-frames according to human visual system. At last uncompressed video is re-reconstructed with all frames of the video in appropriate order of appearance. The watermark extraction is blind i.e., except the keys nothing is needed for watermark extraction.

The depth-image-based rendering is a key technique for 3D video and free viewpoint video synthesis. One of the critical problems in current synthesis methods is that the background (BG) occluded by the foreground objects might be exposed in the new view, and some holes are produced in the synthesized video. However, most of the traditional hole-filling approaches may bring some blurry effect or artifacts in the virtual view. In this paper, a foreground removal approach for hole filling is proposed, in which the foreground objects are removed from both the 2D video and its corresponding depth map, and then a BG video and its depth map are generated before the 3D warping and used to eliminate the holes in the synthesized video. Moreover, a BG extension method is applied in the reference view to prevent the large holes occurring along the border areas in the virtual view. Our analysis and experimental results have indicated that the proposed approach has better performance compared with the other methods in terms of the quality of synthesized video, computational complexity, and running time in multiview synthesis or multiframe synthesis.

Recent works on video captioning mainly learn the map from low-level visual features to language description directly without explicitly representing the high-level semantic video concepts (e.g. objects, actions in the annotated sentences). To bridge the semantic gap, in this paper, addressing it, we propose a novel video attribute representation learning algorithm for video concept understanding and utilize the learned explicit video attribute representation to improve video captioning performance. To achieve it, firstly, inspired by the success of spectrogram in audio processing, a novel mid-level video representation named “video response map” (VRM) is proposed, by which the frame sequence could be represented by a single image representation. Therefore, the video attributes representation learning could be converted to a well-studied multi-label image classification problem. Then in the captions prediction step, the learned video attributes feature is integrated with the single frame feature to improve previous sequence-to-sequence language generation model by adjusting the LSTM (Long-Short Term Memory) input units. The proposed video captioning framework could both handle variable frame inputs and utilize high-level semantic video attribute features. Experimental results on video captioning tasks show that the proposed method, utilizing only RGB frames as input without extra video or text training data, could achieve competitive performance with state-of-the-art methods. Furthermore, the extensive experimental evaluations on the UCF-101 action classification benchmark well demonstrate the representation capability of the proposed VRM.

The exponential growth in the number of surveillance videos makes the search and retrieval of their contents an extensive, time-consuming, and tedious task. Video abstraction is a general solution to alleviate this problem by generating a short and concise version of the original video. The existing abstraction approaches have commonly relied on global characteristics of visual content and neglected the local details of video frames. This paper presents an enhanced video abstraction approach called Density-based Surveillance video abstraction (DbSva) to generate a static short-length video. The novelty of DbSva is (a) to integrate the advantages of both the global and local features of video contents by fusion and (b) to employ the DENsity-based CLUstEring algorithm (DENCLUE) to significantly improve the quality of abstract videos. Utilizing fusion and the DENCLUE algorithm resulted in the extraction of more informative parts of the videos and increased the robustness of the proposed approach to handle large-scale and noisy videos with no further tuning of the input parameters. A number of qualitative and quantitative experiments support the effectiveness of the proposed approach in generating higher-quality abstract videos compared to the other approaches.

A new image enhancement algorithm employing guided filtering is proposed in this work for enhancement of solar images and videos, so that users can easily figure out important fine structures imbedded in the recorded images/movies for solar observation. The proposed algorithm can efficiently remove image noises, including Gaussian and impulse noises. Meanwhile, it can further highlight fibrous structures on/beyond the solar disk. These fibrous structures can clearly demonstrate the progress of solar flare, prominence coronal mass emission, magnetic field, and so on. The experimental results prove that the proposed algorithm gives significant enhancement of visual quality of solar images beyond original input and several classical image en-hancement algorithms, thus facilitating easier determi-nation of interesting solar burst activities from recorded images/movies.

With the popularity of high-efficiency video coding (HEVC) standard, a video server usually transcodes a video stream to HEVC for its higher compression ratio. In this paper, a fast H.264/advanced video coding (AVC) to HEVC transcoding method is proposed. In the HEVC encoding procedure, a coding unit (CU), which is a motion-homogeneous block, is first checked based on the analysis of the decoded information from H.264/AVC bit stream. Then, for motion-homogeneous blocks, CU depth and the corresponding prediction unit (PU) mode's early termination strategies are proposed based on the CU size and corresponding prior statistical knowledge. For non-motion-homogeneous blocks, a corresponding PU mode's early termination strategy is also proposed. Experimental results demonstrate the effectiveness of the proposed method.

Video analytics plays an important role in a wide variety of defense-, monitoring- and surveillance-related systems for air and ground environments. In this context, multispectral video processing is attracting increased interest in recent years, due in part to technological advances in video capture. Compared with monochromatic video, multispectral video offers better spectral resolution, and different bands of multispectral video streams can enhance video analytics capabilities in different ways. For example, the infrared bands can provide better separation of shadows from objects, and improved spatial resolution in scenes that are impaired by fog or haze [16].

View synthesis prediction (VSP) is a crucial coding tool for improving compression efficiency in the next generation 3D video systems. However, VSP is susceptible to catastrophic error propagation when multi-view video plus depth (MVD) data are transmitted over lossy networks. This paper aims at accurately modeling the transmission errors propagated in the inter-view direction caused by VSP. Toward this end, we first study how channel errors gradually propagate along the VSP-based inter-view prediction path. Then, a new recursive model is formulated to estimate the expected end-to-end distortion caused by those channel losses. For the proposed model, the compound impact of the transmission distortions of both the texture video and depth map on the quality of the synthetic reference view is mathematically analyzed. Especially, the expected view synthesis distortion due to depth errors is characterized in the frequency domain using a new approach, which combines the energy densities of the reconstructed texture image and the channel errors. The proposed model also explicitly considers the disparity rounding operation invoked for the sub-pixel precision rendering of the synthesized reference view. Experimental results are presented to demonstrate that the proposed analytic model is capable of effectively modeling the channel-induced distortion for MVD-based 3D video transmission.

Soft errors resulting from encoding video sequences on unreliable hardware can create significant artifacts in decoded video sequences, contributing to extreme video quality degradation. Modern systems are required to operate under increasingly challenging constraints, including smaller feature sizes and lower operating voltage, increasing the likelihood of soft errors in the video encoding hardware. These conditions are of particular concern for energy-limited battery-operated systems since they may be required to operate in nonideal environments and/or continue operating with a practically depleted energy source. The proposed parallel independent signature processing design performs error detection and mitigation in video encoding hardware, enabling a graceful degradation of quality when encoding by using unreliable hardware. The effects of soft errors are minimized by preventing the error propagation normally associated with errors in encoded video sequences. This allows for the recovery of quality when errors are present in the video encoding system. Conventional video encoding techniques are designed to handle worst case error rates by increasing gate sizes and/or increasing the operating voltage of the system. Such designs have error-rate limits, and when these limits are reached, the systems tend to fail catastrophically, resulting in an unrecoverable signal. The proposed design allows for single upset events to translate into single transient artifacts in a decoded video sequence.

A new algorithm and its hardware architecture are presented to up-scale high-definition (HD) and full-HD video streams to 4-K ultra-HD video streams in real time. The Lagrange interpolation is employed, as it provides high estimation accuracy and hardware-friendly properties. To enhance the accuracy further, the pixels at the edge regions are specially processed by employing an image-sharpening technique. Experimental results show that the proposed architecture provides the best visual quality at the cost of reasonable hardware resources.

Offline or deferred solutions are frequently employed for high quality and reliable results in current video stabilization. However, neither of these solutions can be used for strict real-time applications. In this paper, we propose a practical and robust algorithm for real-time video stabilization. To achieve this, a novel and efficient motion model based on inter-frame homography estimation is proposed to represent the video motion. An important feature of the proposed motion model is that it updates at each frame input to reduce the accumulation errors caused by parallax or scene changes. We also propose a novel Kalman filter for the motion smoothing and a unique mosaic algorithm for the video completion. The proposed Kalman filter and mosaic algorithm enable the development of a practical real-time video stabilizer that not only produces steady video but also retains the full resolution of the original video. We verify the proposed algorithm through a broad range of video sequences that demonstrate that the proposed algorithm is computationally efficient while being able to robustly stabilize videos with various challenges.

We present a video vectorization method that generates a video in vector representation from an input video in raster representation. A vector-based video representation offers the benefits of vector graphics, such as compactness and scalability. The vector video we generate is represented by a simplified tetrahedral control mesh over the spatial-temporal video volume, with color attributes defined at the mesh vertices. We present novel techniques for simplification and subdivision of a tetrahedral mesh to achieve high simplification ratio while preserving features and ensuring color fidelity. From an input raster video, our method is capable of generating a compact video in vector representation that allows a faithful reconstruction with low reconstruction errors.

Here we present results of research into multi-agent approach to solving the problem of constructing a wireless Ad Hoc network to transmit video stream during optical navigation. When obtaining data from video cameras in real time, there is a problem of realization of effective coordination of video data. To improve the algorithms of capturing an object, it is necessary to improve the architecture of devices that utilize network metadata for the analysis of scenes from video surveillance. As a result of exploring the ways of solving the specified problems, we proposed a method for active coordination of video cameras based on the multi-agent approach (MAC-on-MAA). The application of this method increases efficiency of network cameras through their reconfiguration. We examined the problem of determining an adequate level of illuminance normalization of a video frame based on analysis of the modules throughput of multi-agent system for encoding transmitted video data. It was revealed that the time of analysis of video stream from a network decreases when ignoring the decoding of certain domains of graphic elements in the received video frames. The devised MAC-on-MAA method allows effective updating of routing tables in the reactive protocol AODV. We obtained analytic expressions for the assessment of quantitative characteristics of energy consumption by mobile agents. Protection of communication channels of video surveillance network is improved through the integration of steganocontainers in the free bits of RGB characteristics of video frames pixels.

One of the essential pre-processing steps of semantic video analysis is the video shot boundary detection (SBD). It is the primary step to segment the sequence of video frames into shots. Many SBD systems using supervised learning have been proposed for years; however, the training process still remains its principal limitation. In this paper, a multi-modal visual features-based SBD framework is employed that aims to analyze the behaviors of visual representation in terms of the discontinuity signal. We adopt a candidate segment selection that performs without the threshold calculation but uses the cumulative moving average of the discontinuity signal to identify the position of shot boundaries and neglect the non-boundary video frames. The transition detection is structurally performed to distinguish candidate segment into a cut transition and a gradual transition, including fade in/out and logo occurrence. Experimental results are evaluated using the golf video clips and the TREC2001 documentary video data set. Results show that the proposed SBD framework can achieve good accuracy in both types of video data set compared with other proposed SBD methods.

Multi-version video-on-demand (VoD) providers either store multiple versions of the same video or transcode video to multiple versions in real time to offer multiple-bitrate streaming services to heterogeneous clients. However, this could incur tremendous storage cost or transcoding computation cost. There have been some works regarding trading off between transcoding and storing whole videos, but they did not take into account video segmentation and internal popularity. As a result, they were not cost-efficient. This paper introduces video segmentation and proposes a segment-based storage and transcoding trade-off strategy for multi-version VoD systems in the cloud. First, we split each video into multiple segments depending on the video internal popularity. Second, we describe the transcoding relationships among versions using a transcoding weighted graph, which can be used to calculate the version-aware transcoding cost from one version to another. Third, we take the video segmentation, version-aware transcoding weighted graph, and video internal popularity into account to propose a storage and transcoding trade-off strategy, which stores multiple versions of popular segments and transcodes unpopular segments. We then formulate it as an optimization problem and present a heuristic divide-and-conquer algorithm to get an approximate optimal solution. Finally, we conduct extensive simulations to evaluate the solution; the results show that it can significantly lower the storage and transcoding cost of multi-version VoD systems.

Video streaming over Internet has been gaining momentum and several quality adaptation schemes have been reported to improve quality of the streamed videos. Most of these schemes focus on adjusting the video encoding rate to match certain network conditions. This paper presents a new quality adaptation algorithm for real-time video streaming over Internet. The proposed algorithm is based upon simultaneous adaptation of multiple key parameters such as video frame rate, resolution, and frame quality to achieve the best possible video quality and minimize possibility of service interruption in lossy networks. Furthermore, the current network status and motion of the streamed video sequence have also been taken into account throughout the adaptation process. A video conferencing test-bed which incorporates the Adobe Flash Real-Time Media Flow Protocol (RTMFP) is built and utilized to investigate the effect of various combinations of the video parameters under study on the quality of sample video clips streamed at slow, medium and fast motions. A quality-adaptation algorithm based on experimental investigations is then developed and its performance is assessed using both subjective and objective evaluations. The obtained results and observations demonstrate superior performance of the developed adaptation algorithm as compared to equivalent algorithms with fixed video quality settings at similar test conditions.

In free viewpoint video system, color video and the associated depth video are utilized to synthesize arbitrary virtual viewpoint. Hence, high quality of depth video is a necessity for virtual view rendering. However, depth video estimated by depth estimate software is inconsistent and inaccurate which decreases the quality of virtual views. To solve the problem, a depth video processing algorithm is proposed in this paper. Firstly, depth video is divided into five clusters adaptively by Fuzzy C-means clustering method. Meanwhile, the edges of depth video are detected and expanded into 8×8 block-wise. Secondly, for pixels in non-edge regions of depth video, a cluster dependent filtering method is adopted according to the feature of each cluster. Finally, corners in the corresponding texture video are detected. For pixels in edge regions of depth video, a corner-aware filtering method is used. Experimental results show that the proposed algorithm enhances the depth video and significantly improves the quality of the virtual views. The peak signal to noise ratio of virtual views rendered by using the proposed algorithm is 0.43 dB higher than that of the virtual views rendered by using the original depth video. The proposed algorithm also outperforms Martin's algorithm in terms of virtual view rendering performance.

Full-reference (FR) video quality estimators (QEs) resize either the distorted input video or the reference video to compute the quality when these videos have different spatial resolutions. This resizing operation causes several limitations. Multiscale Image Quality Estimator (MIQE) overcomes those limitations for images, but it does not consider the temporal characteristics of video. In this paper, we develop an FR video QE that integrates MIQE with the motion information to estimate the quality of the distorted video without resampling the reference or the test videos. We also perform subjective tests to compare the proposed algorithm with the existing QEs. In these tests, the reference and the input videos are displayed at their native resolutions. The test results show that the proposed algorithm outperforms other QEs when the reference video and the input video have different spatial resolutions. We have also evaluated the performance of the approach using the Scalable Video Database.

Even though video technologies have continuously advanced, playing a video still requires full length of the videos playtime, high volume of data traffic, and appropriate sound play devices. To overcome these limitations, this paper proposes a method to serve a video in an image carousel format. By providing multiple key frame images in carousel form with full script of the video, the proposed image carousel system transforms any playable video content into a readable form so that users can read the content rather than watch. Reading a video make consumers be freed from sound restrictions, and increase the potential and efficiency in the video linkage and usage in other services. In addition, this approach saves both playtime and data traffic. When a video encoded in H.264 codec with 720p resolution is transformed into the proposed image carousel form, in average 90---95 % of the data traffic were saved.

Content-based video retrieval and video synopsis are generally considered as two different areas. In this paper, we present an efficient approach for video retrieval based on the perceptual synopsis database of the videos. Video synopsis encapsulates an overview of a shot in a single frame. This is the first time video synopsis is used for video indexing providing the user an intuitive link for accessing actions in the video. We propose an enhanced synopsis called meta synopsis for the video database index, which will contain all essential information for retrieval. Various information such as background of a scene, motion trajectory of the foreground objects, color, texture, and mutual information in the synopsis database will empower us to retrieve relevant video content from huge video databases. Experiments were conducted on the OVP, BBC Motion Gallery, TRECVID data set, and other videos. Instead of using key frames as the query frames, the method accepts any arbitrary query frames. The experimental results illustrate that our proposed method can accurately identify a pertinent video from huge video databases.

With the advance of coding and network technology, the multiview video (MVV) system has received growing attention in the 3-D TV market since it can provide consumers with more realistic scenes from different viewpoints. In the MVV system, each video sequence consists of one color-map video and one depth video to be encoded and the depth-image-based rendering technique is often used to create/synthesize virtual views for different viewpoints. To attain the same quality of virtual views, encoding depth videos in a fast and bitrate-saving manner is crucial. Based on the depth no-synthesis-error model developed by Zhao et al. , this paper proposes a new intra-/inter-prediction scheme first and then a fast quadtree structure determination scheme for encoding depth videos in 3-D High Efficiency Video Coding (3D-HEVC). The first proposed scheme has bitrate-saving merit because for each coding unit, each depth value is modified to approach the predicted intra-/inter-depth value as closely as possible, but without causing any synthesis errors in the resultant virtual views. The second proposed scheme has the merit of low computational cost because it can terminate the quadtree structure determination for coding tree units as early as possible. Based on four typical test video sequences, the empirical results demonstrate that on average, our proposed encoding method for depth videos has 16.3% bitrate and 13.8% encoding-time improvement ratios when compared with the traditional method in 3D-HEVC test model, while preserving the quality of the rendered virtual views. In addition, our proposed method outperforms that of Lee et al. in terms of the required bitrate and encoding time.

Video content stored in Video Event Data Recorders (VEDRs) are used as important evidence when certain events such as vehicle collisions occur. However, with sophisticated video editing software, assailants can easily manipulate video records to their advantage without leaving visible clues. Therefore, the integrity of video content recorded through VEDRs cannot be guaranteed, and the number of related forensic issues increases. Existing video integrity detection methods use the statistical properties of the pixels within each frame of the video. However, these methods require ample time, because they check frames individually. Moreover, the frame can easily be replaced and forged using the appropriate public software. To solve this problem, we propose an integrity checking mechanism using the structure of ordered fields in a video file, because existing video editing software does not allow users to access or modify field structures. In addition, because our proposed method involves checking the header information of video content only once, much less detection time is required compared with existing methods that examine the entire frames. We store an ordered file structure of video content as a signature in the database using a customized automated tool. The signature appears according to the video editing software. Then, the suspected video content is compared to a set of signatures. If the file structure matches with a signature, we recognize a manipulated video file by its corresponding editing software. We tested five types of video editing software that cover 99% of the video editing software market share. Furthermore, we arranged 305,981 saving options for all five video editing suites. As a result, we obtained 100% detection accuracy using stored signatures, without false positives, in a collection of 305,981 video files. The principle of this method can be applied to other video formats.

Due to the popularity of online video sharing websites such as YouTube, millions of users have treated online video as a source of information and entertainment. Therefore, video annotation has evoked great interest in the past few years. In this paper, we propose a four-step approach to automatically annotate video shots with sentences. The first step is video preprocessing, converting video shot into a sequence of frame images. The second step is to find related candidate elements of the sentence about the video contents. The main elements in the sentence are objects, events, scenes, and modifiers. These candidate elements are gained by searching for similar images with the video frames in our collected image data sets instead of video data sets. The third step is to select the best elements among these candidate ones by a weighted scoring algorithm. The final step is to construct a sentence with the help of a correlation graph algorithm to analyze the relationships among the best elements. The experimental results indicate that our method is effective to annotate videos with sentences. What is more, the weighted scoring algorithm and the correlation graph algorithm that we propose are efficient in developing the experimental performance.

Modern day datasets continue to increase in both size and diversity. One example of such ‘big data’ is video data. Within the medical arena, more disciplines are using video as a diagnostic tool. Given the large amount of data stored within a video image, it is one of most time consuming types of data to process and analyse. Therefore, it is desirable to have automated techniques to extract, process and analyse data from video images. While many methods have been developed for extracting and processing video data, statistical modelling to analyse the outputted data has rarely been employed. We develop a method to take a video sequence of periodic nature, extract the RGB data and model the changes occurring across the contiguous images. We employ harmonic regression to model periodicity with autoregressive terms accounting for the error process associated with the time series nature of the data. A linear spline is included to account for movement between frames. We apply this model to video sequences of retinal vessel pulsation, which is the pulsatile component of blood flow. Slope and amplitude are calculated for the curves generated from the application of the harmonic model, providing clinical insight into the location of obstruction within the retinal vessels. The method can be applied to individual vessels, or to smaller segments such as 2 × 2 pixels which can then be interpreted easily as a heat map.

Rate control in video compression adjusts the encoding parameters to reach a certain target bitrate for the encoded video. State-of-the-art rate controllers for hybrid video coding typically employ content-dependent video bitrate models and video quality metrics (VQMs). To capture the content characteristics, temporal and spatial video activity measures are determined from the raw video using computationally complex algorithms that require access to the uncompressed source video. In automotive deployments, however, full access to the uncompressed source video and the internal functions of video encoders is typically not possible. As a remedy, in this paper, we present a low-complexity approach to estimate the temporal activity (TA) and spatial activity (SA) measures for videos that are captured by a front-facing camera of a vehicle, based on the context information of the vehicle. To this end, we exploit information about the dynamics of the vehicle and other vehicles in the field-of-view of the front-facing camera. We apply the estimated TA and SA values to a video bitrate model and an objective VQM and use these models to solve the rate control problem to determine the optimal encoding settings for given bitrate constraints. The proposed low-complexity solution offers a similar accuracy in achieving rate constraints and similar perceptual quality characteristics as a solution that uses the computed TA and SA values, with the advantage that no access to the uncompressed source video stream or the internal functions of the video encoder is required.

Convolutional neural networks (CNN) are a special type of deep neural networks (DNN). They have so far been successfully applied to image super-resolution (SR) as well as other image restoration tasks. In this paper, we consider the problem of video super-resolution. We propose a CNN that is trained on both the spatial and the temporal dimensions of videos to enhance their spatial resolution. Consecutive frames are motion compensated and used as input to a CNN that provides super-resolved video frames as output. We investigate different options of combining the video frames within one CNN architecture. While large image databases are available to train deep neural networks, it is more challenging to create a large video database of sufficient quality to train neural nets for video restoration. We show that by using images to pretrain our model, a relatively small video database is sufficient for the training of our model to achieve and even improve upon the current state-of-the-art. We compare our proposed approach to current video as well as image SR algorithms.

In 3-D video (3DV) applications, depth-image-based rendering (DIBR) has been widely employed to synthesize virtual views. However, this approach is performed in a frame-based way, meaning each whole frame is dealt with and the characteristics of different regions in the frame are ignored. As a result, redundant pixels in some regions are abused during the subsequent warping and blending stage. This paper proposes a region-aware 3-D warping approach for DIBR in which warped frames are reasonably divided beforehand so that only the indispensable regions are used. With the proposed scheme, it is possible to avoid noneffective and repeated pixels during the warping stage. In addition, the blending process is also saved. The experimental results show that compared to the state-of-the-art VSRS3.5 and VSRS-1D-fast algorithms, our approach can achieve significant computation savings without sacrificing synthesis quality.

Underexposed video enhancement aims at revealing hidden details that are barely noticeable in LDR video frames with noise. Previous work typically relies on a single heuristic tone mapping curve to expand the dynamic range, which inevitably leads to uneven exposure and visual artifacts. In this paper, we present a novel approach for underexposed video enhancement using an efficient perception-driven progressive fusion. For an input underexposed video, we first remap each video frame using a series of tentative tone mapping curves to generate an multi-exposure image sequence that contains different exposed versions of the original video frame. Guided by some visual perception quality measures encoding the desirable exposed appearance, we locate all the best exposed regions from multi-exposure image sequences and then integrate them into a well-exposed video in a temporally consistent manner. Finally, we further perform an effective texture-preserving spatio-temporal filtering on this well-exposed video to obtain a high-quality noise-free result. Experimental results have shown that the enhanced video exhibits uniform exposure, brings out noticeable details, preserves temporal coherence, and avoids visual artifacts. Besides, we demonstrate applications of our approach to a set of problems including video dehazing, video denoising and HDR video reconstruction.

Color-plus-depth 3D video is booming up with the growing popularity of depth-sensing camera. This imposes new challenges on video resizing since the information embedded in the depth and color video should be taken into consideration simultaneously. In this paper, we propose an energy-constrained video retargeting approach for color-plus-depth 3D video. We first build a 3D video visual attention (3DV-VA) computation model by combining gradient, saliency, motion and depth information. Then, an energy function is generated using spatial, temporal and 3DV-VA constraints. Finally, seam selection and carving operation is conducted to generate a retargeted 3D video. Experimental results on two types of 3D video sequences shows that the proposed approach produces better seam carving results than the related existing methods.

In this paper, we extend the free-energy principle to video quality assessment (VQA) by incorporating with the recent psychophysical study on human visual speed perception (HVSP). A novel video quality metric, namely the free-energy principle inspired video quality metric (FePVQ), is therefore developed and applied to perceptual video coding optimization. The free-energy principle suggests that the human visual system (HVS) can actively predict “orderly” information and avoid “disorderly” information for image perception. Basically, “orderly” is associated with the skeletons and edges of objects, and “disorderly” mostly concerns textures in images. Based on this principle, an image is separated into orderly and disorderly regions, and processed differently in image quality assessment. For videos, visual attention, or fixation, is associated with the objects with significant motion according to HVSP, resulting in a motion strength factor in the FePVQ so that the free-energy principle is extended into spatio-temporal domain for VQA. In addition, we investigate the application of the FePVQ in perceptual rate distortion optimization (RDO). For this purpose, the FePVQ is realized with low computational cost by using the relative total variation model and the block-wise motion vectors of video coding to simulate the free-energy principle and the HVSP, respectively. The experimental results indicate that the proposed FePVQ is highly consistent with the HVS perception. The linear correlation coefficient and Spearman's rank-order correlation coefficient are up to 0.8324 and 0.8281 on the LIVE video database. Better perceptual quality of encoded video sequences is achieved by FePVQ-motivated RDO in video coding.

To manipulate large video contents, effective video indexing and retrieval are required. A large number of video indexing and retrieval algorithms have been presented for frame-wise user query or video content query whereas a relatively few video sequence matching algorithms have been proposed for video sequence query. In this paper, we propose an efficient algorithm that extracts key frames using color histograms and matches the video sequences using edge features. To effectively match video sequences with a low computational load, we make use of the key frames extracted by the cumulative measure and the distance between key frames, and compare two sets of key frames using the modified Hausdorff distance. Experimental results with real sequence show that the proposed video sequence matching algorithm using edge features yields the higher accuracy and performance than conventional methods such as histogram difference, Euclidean metric, Battachaya distance, and directed divergence methods.

This article introduces interactive video maps for the web. The main component of video maps is a video stream that is areally georeferenced to a spatial reference system in the same way rectified raster orthoimages are georeferenced. The areal georeference allows for interactivity that goes beyond the play, pause, and stop functionality of video player software. We highlight two types of functionality, allowing the user to (1) combine the video stream layer with other raster and vector map layers and (2) adjust the projection of the map in real time. We exemplify video maps by A Year in the Life of Earth's CO2, an interactive video map that visualizes the results of a high-resolution NASA computer model of global atmospheric carbon dioxide distribution. The map shows how carbon dioxide travels around the globe over the course of one year. We use a combination of WebGL, a programming interface to the hardware-accelerated graphics pipeline, and HTML5 video for adding an areally georeferenced video ...

This paper investigates a novel learning-based camera motion characterization scheme along with its application to the video stabilization problem. The proposed characterization scheme represents the compressed domain block motion vectors (MVs) using polar angle and magnitude histograms. Discriminative features from these two histograms are extracted and fed to a supervised learning-based hierarchical classifier for recognizing the six camera motion patterns. A comparative analysis with an existing scheme is carried out to support and validate the proposed characterization scheme. The proposed scheme works at the frame level by classifying the inter-frame camera motion patterns. This scheme is extended to classify the video segments and a novel application to video stabilization is investigated. An experimental analysis of a number of test sequences captured using a handheld video camera shows that by characterizing the smooth and jittery motions, selective video stabilization could be carried out only on those video segments that have been degraded. This approach of selective video stabilization saves considerable amount of computational time compared with running the stabilization algorithm on the entire video sequence, as proposed in the literature. The proposed strategy of using a classification scheme prior to applying the video stabilization routine offers a new paradigm to the conventional video stabilization problem. Experimental validation carried out using exhaustive search motion estimation obtained block MVs, and H.264/Advanced Video Coding-obtained MVs shows that by using the idea of selective video stabilization, up to 62% reduction in processing time can be achieved compared with video stabilization approaches wherein the entire video sequence is processed.

A comprehensive scheme of temporal profile of surveillance video is proposed.It provides the sampling selection, analyzes properties of shape and motion of the temporal profile, develop the algorithms to detect dynamic targets, and measure their motion direction.Finally, the visualization of targets with multiple temporal slices with transparency is provided.Further, robust object detection based on background updating, and real time data transmission of is given.A comparison of our method with other video indexing methods is also given. This work converts the surveillance video to a temporal domain image called temporal profile that is scrollable and scalable for quick searching of long surveillance video by human operators. Such a profile is sampled with linear pixel lines located at critical locations in the video frames. It has precise time stamp on the target passing events through those locations in the field of view, shows target shapes for identification, and facilitates the target search in long videos. In this paper, we first study the projection and shape properties of dynamic scenes in the temporal profile so as to set sampling lines. Then, we design methods to capture target motion and preserve target shapes for target recognition in the temporal profile. It also provides the uniformed resolution of large crowds passing through so that it is powerful in target counting and flow measuring. We also align multiple sampling lines to visualize the spatial information missed in a single line temporal profile. Finally, we achieve real time adaptive background removal and robust target extraction to ensure long-term surveillance. Compared to the original video or the shortened video, this temporal profile reduced data by one dimension while keeping the majority of information for further video investigation. As an intermediate indexing image, the profile image can be transmitted via network much faster than video for online video searching task by multiple operators. Because the temporal profile can abstract passing targets with efficient computation, an even more compact digest of the surveillance video can be created.

Reconstructing a full-resolution stereoscopic video from an asymmetric stereoscopic video is a challenging task. The existing approaches require depth information, which imposes an additional challenge in data acquisition. In this paper, we propose a novel scheme that is capable of obtaining super-resolution and depth estimation simultaneously from an asymmetric stereoscopic video. The proposed scheme models the video super-resolution and stereo matching with a unified energy function. Then, we apply an alternating optimization method to minimize this energy function, which can be implemented with a two-step algorithm. In the first step we calculate the initial depth map by using a region-based cooperative optimization technique while considering the temporal consistency in video. In the second step we resolve the super-resolution problem under the guidance of the depth information. It is effective because each step benefits from the additional improvement over the previous step. We iteratively update the two steps until stable depth and super-resolution results are obtained. We have conducted a series of experiments on public stereoscopic video sequences to evaluate the performance of the proposed method. Both objective indexes and subjective visual comparisons verify that the proposed scheme can achieve satisfactory super-resolution results and high-quality depth map simultaneously. In particular, the subjective evaluation experiments on a 3-D monitor show that this scheme outperforms others and achieves the best visual sharpness.

A video is an effective tool to exchange the information in the structure of showing the brief text message due to the advance developed technology. Video capturing is effortless process but the related video retrieval is the difficult process, for that process the videos must be indexed. Retrieval is the method that retrieved a video using a user query. The query will be image or texts depend upon the query result output system that returned a particular video or image based on that query. In this project we create a indexing for video file by using segment based indexing technique. Here video will be divided into a hierarchy which is in storyboards of film making. For instance, a hierarchical based video search is composed into multi stage abstraction for assist the users to locate the specific video segments/frames logically. This paper brings out the reduced bandwidth and reduced delays the video through the network of searching and reviewing. Experimental results verify this.

This paper proposes a novel representation method which is called Depth and residual images based representation (DRIBR) for virtual image rendering in 3D video. The proposed DRIBR is composed of three parts: a single view image from one particular viewpoint, its corresponding depth map as well as a residual image. The residual image is pre-calculated from the combination of the above two components plus an image taken from the other viewpoint. The proposed DRIBR achieves low data redundancy, and resolves the occlusion problem which is one of the most annoying artifacts during the rendering of virtual image in traditional Depth image based rendering (DIBR). This paper addresses details of DRIBR and virtual image rendering algorithms based on the proposed format. We test the DRIBR on various stereo datasets and the experimental results demonstrate that our method can achieve better performance on both subjective and objective quality assessment.

Cloud gaming allows games to be rendered on the cloud server and allows the rendered videos to be encoded and streamed in real time to the player’s devices. Compared with other video streaming applications, cloud gaming offers a unique opportunity to enhance the video encoding process by exploiting rendering information. In this paper, we propose two techniques to improve cloud gaming video encoding, aiming at enhancing the perceived video quality and reducing the computational complexity, respectively. First, we develop a rendering-based prioritized encoding technique to improve the perceived game video quality according to network bandwidth constraints. We first propose a technique to generate a macroblock (MB)-level saliency map for every game video frame using rendering information. Furthermore, based on such a saliency map, a prioritized rate allocation scheme is proposed to dynamically adjust the value of quantization parameter of each MB. The experimental results indicate that the perceptual quality can be greatly improved using the proposed technique. We also develop a rendering-based encoding acceleration technique that utilizes rendering information to reduce the computational complexity of video encoding. This technique mainly consists of two parts. First, we propose a method to directly calculate the motion vectors (MVs) without employing the compute intensive motion search procedure. Second, based on the computed MVs, we propose a fast mode selection algorithm to reduce the number of candidate modes of each MB. The experimental results show that the proposed technique can achieve more than 42% saving in encoding time with very limited degradation in video quality.

The current research concerns the problem of video stabilization “in a point”, which aims to stabilize all video frames according to one chosen reference frame to produce a new video, as by a static camera. Similar task importance relates providing static background in the video sequence that can be usable for correct measurements in the frames when studying dynamic objects in the video. For this aim we propose an efficient combined approach, called “3×3OF9×9”. It fuses our the previous development for fast and rigid 2D video stabilization [2] with the well-known Optical Flow approach, applied by parts via Otsu segmentation, for eliminating the influence of moving objects in the video. The obtained results are compared with those, produced by the commercial software Warp Stabilizer of Adobe-After-Effects CS6.

Free Viewpoint Multiview Video TV (FTV) allows a user to pan horizontally within a video field of view. Interactive live FTV allows a user to experience being live at a soccer game with the interactive capability to freely look from side-to-side. Current efforts to commercialize interactive live FTV have failed due to a lack of standardization and unacceptably high real-world DVB (Direct Video Broadcast) system bandwidth requirements. In this paper, we provide a solution called OLFVmv (Optimized Live Free Viewpoint Multiview Video) that applies adaptive prediction of future video content requirements based on past viewer behavior, thereby dramatically reducing system bandwidth requirements. To expand, OLFVmv entails algorithms and a system architecture that allows for the intelligent segmentation and prioritization of video content based on video compression layering combined with viewer behavior statistics. Based on our positive results, we show that system operators can practically implement OLFVmv into existing systems.

This chapter presents an overview of the acquisition, processing, and rendering pipeline in a multiview video system, targeting the unique feature of rendering virtual views, not captured by the input camera feeds. This is called Virtual View Synthesis and supports Free Navigation similar to The Matrix bullet time effect that was popularized in the late 1990s. A substantial part of the chapter is devoted in explaining how to estimate the respective camera parameters and their relative positions in the system, as well as how to estimate/measure depth, which is an essential information in order to obtain smooth virtual view transitions with Depth Image-Based Rendering.

We present a study of the triad rendering computer made static and/or dynamic images, video games and adult users who interact with a personal computer. Besides, there is a diachronic study of the basic components to design the virtual 3D characters which are included in the video games. The link of the evolution of the interactive games is also analyzed and especially the interactive design characteristics related to the content, navigation, structure and layout. Finally, a table is presented with those components stemming from the rendering of the scenes for the video games, which motivate their fruition by the adult users.