Search

We present a framework for adaptive video encoding based on video content. The basic idea is to analyze the video to determine camera activity (tracking, stationary, or zooming) and then associate each activity with adaptive video quality constraints. We demonstrate our approach on the UT LIVE video quality assessment database that effective camera activity detection and classification is possible based on the motion vectors and the number of prediction units (PU) extracted using x265 HEVC encoding standard. In our results, by applying leave-one-out validation, we get an 79% correct classification rate using kNN binary classifier for the video segments.

It is important to generate both interesting and representative video summary for massive videos. This work proposes a new method to generate dynamic video summary using multiple features and image quality without human's involvement in the whole procedure. Specifically, we first split a video into several video clips. Second, a set of features including visual attention, exposure of light, saturation, hue, rule of thirds, contrast and directionality is computed and the qualities of video clips are also estimated. Then, the importance of each video clip is obtained based on these features and the estimated quality. Finally, based on the importance value, we sort clip values in descending order and select an optimal subset to generate video summary. Experimental results demonstrate that the proposed method enables to generate high-quality video summary.

Plenty of video stuff is created, broadcasted, shared and stored each and every day by industry experts, beginners, and hobbyists. Video summaries aim at showcasing the semantics and content of a clip in reduced time and space to enable a quick overview of video clip relevance. This paper focuses on static summaries showing key frames from the video. The key frames are extracted by leveraging the feature extraction and deep feature learning algorithm. The video summarization finds its application in semantic visualization, copy detection, indexing and browsing, concept detection, video services, video sharing and searching, video streaming, video caching, video quality assessment, and video transfer and video object tracking.

In many practical transcoding applications, such as video surveillance, the source videos are often contaminated by noise. The presence of noise not only results in poor compression efficiency and visual quality, but also imposes an adverse effect on the performance of subsequent video analysis tasks. Thereby it is very necessary to denoise the video. In this paper, we propose an efficient mode decision method for H.264 noisy video transcoding through analysing the effect of noise on mode decision. In the algorithm, we use the information available from previously decoded MBs to decide which modes can be overpassed with little loss to the rate-distortion performance. Experimental results show that our method saves the computational complexity nearly 65%, without noticeable rate-distortion(R-D) loss in comparison with the “Full-coding” (cascaded transcoding) method.

Video Watermarking is a prominent technique used to provide the authentication to the multimedia data such as audio, video and text. It is also used to resolve the copy right protection and ownership problem. The proposed method is used to provide the solution for copy right protection in video data. Here the different mathematical transforms like DWT and SVD are combined for the purpose of achieving the perceptual quality of the video and the key frame selection is achieved by chaotic map. Then the Two level DWT is performed in the video sequence, it produces the four sub bands like HH, HL, LH, LL. In this LL sub band is selected for embedding and extraction process. The watermark image is a binary image with proper resiolution. It is also subjected to SVD and performs the embedding process to embed the secret image into host video. This method survive different kinds of image and video processing attacks.

A high definition(HD) wide dynamic video surveillance system is designed and implemented based on Field Programmable Gate Array(FPGA). This system is composed of three subsystems, which are video capture, video wide dynamic processing and video display subsystem. The images in the video are captured directly through the camera that is configured in a pattern have long exposure in odd frames and short exposure in even frames. The video data stream is buffered in DDR2 SDRAM to obtain two adjacent frames. Later, the image data fusion is completed by fusing the long exposure image with the short exposure image (pixel by pixel). The video image display subsystem can display the image through a HDMI interface. The system is designed on the platform of Lattice ECP3-70EA FPGA, and camera is the Panasonic MN34229 sensor. The experimental result shows that this system can expand dynamic range of the HD video with 30 frames per second and a resolution equal to 1920∗1080 pixels by real-time wide dynamic range (WDR) video processing, and has a high practical value.

An objective video quality assessment method is proposed to evaluate the video quality in voice over internet protocol (VoIP) applications under network distortion. The fluency and the clarity of videos are two main parts of the factors that affect user experience, thus the method evaluates these two parts to assess the distortions of videos in VoIP applications caused by codec and packet loss. The clarity of the video is measured by calculating block artifacts and frame blurring. Video blocking artifacts are measured by splitting the picture into small blocks and calculating the difference of the pixels around each border while video blurring is measured by getting edge information through Sobel operator, and counting the gradient histogram. Then the video clarity can be measured by a weighted sum of block artifacts score and blurring score using linear regression. The scores are also normalized in order to eliminate the impact of different video contents. The video fluency is calculated by counting the wrong frame in the video. Finally, a weighted sum of video clarity score and video fluency score can represent the quality of the video. The experimental results show that the objective quality scores have a strong correlation with the subjective quality scores, and the algorithm concludes two parts of user experience other than just image quality, which is more comprehensive and it can be used in video quality assessment in VoIP applications.

Capturing or acquiring video data from advanced gadgets, like cell phone cameras, smart phones, camcorders, etc., is very recent and up growing trend. With the advancement of the video camera as the integrated part of these multimedia devices generates necessity for video stabilization. Digital video stabilization or video frame alignment techniques are used to improve quality of input video by removing the undesired camera motions jitters; undesirable jiggles etc. Many video stabilization techniques are developed till now but only fast and low computational complexity approaches can be applied on the embedded devices. The paper presents the review of fast block matching local motion vector estimation and alignment algorithm. Effectiveness of the algorithm is shown by quantitative evaluation. The proposed work will also help to boost the performance of various error concealment, error resilience and video quality improvement techniques in H.264/HEVC video codec.

Designing an effective and high performance network requires an accurate characterization and modeling of network traffic. The modeling of video frame sizes is normally applied in simulation studies and mathematical analysis and generating streams for testing and compliance purposes. Besides, video traffic assumed as a major source of multimedia traffic in future heterogeneous network. Therefore, the statistical distribution of video data can be used as the inputs for performance modeling of networks. The finding of this paper comprises the theoretical definition of distribution which seems to be relevant to the video trace in terms of its statistical properties and finds the best distribution using both the graphical method and the hypothesis test. The data set used in this article consists of layered video traces generating from Scalable Video Codec (SVC) video compression technique of three different movies.

Motion compensation is a fundamental technology in video coding to remove the temporal redundancy between video frames. To further improve the coding efficiency, sub-pel motion compensation has been utilized, which requires interpolation of fractional samples. The video coding standards usually adopt fixed interpolation filters that are derived from the signal processing theory. However, as video signal is not stationary, the fixed interpolation filters may turn out less efficient. Inspired by the great success of convolutional neural network (CNN) in computer vision, we propose to design a CNN-based interpolation filter (CNNIF) for video coding. Different from previous studies, one difficulty for training CNNIF is the lack of ground-truth since the fractional samples are actually not available. Our solution for this problem is to derive the "ground-truth" of fractional samples by smoothing high-resolution images, which is verified to be effective by the conducted experiments. Compared to the fixed half-pel interpolation filter for luma in High Efficiency Video Coding (HEVC), our proposed CNNIF achieves up to 3.2% and on average 0.9% BD-rate reduction under low-delay P configuration., Comment: International Symposium on Circuits and Systems (ISCAS) 2017

In this paper, we address the problem of face video retrieval. Given one face video of a person as query, we search the database and return the most relevant face videos, i.e., ones have same class label with the query. Such problem is of great challenge. For one thing, faces in videos have large intra-class variations. For another, it is a retrieval task which has high request on efficiency of space and time. To handle such challenges, this paper proposes a novel Deep Video Code (DVC) method which encodes face videos into compact binary codes. Specifically, we devise a multi-branch CNN architecture that takes face videos as training inputs, models each of them as a unified representation by temporal feature pooling operation, and finally projects the high-dimensional representations into Hamming space to generate a single binary code for each video as output, where distance of dissimilar pairs is larger than that of similar pairs by a margin. To this end, a smooth upper bound on triplet loss function which can avoid bad local optimal solution is elaborately designed to preserve relative similarity among face videos in the output space. Extensive experiments with comparison to the state-of-the-arts verify the effectiveness of our method.

Prediction techniques have been widely investigated over the last decades for image and video compression. Currently, most of the existing image and video compression solutions incorporate some sort of predictive coding in their algorithms. For example, temporal redundancy present in video sequences is typically exploited through the use of motion-compensated prediction methods.

This chapter introduces video processing with the focus on tracking changes in video frame images. Video frame changes can be detected in the changing shapes, locations and distribution of the polygons (regions) in Voronoi tilings of the frames.

Virtual reality and 360-degree video streaming are growing rapidly; however, streaming 360-degree video is very challenging due to high bandwidth requirements. To address this problem, the video quality is adjusted according to the user viewport prediction. High quality video is only streamed for the user viewport, reducing the overall bandwidth consumption. Existing solutions use shallow buffers limited by the accuracy of viewport prediction. Therefore, playback is prone to video freezes which are very destructive for the Quality of Experience(QoE). We propose using layered encoding for 360-degree video to improve QoE by reducing the probability of video freezes and the latency of response to the user head movements. Moreover, this scheme reduces the storage requirements significantly and improves in-network cache performance.

In this paper, we study the foreground object extraction, the trajectory extraction, the trajectory combination optimization and other key technologies of the surveillance video abstract generation technology. Put forward a kind of improved algorithm, through the pre-processing based on Focus Stacking, foreground object extraction of the foreground object shadow removal, the trajectory synthesis optimization. As a result, the foreground extraction accuracy rate was increased by 2.78%; because of shadow removal of the foreground object, the collision rate of the foreground object in the synthesized video is reduced by 15.77%; The use of Semi-Transparent Handling Collision (STHC) makes the trajectory of the foreground object is not interrupted, the video frame information is not lose and the compression rate is increased by about 10%. The algorithm is applied in this paper, and the optimization effect is observed through the whole system test. As a result, the clarity of the synthesized video is increased, the integrity of the video’s information is enhanced, and the compression rate of the video is improved.

Change detection is one of the most commonly encountered low-level tasks in computer vision and video processing. A plethora of algorithms have been developed to date, yet no widely accepted, realistic, large-scale video data set exists for benchmarking different methods. Presented here is a unique change detection video data set consisting of nearly 90000 frames in 31 video sequences representing six categories selected to cover a wide range of challenges in two modalities (color and thermal infrared). A distinguishing characteristic of this benchmark video data set is that each frame is meticulously annotated by hand for ground-truth foreground, background, and shadow area boundaries-an effort that goes much beyond a simple binary label denoting the presence of change. This enables objective and precise quantitative comparison and ranking of video-based change detection algorithms. This paper discusses various aspects of the new data set, quantitative performance metrics used, and comparative results for over two dozen change detection algorithms. It draws important conclusions on solved and remaining issues in change detection, and describes future challenges for the scientific community. The data set, evaluation tools, and algorithm rankings are available to the public on a website 1 and will be updated with feedback from academia and industry in the future.

In this paper, a rate-distortion optimized coding mode switching scheme is proposed to improve error resilience for multi-view video plus depth (MVD) based 3-D video transmission over lossy networks. First, we derive a new end-to-end distortion model for MVD-based 3-D video transmission. As compared with the previous MVD-based video distortion models in which distortion is measured by only investigating the expected texture video errors and depth errors on the synthesized virtual view, the proposed scheme characterizes both the end-to-end distortions in the rendered virtual view and the coded texture video due to packet losses. Moreover, inter-view error propagation for the texture video and depth map is also considered. Based on the proposed distortion model, an optimal mode decision algorithm is then performed in the texture video and depth map coding process. Experimental results show that the proposed method provides significant improvements in terms of both objective and subjective evaluations.

In free viewpoint video (FVV) and 3DTV, the depth image-based rendering method has been put forward for rendering virtual view video based on multi-view video plus depth (MVD) format. However, the projection with slightly different perspective turns the covered background regions into hole regions in the rendered video. This paper presents a depth enhanced image summarization generation model for the hole-filling via exploiting the texture fidelity and the geometry consistency between the hole and the remaining nearby regions. The texture fidelity and the geometry consistency are enhanced by drawing texture details and pixel-wise depth information into the energy cost of similarity measure correspondingly. The proposed approach offers significant improvement in terms of 0.2dB PSNR gain, 0.06 SSIM gain and subjective quality enhancement for the hole-filling images in virtual viewpoint video.

This paper presents a fully automatic algorithm for selecting the most appropriate areas of every video frame to show on heterogeneous display devices. The algorithm is used to analyze cinematic features in video and identify important parts of each frame. The proposed method is a client-server model. An importance map of video is created on the server and transmitted along with the video stream. The client uses this map to determine the cropping area of video frames to fit its display screen. An experimental comparison involving users' satisfaction of 160 participants who compared the results of our algorithm with others, such as letterbox, and commercially cropped editions of well-known motion pictures was performed. The results show that the audience preferred the video processed by our algorithm as the algorithm requires little additional preprocessing and little extra information embedded in the video stream. The results suggest that this algorithm could improve the video viewing experience for mobile users.

This paper presents an algorithm that is capable of generating 3-D video from monocular video through a hierarchical approach that characterizes the low-level features and high-level semantics of video content to generate depth map. Color and texture features are used to locally indicate video's characteristics followed by segmentation of the video into several regions. Subsequently, semantics, including shape and motion semantics, of segmented regions are delineated from a higher perspective to refine segmentation result via measuring the interrelations among segmented regions. Subsequently, according to refined segmentation map and semantics of segmented regions, the proposed method generates a depth map with high stability using both spatial and temporal information. The stable depth map minimized visual quality degradation, such as flicker and blurring when viewing the 3-D video. The experimental results show the capability of the proposed algorithm in generating a high quality and stable depth map. In addition, subjective viewing evaluation shows that the proposed algorithm surpasses those of the commercial products for converting 2-D video to 3-D video, including TriDef 3D and CyberLink Power DVD.

Semantic video indexing, also known as video annotation or video concept detection in literatures, has been attracting significant attention in recent years. Due to deficiency of labeled training videos, most of the existing approaches can hardly achieve satisfactory performance. In this paper, we propose a novel semantic video indexing approach, which exploits the abundant user-tagged Web images to help learn robust semantic video indexing classifiers. The following two major challenges are well studied: 1) noisy Web images with imprecise and/or incomplete tags; and 2) domain difference between images and videos. Specifically, we first apply a non-parametric approach to estimate the probabilities of images being correctly tagged as confidence scores. We then develop a robust transfer video indexing (RTVI) model to learn reliable classifiers from a limited number of training videos together with the abundance of user-tagged images. The RTVI model is equipped with a novel sample-specific robust loss function, which employs the confidence score of a Web image as prior knowledge to suppress the influence and control the contribution of this image in the learning process. Meanwhile, the RTVI model discovers an optimal kernel space, in which the mismatch between images and videos is minimized for tackling the domain difference problem. Besides, we devise an iterative algorithm to effectively optimize the proposed RTVI model and a theoretical analysis on the convergence of the proposed algorithm is provided as well. Extensive experiments on various real-world multimedia collections demonstrate the effectiveness of the proposed robust semantic video indexing approach.

In this paper, we propose a real-time video transcoding scheme for N-Screen service based on cloud computing. This scheme creates an intro-block and several playback blocks by splitting the original video. And there is the first service request, after transmitting the intro-block, transmits the playback blocks that converting the blocks on real-time. In order to completing trans-coding within playback time of each block, we split and allocate the block to node according to performance of each node. Also, in order to provide real-time video playback service, the previous ∙제1저자 : 임헌용 ∙교신저자 : 이원주 ∙투고일 : 2014. 6. 26, 심사일 : 2014. 7. 24, 게재확정일 : 2014. 8. 18 * 한양대학교 ERICA 캠퍼스 컴퓨터공학과(Dept. of Computer Science & Engineering, Hanyang University ERICA Campus) ** 인하공업전문대학 컴퓨터정보과(Dept. of Computer Science, Inha Technical College) 12 Journal of The Korea Society of Computer and Information September 2014 scheme convert original video into all format and resolution. However we show that the proposed scheme can reduce storage usage by converting original video into format with proper resolution suitable to device and platform of client. Through simulation, we show that it is more effective to real-time video playback for N-screen service than the previous method. We also show that the proposed scheme uses less storage usage than previous method. ▸

The design of large-scale video tracking software system is studied. With the continuous development of computer video processing technology, large-scale video tracking technology has become very important. This paper presents a design method for large-scale video tracking software system based on Radon detection algorithm. During the video tracking process, numerous video images need to be collected, and then preprocessed with filtering algorithm, through Radon detection method to predict and compensate moving objects trajectory obtained in video to make up for the tracking lag caused by mutated direction. Experimental results show that the proposed algorithm for large-scale video tracking software design can improve the tracking accuracy effectively, and achieve satisfactory results.

Now a days, a number of videos are available in video databases, social networking sites and other web servers. Large size of these video database make it difficult to trace the video content. To ensure the copy-right of the videos in video database, a video copy detection system is needed. A Video copy detection system stores the video features that characterize a video along with the video in the database. Existing copy detection systems store the video features as simple codewords. A simple and compact representation of video features makes the system more efficient. Moreover, the memory constraint problem can also be solved. This paper propose a sparse-coding technique that can represent the video features as sparse-codes. Proposed video copy detection system using sparse-codes works as follows: keyframes of the videos in the database are extracted using abrupttransition detection algorithm. Salient regions of keyframes are detected by Harris-Laplacian detector and its local features are described by Flip-Invariant SIFT(F-SIFT) descriptor. F-SIFT enriches SIFT with flip invariance property by preserving its feature distinctiveness. F-SIFT is invariant to operations like flip, rotation, scale etc. A 128-Dimensional F-SIFT descriptor is extracted from each salient region. Extracted descriptors are converted to sparse-codes by the proposed sparse-coding technique. Each keyframe is represented by the sparse feature vector. Sparse vectors of all the keyframes of a video forms the sparse code of the video. Sparse-codes of the input video are compared with the sparse-codes stored in video database to identify the near duplicate videos. Experimental results demonstrate that proposed sparse-coding technique reduces the memory constraint problem. It also improves the detection accuracy.

In this paper, we deal with the problem of video concept detection to use the concept detection results toward a more effective concept-based video retrieval. The key novelties of this paper are as follows: 1) the use of spatio-temporal video slices (tomographs) in the same way that visual keyframes are typically used in video concept detection schemes. These spatio-temporal slices capture in a compact way motion patterns that are useful for detecting semantic concepts and are used for training a number of base detectors. The latter augment the set of keyframe-based base detectors that can be trained using different frame representations. 2) The introduction of a generic methodology, built upon a genetic algorithm, for controlling which subset of the available base detectors (consequently, which subset of the possible shot representations) should be combined for developing an optimal detector for each specific concept. This methodology is directly applicable to the learning of hundreds of diverse concepts, while diverging from the one-size-fits-all approach that is typically used in problems of this size. The proposed techniques are evaluated on the datasets of the 2011 and 2012 Semantic Indexing Task of TRECVID, each comprising several hundred hours of heterogeneous video clips and ground-truth annotations for tens of concepts that exhibit significant variation in terms of generality, complexity, and human participation. The experimental results manifest the merit of the proposed techniques.

For a variety of applications such as video surveillance and event annotation, the spatial–temporal boundaries between video objects are required for annotating visual content with high-level semantics. In this paper, we define spatial–temporal sampling as a unified process of extracting video objects and computing their spatial–temporal boundaries using a learnt video object model. We first provide a computational approach for learning an optimal key-object codebook sequence from a set of training video clips to characterize the semantics of the detected video objects. Then, dynamic programming with the learnt codebook sequence is used to locate the video objects with spatial–temporal boundaries in a test video clip. To verify the performance of the proposed method, a human action detection and recognition system is constructed. Experimental results show that the proposed method gives good performance on several publicly available datasets in terms of detection accuracy and recognition rate.

With the increasing volume of video data, how to analyze and browse video in a fast and effective way has become an urgent problem in applications. This letter proposes a novel video synopsis method in compressed domain for browsing video captured by static cameras. Synopsis video is a video abstraction, which displays moving objects from different periods simultaneously on the primary background contents of original video. To overcome the low efficiency of traditional video synopsis for compressed video, our method presents a new graph cut algorithm to extract objects tubes and meanwhile gives a fast solution to minimize energy function in compressed domain. Experimental results in H.264 video have demonstrated the high-efficiency of this new video synopsis scheme for massive video browsing.

Digital video watermarking is an efficient way of copyright protection. A robust watermarking was proposed that ccording to the video features using the evolutional method of video image classification processing, suitable for video performance and human visual characteristics of video watermarking, and put forward a kind of using Bees evolutional algorithm for video image classification processing of video watermarking algorithm. All the key frame numbers are converts to a binary matrix after data transformation. At last the binary image is produced. From this, the watermarking image reflects the video’s features, so it is effective to resist the watermark copy attack.

We present a novel approach to free-viewpoint video. Our main contribution is the formulation of a hybrid approach between image morphing and depth-image based rendering. When rendering the scene from novel viewpoints, we use both dense pixel correspondences between image pairs as well as an underlying, view-dependent geometrical model. Our novel reconstruction scheme iteratively refines geometric and correspondence information. By combining the strengths of both depth and correspondence estimation, our approach enables free-viewpoint video also for challenging scenes as well as for recordings that may violate typical constraints in multiview reconstruction. For example, our method is robust against inaccurate camera calibration, asynchronous capture, and imprecise depth reconstruction. Rendering results for different scenes and applications demonstrate the versatility and robustness of our approach.

Efficient and robust video copy detection is an important topic for many applications, such as commercial monitoring and social media retrieval. In this paper, with the aim of handling large-scale video data, we propose an efficient and robust video copy detection method jointly utilizing the characteristics of temporal continuity and multi-modality of video. The video is converted to a continuous sequence of states, and both the visual and auditory features are extracted for temporal frames. To facilitate tolerance of the length variations caused during video re-targeting, an efficient dynamic path search method is proposed to detect the target video clips, and highly compact audio fingerprint and visual ordinal features are jointly utilized in a flexible frame. The proposed scheme not only achieves high computational efficiency but also guarantees effectiveness in real applications. Comparison experiments were conducted using video commercials and real television programs from four channels as well as a benchmark video copy detection dataset, and the results demonstrate both the high efficiency and high robustness of the proposed method.

This paper presents a method to detect video tampering and distinguish it from common video processing operations, such as recompression, noise, and brightness increase, using a practical watermarking scheme for real-time authentication of digital video. In our method, the watermark signals represent the macroblock's and frame's indices, and are embedded into the nonzero quantized discrete cosine transform value of blocks, mostly the last nonzero values, enabling our method to detect spatial, temporal, and spatiotemporal tampering. Our method can be easily configured to adjust transparency, robustness, and capacity of the system according to the specific application at hand. In addition, our method takes advantage of content-based cryptography and increases the security of the system. While our method can be applied to any modern video codec, including the recently released high-efficiency video coding standard, we have implemented and evaluated it using the H.264/AVC codec, and we have shown that compared with the existing similar methods, which also embed extra bits inside video frames, our method causes significantly smaller video distortion, leading to a PSNR degradation of about 0.88 dB and structural similarity index decrease of 0.0090 with only 0.05% increase in bitrate, and with the bit correct rate of 0.71 to 0.88 after H.264/AVC recompression.

4D Video Textures 4DVT introduce a novel representation for rendering video-realistic interactive character animation from a database of 4D actor performance captured in a multiple camera studio. 4D performance capture reconstructs dynamic shape and appearance over time but is limited to free-viewpoint video replay of the same motion. Interactive animation from 4D performance capture has so far been limited to surface shape only. 4DVT is the final piece in the puzzle enabling video-realistic interactive animation through two contributions: a layered view-dependent texture map representation which supports efficient storage, transmission and rendering from multiple view video capture; and a rendering approach that combines multiple 4DVT sequences in a parametric motion space, maintaining video quality rendering of dynamic surface appearance whilst allowing high-level interactive control of character motion and viewpoint. 4DVT is demonstrated for multiple characters and evaluated both quantitatively and through a user-study which confirms that the visual quality of captured video is maintained. The 4DVT representation achieves >90% reduction in size and halves the rendering cost.

3D video services are emerging in various application domains including cinema, TV broadcasting, Blu-ray discs, streaming and smartphones. A majority of the 3D video content in market is still based on stereo video, which is typically coded with the multiview video coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard or as frame-compatible stereoscopic video. However, the 3D video technologies face challenges as well as opportunities to support more demanding application scenarios, such as immersive 3D telepresence with numerous views and 3D perception adaptation for heterogeneous 3D devices and/or user preferences. The Multiview Video plus Depth (MVD) format enables depth-image-based rendering (DIBR) of additional viewpoints in the decoding side and hence helps in such advanced application scenarios. This paper reviews the MVC+D standard, which specifies an MVC-compatible MVD coding format.

Due to advances in recent multimedia technologies, various digital video contents become available from different multimedia sources. Efficient management, storage, coding, and indexing of video are required because video contains lots of visual information and requires a large amount of memory. This paper proposes an efficient video indexing method for video with rapid motion or fast illumination change, in which motion information and feature points of specific objects are used. For accurate shot boundary detection, we make use of two steps: block matching algorithm to obtain accurate motion information and modified displaced frame difference to compensate for the error in existing methods. We also propose an object matching algorithm based on the scale invariant feature transform, which uses feature points to group shots semantically. Computer simulation with five fast-motion video shows the effectiveness of the proposed video indexing method.

Recently, many researches have been focused on multi-view video applications and services such as wireless video surveillance networks, wireless video sensor networks and wireless mobile video. In multi-view video signal processing, to exploit the strong correlation between images acquired by different cameras plays great role in developing a core technique of multi-view video coding. This paper proposes an adaptive multi-view video interpolation technique which is applicable for multi-view distributed video coding without requiring any cooperation amongst the cameras. The proposed algorithm estimates the non-linear moving blocks and employs disparity compensated view prediction, and then fills in the unreliable blocks. Through computer simulations, it is shown that the proposed method outperforms the conventional methods.

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is typically synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this goal, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos-a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing pixel samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will mitigate errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Finally, quantization parameter (QP) is adaptively selected per frame, optimally trading off source distortion due to compression with channel distortion due to potential packet losses. Experimental results show that the proposed scheme can outperform previous work by up to 2.9 dB at 5% packet loss rate.

In this letter, we propose a depth-assisted frame rate up-conversion (DA-FRUC) scheme which finds applications in 3D video processing. By considering the depth cue in video plus depth representation, we categorize the blocks of the interpolated frame as depth-continuous and depth-discontinuous groups. The motion vector (MV) outliers of the depth continuous blocks are then detected and corrected by layer-constrained MV refinement method. Moreover, a depth-based adaptive interpolation and block segmentation method is proposed to deal with the disocclusion and occlusion at the boundary of the foreground area. Experimental results show that the proposed method achieves better subjective and objective performance for the interpolated color frames. Additionally, the interpolated depth frames obtained by the proposed method is more accurate, which benefit the video quality in view synthesis.

Video transcoding is a legitimate operation widely used to modify video format in order to access the video content in the end-user's devices, which may have some limitations in the spatial and temporal resolutions, bit-rate and video coding standards. In many previous watermarking algorithms the embedded watermark is not able to survive video transcoding, because this operation is a combination of some aggressive attacks, especially when lower bit-rate coding is required in the target device. As a consequence of the transcoding operation, the embedded watermark may be lost. This paper proposes a robust video watermarking scheme against video transcoding performed on base-band domain. In order to obtain the watermark robustness against video transcoding, four criteria based on Human Visual System (HVS) are employed to embed a sufficiently robust watermark while preserving its imperceptibility. The quantization index modulation (QIM) algorithm is used to embed and detect the watermark in 2D-Discrete Cosine Transform (2D-DCT) domain. The watermark imperceptibility is evaluated by conventional peak signal to noise ratio (PSNR) and structural similarity index (SSIM), obtaining sufficiently good visual quality. Computer simulation results show the watermark robustness against video transcoding as well as common signal processing operations and intentional attacks for video sequences.

Since July of 2012, the 3D video extension of H.264/AVC has been under development to support the multi-view video plus depth format. In 3D video applications such as multi-view and free-view point applications, synthesized views are generated using coded texture video and coded depth video. Such synthesized views can be distorted by quantization noise and inaccuracy of 3D wrapping positions, thus it is important to improve their quality where possible. To achieve this, the relationship among the depth video, texture video, and synthesized view is investigated herein. Based on this investigation, an edge noise suppression filtering process to preserve the edges of the depth video and a method based on a total variation approach to maximum a posteriori probability estimates for reducing the quantization noise of the coded texture video. The experiment results show that the proposed methods improve the peak signal-to-noise ratio and visual quality of a synthesized view compared to a synthesized view without post processing methods.

In this paper, we study the impact of quantization, frame dropping and spatial down-sampling on the perceived quality of compressed video streams. Based on the analysis of quality ratings obtained from extensive subjective tests, we propose a no-reference metric (named MDVQM) for video quality estimation in the presence of both spatial and temporal quality impairments. The proposed metric is based on the per-pixel bitrate of the encoded stream and selected spatial and temporal activity measures extracted from the video content. All the values required to compute the proposed video quality metric can be obtained without using the original reference video which makes the metric for instance useful for making transcoding decisions in a wireless video transmission scenario. Different from comparable metrics in the literature, we have also considered the case when both frame rate and frame size are changed simultaneously. The validation results show that the proposed metric provides more accurate estimation of the video quality than the state of the art metrics.

In software driven multi-user video conferencing systems, consumer progression towards ultra-high definition (i.e., 8 k) resolution introduces new challenges to the video combination process in sustaining smooth combined video playback at high frame rates. As such, this paper analyzes the underlying architecture of a conventional video combiner to identify the performance impact of combined video frame rates at ultra-high definition resolutions. Then, two straightforward parallel video combination architectures using software application threads, namely PVC-1 and PVC-2, are studied as a benchmark. In PVC-1, the number of application threads tallies with the number of client videos to be combined, which improves playback performance at the expense of inconsistencies in inset client frame rates within a combined video. PVC-2 includes a synchronizer to address these inconsistencies, but exhibits marginal performance gains. To address the aforementioned problems, a balanced workload parallel video combiner architecture is proposed, namely PVC-3. In this architecture, a balanced workload management algorithm stitches client videos into an ultra-high definition combined frame, based on the number of available logical processors on a multi-core processor. This method improves frame rate performance and sustains consistent client frame rates within a combined video. Experimental results suggest that PVC-3 is superior over PVC-2 and achieves a frame rate performance gain of 27 % against a conventional video combiner for a combined video of 16 clients (each at a resolution of 720p) and zero standard deviation in combined frame rate consistency.

Cloud mobile rendering (CMR), where compute intensive rendering is performed on cloud servers instead of on mobile devices, can be a promising approach to enable rich rendering based multimedia applications on battery and CPU constrained mobile devices. However, since the video rendered in the cloud has to be streamed to the mobile device over a wireless network with fluctuating and constrained bandwidth, the resulting user experience can be impacted. In the work of Wang and Dey, 2013, adaptive rendering was proposed as a solution, wherein multiple rendering factors can be adapted such that the encoding bit rate of the rendered video is compatible with network bandwidth. However, changing the rendering factors may itself have adverse impact on user experience, which has not been studied earlier. Moreover, the impact can vary depending on the content scene and complexity. In this paper, we analyze the impairment of rendering factors on user experience while considering content characteristics, and combine the rendering impairments with impairments due to video encoding factors (like bit rate and frame rate) and network factors (like bandwidth and delay) to propose a content-aware model to measure user experience. We demonstrate the accuracy of the content-aware model through subjective testing. Furthermore, we use the model to: 1) demonstrate and investigate the tradeoff between rendering quality and encoding quality of the resulting video; and 2) derive a content-aware adaptive rendering (CAAR) algorithm which can dynamically adjust the rendering factors depending on current network conditions, so as to obtain an optimal tradeoff between rendering and encoding quality and maximize the overall user experience at any time. By conducting experiments in a real cellular network, we demonstrate that the proposed CAAR algorithm can greatly enhance the user experience of cloud rendered and streamed mobile video.

In digital forensics, recovery of a damaged or altered video file plays a crucial role in searching for evidences to resolve a criminal case. This paper presents a frame-based recovery technique of a corrupted video file using the specifications of a codec used to encode the video data. A video frame is the minimum meaningful unit of video data. Many existing approaches attempt to recover a video file using file structure rather than frame structure. In case a target video file is severely fragmented or even has a portion of video overwritten by other video content, however, video file recovery of existing approaches may fail. The proposed approach addresses how to extract video frames from a portion of video to be restored as well as how to connect extracted video frames together according to the codec specifications. Experiment results show that the proposed technique successfully restores fragmented video files regardless of the amount of fragmentations. For a corrupted video file containing overwritten segments, the proposed technique can recover most of the video content in non-overwritten segments of the video file.

VC-1 is one of the three video coding standards for Blu-ray DVD, which also includes MPEG-2 and H.264. In this paper, an efficient transcoding algorithm from VC-1 video to H.264 video is discussed, we present a transcoder that addresses I, P, B, and interlaced pictures. The main differences between the two standards are analyzed and a solution for transforming from one to the other is presented. The paper proposes a pixel domain transcoder which exploits the variable size transform used in VC-1 to select the variable block size for motion compensation in H.264. We also discuss transcoding of high profile video features; in particular, adaptive size transform and interlacing. Experimental results show that the pixel domain approach reduces computational complexity by more than 50 % as compared to a cascaded one with negligible drop in peak-signal-to-noise-ratio (PSNR).

The new video coding standard, High Efficiency Video Coding (HEVC), was developed to succeed the current standard, H.264/AVC, as the state of the art in video compression. However, there is a lot of legacy content encoded with H.264/AVC. This paper proposes and evaluates several transcoding algorithms from the H.264/AVC to the HEVC format. In particular, a novel transcoding architecture, in which the first frames of the sequence are used to compute the parameters so that the transcoder can learn the mapping for that particular sequence, is proposed. Then, two types of mode mapping algorithms are proposed. In the first solution, a single H.264/AVC coding parameter is used to determine the outgoing HEVC partitions using dynamic thresholding. The second solution uses linear discriminant functions to map the incoming H.264/AVC coding parameters to the outgoing HEVC partitions. This paper contains experiments designed to study the impact of the number of frames used for training in the transcoder. Comparisons with existing transcoding solutions reveal that the proposed work results in lower rate-distortion loss at a competitive complexity performance.

We present a user-based method that detects regions of interest within a video in order to provide video skims and video summaries. Previous research in video retrieval has focused on content-based techniques, such as pattern recognition algorithms that attempt to understand the low-level features of a video. We are proposing a pulse modeling method, which makes sense of a web video by analyzing users'Replayinteractions with the video player. In particular, we have modeled the user information seeking behavior as a time series and the semantic regions as a discrete pulse of fixed width. Then, we have calculated the correlation coefficient between the dynamically detected pulses at the local maximums of the user activity signal and the pulse of reference. We have found that users'Replayactivity significantly matches the important segments in information-rich and visually complex videos, such as lecture, how-to, and documentary. The proposed signal processing of user activity is complementary to previous work in content-based video retrieval and provides an additional user-based dimension for modeling the semantics of a social video on the web.

Resolution is one of the important factors for video visual quality. With the development of the video editing software, one video sequence can be easily enlarged and inserted into another video sequence to generate a forgery video. The detection of video resolution modifications has become an important issue for digital video authentication. In this paper, we propose a passive detection algorithm based on the zero-crossings of the high-frequency component of the key frames in frequency domain. Firstly, the key frames of video are extracted before the second-order differential is executed. Then, the zero crossings of differential are calculated to generate a one-dimensional binary sequence. Finally, the frequency analysis is applied to expose the original resolution of the video for detection. Based on this method, we also develop a forgery detection method for sequence inserting. The experimental results demonstrate that the proposed method can accurately estimate the video original resolution, as well as detect the forgery traces of synthetic videos with maliciously inserted irrelevant video sequences.

Network traffic associated with video increases sharply, how to choose the interested information for a number of Internet users is challenging. So, technologies and applications related with video, such as video search, video fast browsing, video index and storage are in great demand. Behind these technologies and applications, a core problem is how to quickly browse massive video data and obtain the main content of the video. To solve this problem, different key frame extraction algorithms have been proposed. Due to the diversity of video content, different video have different characteristics. So the design of general video key frame extraction algorithm to solve the problem is not the reality. The main trend for the problem is to design the key frame extraction algorithm based on the characteristics of the video itself. In this article, we mainly focus on videos with edited boundaries and shot conversions. Aiming at this kind of video, we have designed and implemented video key frame extraction algorithm based on sliding window, the global feature Gist and local feature point detection algorithm SURF. In this algorithm, we use Gist feature to construct the global scene information of frames, and the SURF keypoint detection algorithm to extract local keypoints as local feature for each frame. Then, shot segmentation based on sliding window and shot merging algorithm is applied to dividing the original video into several shots. After that, we select the most representative frames in each video shot as key frames. Finally we evaluate the result of the algorithm from the subjective and objective perspective. Results show that key frames extracted in the algorithm are of high quality and can basically cover the main content of the original video.

Although Query-by-Example techniques based on Euclidean distance in a multidimensional feature space have proved to be effective for image databases, this approach cannot be effectively applied to video since the number of dimensions would be massive due to the richness and complexity of video data. The above issue has been addressed in two recent solutions, namely Deterministic Quantization (DQ) and Dynamic Temporal Quantization (DTQ). DQ divides the video into equal segments and extracts a visual feature vector for each segment. The bag-of-word feature is then encoded by hashing to facilitate approximate nearest neighbor search using Hamming distance. One weakness of this approach is the deterministic segmentation of video data. DTQ improves on this by using dynamic video segmentation to obtain varied-length video segments. As a result, feature vectors extracted from these video segments can better capture the semantic content of the video. To support very large video databases, it is desirable to minimize the number of segments in order to keep the size of the feature representation as small as possible. We achieve this by using only one video segment (i.e., no video data segmentation is even necessary) with even better retrieval performance. Our scheme models video using differential long short-term memory (DLSTM) recurrent neural networks and obtains a highly compact fixed-size feature representation with the output of hidden states of the DLSTM. Each of these features are further compressed by hashing them into binary bits via quantization. Experimental results based on two public data sets, UCF101 and MSRActionPairs, indicate that the proposed video modeling technique outperforms DTQ by a significant margin.

Although PRNU (Photo Response Non-Uniformity)-based methods have been proposed to verify the source camera of a non-stabilized video, these methods may not be adequate for stabilized videos. The use of video stabilization has been increasing in recent years with the development of novel stabilization software and the availability of stabilization in smart-phone cameras. This paper presents a PRNU-based source camera attribution method for out-of-camera stabilized video (i.e., stabilization applied after the video is captured). The scheme can (i) automatically determine if a given video is stabilized, (ii) calculate the fingerprint from a stabilized video, and (iii) effectively correlate the fingerprint computed from a stabilized video (i.e., anonymous video) with a fingerprint computed from another stabilized or non-stabilized video (i.e., a known video). Furthermore, experimental results show that the source camera of an anonymous non-stabilized video can be verified using a fingerprint computed from a set of images.