Search

Video inpainting is a technique that fills in the missing regions or gaps in a video by using its known pixels. The existing video inpainting algorithms are computationally expensive and introduce seam in the target region that arises due to variation in brightness or contrast of the patches. To overcome these drawbacks, the authors propose a novel two‐stage framework. In the first step, sub‐bands of wavelets of a low‐resolution image are obtained using the dual‐tree complex wavelet transform. Criminisi algorithm and auto‐regression technique are then applied to these sub‐bands to inpaint the missing regions. The fuzzy logic‐based histogram equalisation is used to further enhance the image by preserving the image brightness and improve the local contrast. In the second step, the image is enhanced using super‐resolution technique. The process of down‐sampling, inpainting and subsequently enhancing the video using the super‐resolution technique reduces the video inpainting time. The framework is tested on video sequences by comparing and analysing the structural similarity index matrix, peak‐signal‐to‐noise ratio, visual information fidelity in pixel domain and execution time with the state‐of‐the‐art algorithms. The experimental analysis gives visually pleasing results for object removal and error concealment. [ABSTRACT FROM AUTHOR]

360-degree video synthetic aperture radar (VideoSAR) presents a powerful capability for the information perception of dynamic scenes. In comparison to time-consuming airborne SAR, miniature SAR (MiniSAR) mounted on an unmanned aerial vehicle platform is a cost-effective configuration that provides a higher operation flexibility for video acquisition. Nevertheless, MiniSAR system development and signal processing limited by multiple factors are still challenging issues, including restricted weight, volume, power consumption, complicated trajectory imaging, and high-efficient video processing. In this article, we develop a multirotors VideoSAR system at X-band in Nanjing University of Aeronautics and Astronautics, concerning hardware architecture, field experiment, and video signal processing. It has the capability of ultrahigh resolution imaging with three available channels up to 1.8-GHz transmission bandwidth. Field programmable gate array-based unified signal processing architecture can further accelerate the generation of massive VideoSAR sequences in terms of both circular spotlight and stripmap modes. Several experimental results have been presented to demonstrate the validity of our developed system. [ABSTRACT FROM AUTHOR]

Depth-image-based rendering (DIBR), as the most popular view synthesis method, is commonly used in the application of multi-view and free-viewpoint videos. However, the quality evaluation of DIBR-synthesised videos remains largely unexplored, which may hinder the development of more advanced view synthesis technology. With this motivation, this Letter presents a new quality metric for DIBR-synthesised videos. Specifically, the disoccluded regions are first detected based on an adaptive threshold to quantify geometric distortions. An energy-based sequence mapping strategy is proposed to portray spatiotemporal inconsistency by calculating first-order and second-order similarities in the gradient magnitude domain and the Laplace-of-Gaussian domain, respectively. Finally, the overall quality score is generated by pooling the scores of geometric distortion and spatiotemporal inconsistency. Experimental results demonstrate that the proposed metric outperforms the state-of-the-art metrics dedicated to DIBR-synthesised images and videos. [ABSTRACT FROM AUTHOR]

In action recognition, the dynamic image (DI) approach is recently proposed to code a video signal to a still image. Since DI descriptor is strongly dependent on first frames, it cannot extract dynamics that do not occur in the first frames or even long dynamics. On the other hand, most of the video frames are not informative for the task of action recognition. Therefore, the authors' intuition is that the process of representing a video using all frames is inefficient. Thus, in this study, they proposed to remove the existing redundancy inside the frames and extract some processed informative images based on the information theory which are called key frames. The proposed method is capable enough to extract sufficient frames regardless of the duration and the position of frames in the entire video. Motivated by this method and DI, they proposed a novel key frames dynamic image (KFDI) approach. Experimental results on popular UCF11, Olympic Sports, and J‐HMDB datasets show the superiority of the proposed KFDI approach compared to the DI in capturing long dynamics of videos for action recognition. Their experiments show KFDI improves the accuracy 2–6%compared to DI. [ABSTRACT FROM AUTHOR]

Video summarisation greatly improves the efficiency of people browsing videos and saves storage space. A good video summary should satisfy human visual interestingness and preserve the theme of the original video at the semantic level. Unlike many existing methods that consider only visual features to generate video summaries, this study proposes a method that combines visual and semantic cues to extract important information for dynamic video summarisation. The authors propose visual‐verbal saliency consistency to add semantic information and propose a novel attention motion, along with other visual features to fully represent visual interestingness. Based on the importance score of each frame calculated by combining these features, they select an optimal subset of segments to generate an important and interesting summary. They evaluate their method using the SumMe and TVSum datasets and experimental results show that their method generates high‐quality video summaries. [ABSTRACT FROM AUTHOR]

In this work, the authors propose a novel self‐supervised learning method based on mutual information to learn representations from the videos without manual annotation. Different video clips sampled from the same video usually have coherence in the temporal domain. To guide the network to learn such temporal coherence, they maximise the mutual information between global features extracted from different clips sampled from the same video (Global‐MI). However, maximising the Global‐MI leads the network to seek shared content from different video clips and may make the network degenerate to focus on the background of the video. Considering the structure of the video, they further maximise the average mutual information between the global feature and local patches of multiple regions of the video clip (multi‐region Local‐MI). Their approach, which is called Max‐GL, learns the temporal coherence by jointly maximising the Global‐MI and multi‐region Local‐MI. Experiments are conducted to validate the effectiveness of the proposed Max‐GL. Experimental results show that the Max‐GL can serve as an effective pre‐training method for the task of action recognition in videos. Additional experiments for the task of action similarity labelling and dynamic scene recognition also validate the generalisation of the learned representations of the Max‐GL. [ABSTRACT FROM AUTHOR]

Based on the image self‐similarity and singular value decomposition (SVD) techniques, the authors propose an iterative adaptive global denoising method. For the structural differences between image patches, they adaptively determine the size of the search window. In each window, a similar image patch matrix is constructed based on the multi‐scale similarity measure. In order to ensure the speed of the method, the adaptive step size and the number of image patches are introduced, and all image patches are denoised in different iterations. This not only ensures the speed of the method, suppresses residual noise, but also reduces the artefacts caused by the fixed step size and the number of image patches. Therefore, the problem of image denoising is converted to the estimation of low‐rank matrix. New singular values are estimated according to the noise level, and similar image patch matrices without noise are estimated using them and corresponding singular vectors. Experimental results show that compared with the state‐of‐the‐art denoising algorithms, this method has a higher PSNR and FSIM, and has a good visual effect. The new method can be applied to image and video restoration, target recognition and image classification. [ABSTRACT FROM AUTHOR]

In this work, the authors have proposed a method for shadow detection and removal from videos by utilising methods of machine learning. From literature, various algorithms on shadow detection and removal have been accounted with advantages and disadvantages. Here some algorithms have a need for manual alignment and predefined explicit parameters, but fail to give precise outcome in different lighting and ecological surroundings. In this work, the authors propose a three‐phase framework. In first stage, key frames are chosen by utilising features based K‐means clustering which selects the key frames using features like colour, shape and surface. In second stage, they utilised two‐stage segmentation techniques to segment the shadows by marking the region of interest. In the final step, they use threshold based segmentation to remove the shadow in videos. The performance of the proposed method is compared by performance evaluation of all state‐of‐the‐art methods. The proposed strategies are established to achieve superior results in comparison to other state‐of‐the‐art methods. [ABSTRACT FROM AUTHOR]

Providing an easy-to-use and non-invasive heart rate estimation system for home healthcare is imperative for detecting health deterioration in the early stages and for providing preventive treatments. To this end, we propose a ballistocardiographic method that estimates heart rate by analysing face videos and exploiting cyclic head motions caused by blood pumping through the carotid arteries during heartbeats. The proposed method addresses hand motion artefacts that occur while tracking head motions on a mobile platform. This is achieved by training an auto-encoder that retrieves a heart rate-related signal by learning consistent and statistically dominant relationships between facial landmarks. Moreover, unlike conventional methods, where a feature point tracker was used to track the facial landmarks, we propose using region-based tracking to increase tracking robustness in challenging lighting conditions. Experimental results demonstrate that the proposed method performs best in challenging light conditions and can robustly manage hand motion artefacts on a mobile platform. [ABSTRACT FROM AUTHOR]

Human emotion recognition is studied by means of unimodal channels over the last decade. However, efforts continue to answer tempting questions about how variant modalities can complement each other. This study proposes a multimodal approach using three‐dimensional (3D) convolutional neural networks (CNNs) to model human emotion through a modality‐referenced system while investigating the solution to such questions. The proposed modality‐referenced system selects the input data based on one of the modalities regarded as reference or master. The other modality which is referred to as a slave simply adjusts or attunes itself with the master in the temporal domain. In this context, the authors developed three multimodal emotion recognition system, namely, video‐referenced system, audio‐referenced system, and the audio–visual‐referenced system to explore the congruence impact of audio and video modalities on each other. Two pipelines of 3D CNN architectures are employed where k‐means clustering is used in the master pipeline and the slave pipeline adapts itself in a temporal sense. The outputs of the two pipelines are fused to improve recognition performance. In addition, canonical correlation analysis and t‐distributed stochastic neighbour embedding is used validating the experiments. Results show that temporal alignment of the data between two modalities improves the recognition performance significantly. [ABSTRACT FROM AUTHOR]

In the tracking‐by‐detection approach of online multiple object tracking (MOT), a major challenge is how to associate object detections on the new video frame with previously tracked objects. Two important aspects that directly influence the performance of MOT are quality of detection and accuracy in data association. The authors propose an efficient and unified MOT framework for improved object detection, followed by enhanced object tracking. The object detection and tracking are considered as two independent functions in the tracking‐by‐detection paradigm. In this study, object detection accuracy has been increased by employing a faster region‐based convolutional neural network (Faster R‐CNN) modified with the feedback region proposals from the tracker. Target association is performed by the correlation filter‐based Siamese CNN model, which finds the similarity score between the input image patches. The Siamese CNN is trained using a supervised hard sample mining strategy. An optical flow‐based motion model is employed to predict the next probable location of the targets from the tracker and these region proposals are fed back to the classifier module of Faster R‐CNN. The authors' extensive analysis of publicly available MOT benchmark datasets and comparison with the state‐of‐the‐art tracking methods demonstrate competitive tracking performance of the proposed MOT framework. [ABSTRACT FROM AUTHOR]

Multiple object tracking sets the foundation for many intelligent video applications. The authors present a novel tracking solution using the ability of recurrent neural networks to effectively model complex temporal dynamics between objects irrespective of appearances, pose, occlusions, and illumination. For online tracking, a real-time and accurate association of objects with active tracks poses the major algorithmic challenge. Additionally, re-entry of objects should also be correctly resolved. They follow tracking-by-detection methodology using hierarchical long short term memory (LSTM) network structure for modelling the motion dynamics between objects by learning the fusion of appearance and motion cues. Existing works capture object's perspective for tracking within the detected bounding boxes. They also incorporate object instance segments for track modelling by applying the maskRCNN detector. They present a novel motion coding scheme that anchors the LSTM structure to effectively model the motion and relative position between objects in a single representation scheme. The proposed motion representation and deep features representing objects appearances are fused in an embedded space learned by the hierarchical LSTM structure for predicting the object to track association. The authors present experimental validation of the proposed approach on multiple object tracking challenge datasets and demonstrate that their solution naturally deals with major tracking challenges under all uncertainties. [ABSTRACT FROM AUTHOR]

Gesture is a natural form of human communication, and it is of great significance in human–computer interaction. In the dynamic gesture recognition method based on deep learning, the key is to obtain comprehensive gesture feature information. Aiming at the problem of inadequate extraction of spatiotemporal features or loss of feature information in current dynamic gesture recognition, a new gesture recognition architecture is proposed, which combines feature fusion network with variant convolutional long short‐term memory (ConvLSTM). The architecture extracts spatiotemporal feature information from local, global and deep aspects, and combines feature fusion to alleviate the loss of feature information. Firstly, local spatiotemporal feature information is extracted from video sequence by 3D residual network based on channel feature fusion. Then the authors use the variant ConvLSTM to learn the global spatiotemporal information of dynamic gesture, and introduce the attention mechanism to change the gate structure of ConvLSTM. Finally, a multi‐feature fusion depthwise separable network is used to learn higher‐level features including depth feature information. The proposed approach obtains very competitive performance on the Jester dataset with the classification accuracies of 95.59%, achieving state‐of‐the‐art performance with 99.65% accuracy on the SKIG (Sheffifield Kinect Gesture) dataset. [ABSTRACT FROM AUTHOR]

In human tracking, sparse representation successfully localises the human in a video with minimal reconstruction error using target templates. However, the state‐of‐the‐art approaches use colour and local appearance of a human to discriminate the human from the background regions, and hence fail when the human is occluded and appears in the varying illumination environment. In this study, a robust tracking algorithm is proposed that utilises gradient orientation and fine and coarse sparse representation of the target template. Sparse representation‐based human appearance model utilises weighted gradient orientation that is insensitive to illumination variation. Coarse and fine representation of sparse code facilitates tracking under varying scales. Subspace learning from image gradient orientation is enforced with occlusion detection during the dictionary updation stage to capture the visual characteristics of the local human appearance that supports tracking under partial occlusion with lesser tracking error. The proposed human tracking algorithm is evaluated on various datasets and shows efficient human tracking performance when compared to the other state‐of‐the‐art approaches. Furthermore, the proposed human tracking algorithm is suitable for surveillance applications. [ABSTRACT FROM AUTHOR]

The texture is a repetition of a particular structure. Textures classified into static texture and dynamic texture. There are two approaches for synthesising dynamic textures: image‐based approach and physics‐based approach. In the proposed work, the synthesis of different dynamic textures is shown. The physics‐based approach is used to synthesise dynamic texture videos. Different mathematical models are proposed which are suitable to give appropriate motion to the dynamic textures. The raw frames are created, and these frames are further used to synthesise the dynamic textures using physics laws and mathematical formulae. The flexibility of each model is demonstrated. The proposed model has less specificity and less computational complexity. The proposed algorithms are implemented on the graphics processing unit to reduce the overall execution time and time complexity. High‐quality videos are produced, and the evaluation of every frame assures the quality. [ABSTRACT FROM AUTHOR]

Colour grading is an essential step in movie post‐production, which is done in the industry by experienced artists on expensive edit hardware and software suites. This paper presents a review of the advances made to automate this process. The review looks in particular at how the state‐of‐the‐art in optimal transport and deep learning has advanced some of the fundamental problems of colour transfer, and how far are we still from being able to automatically grade images. [ABSTRACT FROM AUTHOR]

An end‐to‐end pipeline for the creation of high‐quality animatable volumetric video of human performances is presented. Going beyond the application of free‐viewpoint video, the authors allow re‐animation and alteration of an actor's performance through the enrichment of the captured data with semantics and animation properties. Hybrid geometry‐ and video‐based animation methods are applied that allow a direct animation of the high‐quality data itself instead of creating a CG model that resembles the captured data. Semantic enrichment and animation are achieved by establishing temporal consistency followed by automatic rigging of each 3D frame using a parametric human body model. The hybrid approach combines the flexibility of classical CG animation with the realism of real captured data. For the face, coarse movements are modelled in the geometry only, while very fine and subtle details, often lacking in purely geometric methods, are captured in video textures, which can interactively be combined to form new facial expressions. On top of that, regions that are challenging to synthesise, such as the teeth or the eyes, are learned and filled in realistically in an autoencoder‐based approach. This study covers the full pipeline from capturing, volumetric video production, and enrichment with semantics for the final hybrid animation. [ABSTRACT FROM AUTHOR]

Video‐based person re‐identification aims to verify the pedestrian identity from image sequences. The sequences are captured by cameras located in different directions at different times. Existing studies have certain limitations in the case of occlusions and pose variations. To solve the aforementioned problems, this study proposes a new two‐stage framework, from which the key‐image‐based fusion spatial–temporal feature (KISTF) of the pedestrian can be extracted from the video. The image‐level features at all timestamps are aggregated into the sequence‐level feature representation of the video by using an long short‐term memory network. Additionally, the concept of key image is defined for the image sequence, and the frame‐level feature of the pedestrian is extracted from these key images. The proposed spatial–temporal feature, KISTF, is obtained by fusing the sequence‐level feature and the frame‐level feature. It aims to solve the problem of pedestrian representation in small video data sets. Experiments are conducted on the iLIDS‐VID and PRID2011 data sets. The results demonstrate that the proposed approach outperforms state‐of‐the‐art video‐based re‐identification methods. [ABSTRACT FROM AUTHOR]

Frame interpolation is the process of synthesising a new frame in‐between existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects. In the context of this special issue, this study provides a review of the technology used to create in‐between frames and presents a Bayesian framework that generalises frame interpolation algorithms using the concept of motion interpolation. Unlike existing literature in this area, the authors also compare performance using the top industrial toolkits used in the post production industry. They find that all successful techniques employ motion‐based interpolation, and the commercial version of the Bayesian approach performs best. Another goal of this study is to compare the performance gains with recent convolutional neural network (CNN) algorithms against the traditional explicit model‐based approaches. They find that CNNs do not clearly outperform the explicit motion‐based techniques, and require significant compute resources, but provide complementary improvements in certain types of sequences. [ABSTRACT FROM AUTHOR]

This study proposes a three‐stream model using two different types of deep convolutional neural networks (CNNs): (i) a spatial stream with a CNN on images; (ii) a ResNet (residual network) on optical flows; and, (iii) a ResNet on the concatenation of motion features. This model is applied to four datasets: (i) UCF Sports; (ii) Youtube Sports; (iii) SBU action interaction; and (iv) a subset of the UCF‐1M Sports. Using two optical flow estimation methods: (i) a motion boundary emphasised Epicflow (Edge Preserving Interpolation Correspondences for Optical Flow) method, (MBEpicflow); and (ii) the Flownet 2 method, a learning optical flow estimation method. It was found that (i) the proposed MBEpicflow outperforms the Flownet 2 method on the SBU dataset, while the Flownet 2 performs equally well or better than the MBEpicflow method on the other three datasets, and these results are the best when compared with those obtained using other approaches on all datasets evaluated. These results showed the importance of accurate optical flow plays in human action recognition, an aspect which has been seldom addressed. Moreover, it showed that if some measure of the global behaviours of motion is incorporated, the generalisation performance is often improved by 1–2%. [ABSTRACT FROM AUTHOR]

Most existing methods of motion assessment system used the contact sensor, infrared sensor, and depth sensor, and few works provided the solution of digital camera. To solve this problem, the authors propose a new motion assessment system based on light camera. In this work, the motion assessment was regarded as pattern regression problem of skeleton joint trajectory. Firstly, the system uses the camera to capture the image sequences. The pose estimation method is used to obtain body skeleton from image. Secondly, due to the difference of motion frequency of each person, the length of the image sequences is different and the length of each joint trajectory also will be different. Fourier transform is applied to normalise the trajectory and use the coefficients of Fourier transform as the joint trajectory feature. Finally, the regression model is built to assess the motion quality. Some experimental results and discussion on action video data are used to verify the effectiveness of the system. [ABSTRACT FROM AUTHOR]

The pressing demand for workload along with social media interaction leads to diminished alertness during work hours. Researchers attempted to measure alertness level from various cues like EEG, EOG, video‐based eye movement analysis, etc. Among these, video‐based eyelid and iris motion tracking gained much attention in recent years. However, most of these implementations are tested on video data of subjects without spectacles. These videos do not pose a challenge for eye detection and tracking. In this work, the authors have designed an experiment to yield a video database of 58 human subjects wearing spectacles and are at different levels of alertness. Along with spectacles, they introduced variation in session, recording frame rate (fps), illumination, and time of the experiment. They carried out an analysis to detect the reliableness of facial and ocular features like yawning and eye‐blinks in the context of alertness level detection capability. Also, they observe the influence of spectacles on ocular feature detection performance under spectacles and propose a simple preprocessing step to alleviate the specular reflection problem. Extensive experiments on real‐world images demonstrate that the authors' approach achieves desirable reflection suppression results within minimum execution time compared to the state‐of‐the‐art. [ABSTRACT FROM AUTHOR]

Compared with conventional image sensors, event cameras have been attracting attention thanks to their potential in environments under fast motion and high dynamic range (HDR). To tackle the lost‐track issue due to fast illumination changes under HDR scene such as tunnels, an object tracking framework has been presented based on event count images from an event camera. The framework contains an offline‐trained detector and an online‐trained tracker which complement each other: The detector benefits from pre‐labelled data during training, but may have false or missing detections; the tracker provides persistent results for each initialised object but may suffer from drifting issues or even failures. Besides, process and measurement equations have been modelled, and a Kalman fusion scheme has been proposed to incorporate measurements from the detector and the tracker. Self‐initialisation and track maintenance in the fusion scheme ensure autonomous real‐time tracking without user intervene. With self‐collected event data in urban driving scenarios, experiments have been conducted to show the performance of the proposed framework and the fusion scheme. [ABSTRACT FROM AUTHOR]

The term 'collective' is used to refer to a group of animals such as a flock of birds or a herd of elephants. Aggregate motions of such collectives often give rise to visually pleasing shapes and patterns (e.g. V-shape formation of geese while they migrate from one place to another). While shapes of moving collectives are of great interest in many scientific studies, scant attention has been given to algorithmically extract and render these shapes via polygonal boundaries or graphs. In this Letter, the authors present a multi-stage, proof of concept framework for tracking geometric shapes and extracting video frames containing a user defined shape of moving collectives, by employing a deep-learning based object detection, well-known alpha shapes and a modified shape context. They demonstrate the usefulness of the proposed framework on a couple of test videos and discuss its potential applications in a wider area. [ABSTRACT FROM AUTHOR]

Background subtraction in traffic surveillance videos plays a crucial role in many high-level analytics and applications. Although great progress has been made in background subtraction, there are still many challenges in real traffic surveillance circumstances, such as camouflaged vehicles and dynamic background. This paper proposes a universal and accurate sampled-based vehicle detection method. The proposed method uses the color and Haar features to construct the background model, which can increase the accuracy of vehicle detection for camouflaged vehicles and is robust to low visibility. Besides, to reduce the incorrect samples of the initialized background model from a single image, the samples of the initial background model are randomly chosen from a similar candidate set. Furthermore, a novel random strategy is proposed to update the background pixel itself, while a combination update strategy with certainty and randomness is adopted for its neighborhood. This updating mechanism speeds up the suppression of ghosts in the background model. Experimental results verify the excellent behavior of our proposed method when compared to other mainstream methods. [ABSTRACT FROM AUTHOR]

Fish‐cage dysfunction in aquaculture installations can trigger significant negative consequences affecting the operational costs. Low oxygen levels, due to excessive fooling's, leads to decrease growth performance, and feed efficiency. Therefore, frequent periodic inspection of fish‐cage nets is required, but this task can become quite expensive with the traditional means of employing professional divers that perform visual inspections at regular time intervals. The modern trend in aquaculture is to take advantage of IT technologies with the use of a small‐sized, low‐cost autonomous underwater vehicle, permanently residing within a fish cage and performing regular video inspection of the infrastructure for the entire net surface. In this study, we explore specialised image processing schemes to detect net holes of multiple area size and shape. These techniques are designed with the vision to provide robust solutions that take advantage of either global or local image structures to provide the efficient inspection of multiple net holes. [ABSTRACT FROM AUTHOR]

Action detection and recognition are popular subjects of research in the field of computer vision. The task of action detection can be regarded as the sum of action location and recognition. Action features described by using information concerning the human skeleton have the advantages of robustness against external factors and requiring a small amount of calculation. This study proposes a skeleton-based action analysis model based on a recurrent neural network framework. The model learns action features by modelling static and dynamic features of skeleton joints and the importance of different video frames by introducing an attention module. For action location, conditional random field loss function is introduced to establish the context dependency of output labels. In the aspect of action recognition, the hierarchical training mechanism with triple loss models action features at coarse-grained and fine-grained levels. The authors' proposed method delivers state-of-the-art results on action location and recognition tasks. [ABSTRACT FROM AUTHOR]

Polyps are a group of cells growing on the inner surface of the colon. Over time, some polyps can lead to colon cancer, which is often fatal if found in its later stages. Colon cancer can be prevented if the polyps are identified and removed in their early stages. Colonoscopy is a very effective screening method to remove polyps and it largely prevents colon cancer. However, some polyps may not be detected during a colonoscopy due to human error. Over the past two decades, many studies have been conducted on computer-aided detection to reduce the miss rate of polyps. This study consists of two distinct parts, the detection of frames containing polyps and polyp segmentation. In the first section, a new convolutional neural network based on the VGG network is proposed. The proposed network has an accuracy of 86% on a newly collected dataset. In the polyp segmentation section, a fully convolutional network and an effective post-processing algorithm are presented. An evaluation of the proposed polyp segmentation system on the ETIS-LARIB database achieves an overall 82.00% F2 score, which outperforms the methods that participated in the sub-challenge of MICCAI. [ABSTRACT FROM AUTHOR]

In this Letter, the authors present a novel graph clustering-based method for crowd counting only using very limited labelled samples. Based on an intuitional observation that the distribution of low-level features of a specific scene containing the same or similar number of pedestrians are close to each other in the feature space, the authors adopt a first neighbour propagation (FNP) based clustering method to divide all unlabelled data into different groups. Next, an active sampling learning strategy that measures representativeness and diversity of the training data is used to obtain a few informative samples for annotation. Finally, the counts of those labelled informative samples are effectively propagated to predict the unlabelled samples in the constructed clusters by FNP-based clustering. The compelling results on two benchmark datasets demonstrate that the proposed method is not only effective to estimate crowd counts with very few labelled samples but also applicable to annotate a large number of unknown video frames for scene-specific crowd counting models. [ABSTRACT FROM AUTHOR]

Detection of abnormal events in the traffic scene is very challenging and is a significant problem in video surveillance. The authors proposed a novel scheme called super orientation optical flow (SOOF)‐based clustering for identifying the abnormal activities. The key idea behind the proposed SOOF features is to efficiently reproduce the motion information of a moving vehicle with respect to superorientation motion descriptor within the sequence of the frame. Here, the authors adopt the mean absolute temporal difference to identify the anomalies by motion block (MB) selection and localisation. SOOF features obtained from MB are used as motion descriptor for both normal and abnormal events. Simple and efficient K‐means clustering is used to study the normal motion flow during the training. The abnormal events are identified using the nearest‐neighbour searching technique in the testing phase. The experimental outcome shows that the proposed work is effectively detecting anomalies and found to give results better than the state‐of‐the‐art techniques. [ABSTRACT FROM AUTHOR]

Natural scene text usually carries considerable abstract semantic information, which is closely related to the surrounding environment. Thus, natural scene text detection plays a vital role in image content retrieval and understanding. In this study, the authors propose a novel specific category region proposal network (SCRPN) based on maximally stable extremal regions (MSER) and fully convolutional network (FCN) for natural scene text detection. First, FCN for pixel‐level recognition is utilised to obtain the text saliency map and MSER is used to obtain oversegmented regions. Then, the multiple features of oversegmented regions and text saliency map are used for region aggregation. Next, single‐linkage clustering method is adopted to cluster the segmentation regions to obtain a hierarchical structure of text region proposals. Finally, for the top‐ranking region proposals, SCRPN built an end‐to‐end pipeline for scene text detection directly. Experiments on street view text and international conference on document analysis and recognition (ICDAR) 2013 have demonstrated the effectiveness of SCRPN for generating the text proposals. SCRPN could work with various two‐stage text detection networks; thus, faster region convolutional neural network was used as the text detection framework to evaluate the performance of SCRPN in the ICDAR 2015 and MSRA‐TD500 benchmarks. The experimental results confirmed that SCRPN makes text detection more robust in complex scenarios. [ABSTRACT FROM AUTHOR]

In this paper, we propose a motion preprocessing method for the use in the game application pipeline, which is composed of two stages: the proposed preprocessing method and the High Efficiency Video Coding (HEVC) encoder. The method accepts the object information from the game application, preprocesses the motion vectors of objects, and pass the preprocessed motion data to the HEVC encoder. The HEVC encoder takes the motion data as the initial (or the dedicated) value of motion estimation. Therefore, the traditional diamond search can be skipped and hence increase the encoding performance of the HEVC encoder. In the motion preprocessing method, the following three steps are taken: a coordination system transformation, determining motion vectors for $4\times 4$ -checkerboard blocks [Atomic Block (AB)], and the selection of proper motion vectors for all varieties of prediction units in the encoder. With a focus on the special issues, such as move-out zone and bi-directional prediction, we are able to further optimize the performance of the encoder. We examined two types of 2D gaming scenes in our experiments. The experimental results show that, as compared with the original diamond search method provided by the encoder, our algorithm is able to achieve up to 49.0% time reduction of video encoding. The Bjontegaard Delta bit rate can achieve up to −17.0% in the random_access mode while combining with the $\times 265$ encoder and up to −26.2% in the lowdelay mode while combining with the HM-16 encoder. [ABSTRACT FROM AUTHOR]

In this study, the authors propose a novel network architecture to address the problem of removing rain streaks from single images. To strengthen the representational power of the network, they adopt the squeeze‐and‐excitation block in the network. Furthermore, they propose a new network connection called reusing original input (ROI). The ROI connection reuses the original input of the network and can provide more texture details of the background. These details can be useful for the restoration of the image after removing the rain streaks. Batch normalisation is applied to further improve the rain removal performance of the network. Despite the fact that the network is trained on synthetic data, experimental results show that the proposed network has a comparable performance on both synthetic images and real‐world images to the state‐of‐the‐art methods. [ABSTRACT FROM AUTHOR]

Similar to the basic facial expression recognition, one challenge for pain intensity recognition is some individual characteristics, e.g. face shapes, may cause great diversities in the same emotion. So it is usually very difficult to distinguish two adjacent intensity levels of pain expression as each intensity has a large variation. In this study, a coarse‐to‐fine combination method is proposed for pain intensity recognition. The results of multi‐scale outputs from multiple base deep network are combined in a probabilistic way for improving the discrimination between visually similar adjacent levels. A two‐layer tree classifier is proposed in a multi‐task framework for pain intensity recognition as well as face shape recognition, replacing the planar Softmax classifier in each base deep network. In the first layer of tree classifier, multi‐scale classifiers are constructed for recognizing facial pain intensities and the conventional classifiers are constructed for face shape recognition in the second layer. Finally, the tree classifier including multi‐scale classifiers and conventional classifiers is jointly optimised during the training phase and only high level classifiers are used for recognising pain intensities in the test phase. The extensive experiments on UNBC shoulder pain dataset show the proposed method gets promising results in pain intensity recognition. [ABSTRACT FROM AUTHOR]

This study presents a technique to automatically extract the aircraft from video sequences. The technique incorporates object tracking algorithm to separate the region of interest (ROI), i.e. movement of aircraft, from video sequence, to reduce the computational complexity. Trimaps are generated automatically for given ROI using non‐rigid image registration to obtain the prior information of the object and alpha mattes are estimated for each frame using KNN matting algorithm. However, the alpha matte of static region is estimated only once, that is combined with the alpha matte of moving region to generate final alpha matte. Simulations performed on different videos show that the proposed technique not only reduces the computational complexity, but also gives competitive results by generating high quality alpha maps. [ABSTRACT FROM AUTHOR]

In this study, the authors present a new approach to segment and classify moving objects in video sequences by combining an unsupervised anomaly discovery framework called DeepSphere and generative adversarial networks. The proposed deep detector classifier employs and validates DeepSphere, which aims mainly to identify the anomalous cases in the spatial and temporal context in order to perform foreground objects segmentation. For post‐processing, some morphological operations are considered to better segment and extract the desired objects. Finally, they take advantage of the power of generative models, which recognise the problem of semi‐supervised learning as a specific missing data imputation task in order to classify the segmented objects. They evaluate the method with multiple datasets and the results confirm the effectiveness of the proposed approach, which achieves superior performance over the state‐of‐the‐art methods having the capabilities of segmenting and classifying moving objects from videos surveillance. [ABSTRACT FROM AUTHOR]

Fire detection systems in ships are based on smoke and heat detection in accordance with safety regulations. The rapid advancement of machine vision technology has led to the development of video smoke detection (VSD) systems. In this study, a VSD system is applied to smoke detection within the engine room of the ship. A dataset for a range of scenarios was created with a smoke generator. The method for smoke detection was based on motion detection and a support vector machine classifier, which was used to make candidate regions and perform classification. A local binary pattern descriptor was used to extract the feature vector. A training set was made from a variety of video frames, randomly. Experimental results seldom produced false positive windows in the non‐smoke region. However, if the greyscale value of difference image between background and the smoke is lower than the setting value for motion detection, the system could not detect smoke. Processing time is sufficiently fast for use in real‐time smoke detection systems. To install a VSD system on‐board a vessel, the authors recommend a performance standard of the system which must be met. [ABSTRACT FROM AUTHOR]

The authors consider the problem of compressive sensed video recovery via iterative thresholding algorithm. Traditionally, it is assumed that some fixed sparsifying transform is applied at each iteration of the algorithm. In order to improve the recovery performance, at each iteration the thresholding could be applied for different transforms in order to obtain several estimates for each pixel. Then the resulting pixel value is computed based on obtained estimates using simple averaging. However, calculation of the estimates leads to significant increase in reconstruction complexity. Therefore, the authors propose a heuristic approach, where at each iteration only one transform is randomly selected from some set of transforms. First, they present simple examples, when block‐based 2D discrete cosine transform is used as the sparsifying transform, and show that the random selection of the block size at each iteration significantly outperforms the case when fixed block size is used. Second, building on these simple examples, they apply the proposed approach when video block‐matching and 3D filtering (VBM3D) is used for the thresholding and show that the random transform selection within VBM3D allows to improve the recovery performance as compared with the recovery based on VBM3D with fixed transform. [ABSTRACT FROM AUTHOR]

During the process of watching 3D synthesised video (3D‐SV) and switching viewpoints, there is a case of asymmetric distortion, the left(right) viewpoint is a synthesised video generated by rendering technique, and the right(left) viewpoint is a real video taken by the camera. How to accurately estimate the quality of 3D‐SV with binocular asymmetric distortions is a new and challenging problem. Aiming at this problem, a blind quality assessment method for 3D‐SV with binocular asymmetric distortions is proposed. Firstly, the local edge deformations of synthesised videos at different scales are measured by calculating their standard deviations. Secondly, the global naturalness of synthesised videos is computed by analysing their natural statistical characteristics. Thirdly, a strategy for fusing left and right quality scores is proposed, which considers their texture information in different directions. Finally, the random forest is used to obtain an objective quality score. The experimental results show the superiority of the proposed method on asymmetry 3D‐SV database. [ABSTRACT FROM AUTHOR]

In many scenarios, where videos are transmitted through bandwidth‐limited channels for subsequent semantic analytics, the choice of frame rates has to balance between bandwidth constraints and analytics performance. Faced with this practical challenge, this study focuses on enhancing object tracking at low frame rates and proposes a learning Interpolation for tracking framework. This framework embeds an implicit video frame interpolation sub‐network, which is concatenated and jointly trained with another object tracking sub‐network. Once a low frame‐rate video is an input, it is first mapped into a high frame‐rate latent video, based on which the tracker is learned. Novel strategies and loss functions are derived to ensure the effective end‐to‐end optimisation of the authors' network. On several challenging benchmarks and settings, their method achieves a highly competitive tradeoff between frame rate and tracking accuracy. As is known, the implications of interpolation on semantic video analytics and tracking remain unexplored, and the authors expect their method to find many applications in mobile embedded vision, Internet of Things and edge computing. [ABSTRACT FROM AUTHOR]

Action recognition has been an active research area for many years. Extracting discriminative spatial and temporal features of different actions plays a key role in accomplishing this task. Current popular methods of action recognition are mainly based on two‐stream Convolutional Networks (ConvNets) or 3D ConvNets. However, the computational cost of two‐stream ConvNets is high for the requirement of optical flow while 3D ConvNets takes too much memory because they have a large amount of parameters. To alleviate such problems, the authors propose a Dual Attention ConvNet (DANet) based on dual attention mechanism which consists of spatial attention and temporal attention. The former concentrates on main motion objects in a video frame by using ConvNet structure and the latter captures related information of multiple video frames by adopting self‐attention. Their network is entirely based on 2D ConvNet and takes in only RGB frames. Experimental results on UCF‐101 and HMDB‐51 benchmarks demonstrate that DANet gets comparable results among leading methods, which proves the effectiveness of the dual attention mechanism. [ABSTRACT FROM AUTHOR]

Salient object detection has been revolutionised by convolutional neural network (CNN) recently. However, it is hard to transfer the state‐of‐the‐art still‐image based saliency detectors to videos directly, owing to the neglect of temporal contexts between frames. In this study, the authors propose a flow‐driven attention network (FDAN) to exploit motion information for video salient object detection. FDAN consists of an appearance feature extractor, a motion‐guided attention module and a saliency map regression module. It extracts the appearance feature per frame, refines appearance feature with optical flow and infers the ultimate saliency map, respectively. Motion‐guided attention module is the core of FDAN, which extracts motion information in the form of attention. This attention mechanism is a two‐branch CNN, fusing optical flow and appearance features. In addition, a shortcut connection is applied to the attention multiplied feature map for noise suppression intensively. Experimental results show that the proposed method can achieve performance on par with the state‐of‐the‐art method flow‐guided recurrent neural encoder on challenging benchmarks of Densely Annotated Video Segmentation and Freiburg–Berkeley Motion Segmentation while being two times faster in detection. [ABSTRACT FROM AUTHOR]

In order to improve the accuracy of small‐range human motion recognition in video and the computational efficiency of large‐scale data sets, a multi‐dimensional data model of motion recognition and motion capture in video image based on deep‐learning framework was proposed. First, the moving foreground of the target is extracted by the Gauss mixture model, and the human body is recognised by the gradient histogram. At the second level, the dense trajectory feature and the deep learning feature are fused, according to the integration of global encoding algorithm and convolutional neural network. In the deep learning feature, the fusion of the deep video feature and the video RGB tricolour feature is taken as the feature of deep learning. Finally, the classification is based on the deep learning network model. The simulation experiments based on large‐scale real data sets and small‐scale gesture data sets show that the algorithm has high recognition accuracy for large‐scale data sets and small‐scale gesture actions. In addition, Imperial Computer Vision & Learning Lab human behaviour data set is used to classify the experimental data. The average classification accuracy is 85.79%. The algorithm can run at a speed of about 20 frames per second. [ABSTRACT FROM AUTHOR]

Body mass index (BMI) defines as a person's weight divided by the square of height (BMI =(weight(lb)/height(in)2)×703), which is an important indicator of the health condition. The authors study BMI estimation from the three‐dimensional (3D) visual data by measuring the correlation between the estimated body volume and BMIs, and then develop an efficient BMI computation method. Their approach consists of body weight and height estimation from normally dressed people in 3D space. To address the influence of loose clothes on body volume estimation, two clothes models are developed to make the volume estimation more accurate. A new RGB‐D video dataset is collected for this study, and the reconstructed 3D data are provided by the KinectFusion on depth data. Experimental results show the effectiveness of the approach to work on normal conditions of dressed people. The mean absolute error of the estimated BMI can achieve 2.54 in their experiments. [ABSTRACT FROM AUTHOR]

There is an urgent need for a robust video quality assessment (VQA) model that can efficiently evaluate the quality of a video content varying in terms of the distortion and content type in the absence of the reference video. Considering this need, a novel no reference (NR) model relying on the spatiotemporal statistics of the distorted video in a three‐dimensional (3D)‐discrete cosine transform (DCT) domain is proposed in this study. While developing the model, as the first contribution, the video contents are adaptively segmented into the cubes of different sizes and spatiotemporal contents in line with the human visual system (HVS) properties. Then, the 3D‐DCT is applied to these cubes. Following that, as the second contribution, different efficient features (i.e. spectral behaviour, energy variation, distances between spatiotemporal frequency bands, and DC variation) associated with the contents of these cubes are extracted. After that, these features are associated with the subjective experimental results obtained from the EPFL‐PoliMi video database using the linear regression analysis for building the model. The evaluation results present that the proposed model, unlike many top‐performing NR‐VQA models (e.g. V‐BLIINDS, VIIDEO, and SSEQ), achieves high and stable performance across the videos with different contents and distortions. [ABSTRACT FROM AUTHOR]

Human motion analysis has received a lot of attention in the computer vision community during the last few years. This research domain is supported by a wide spectrum of applications including video surveillance, patient monitoring systems, and pedestrian detection, to name a few. In this study, an improved cascaded design for human motion analysis is presented; it consolidates four phases: (i) acquisition and preprocessing, (ii) frame segmentation, (iii) features extraction and dimensionality reduction, and (iv) classification. The implemented architecture takes advantage of CIE‐Lab and National Television System Committee colour spaces, and also performs contrast stretching using the proposed red–green–blue* colour space enhancement technique. A parallel design utilising attention‐based motion estimation and segmentation module is also proposed in order to avoid the detection of false moving regions. In addition to these contributions, the proposed feature selection technique called entropy controlled principal components with weights minimisation, further improves the classification accuracy. The authors claims are supported with a comparison between six state‐of‐the‐art classifiers tested on five standard benchmark data sets including Weizmann, KTH, UIUC, Muhavi, and WVU, where the results reveal an improved correct classification rate of 96.55, 99.50, 99.40, 100, and 100%, respectively. [ABSTRACT FROM AUTHOR]

Nowadays, live broadcasting video has become increasingly popular and high‐quality live broadcasting video is highly needed. In practice, live broadcasting videos usually undergo several processing stages, which inevitably introduce multiple distortions, e. g. frame freezing and intensity mutation, causing the degraded quality of experience. However, little work has been done to the quality evaluation of live broadcasting videos, which may hinder the further development of more advanced live broadcasting video delivery systems. Motivated by this, this study presents a no‐reference quality evaluation model for live broadcasting videos (LBVQA) in temporal and spatial domains. In the temporal domain, statistic features are extracted to measure the frame freezing and intensity mutation, and the entropy‐based feature is extracted to describe the global jitter. In the spatial domain, blurring is measured based on phase coherence, and abnormal exposure ratio is calculated based on an adaptive threshold. Finally, all features are fed into a backpropagation neural network to train the quality prediction model. Experimental results on the Live Broadcasting Video Database demonstrate the advantages of the proposed metric over the state‐of‐the‐art image and video quality metrics. [ABSTRACT FROM AUTHOR]

Video summarisation and key‐frame extraction play a vital role in video processing and content‐based video analysis. Key‐frame extraction facilitates rapid browsing and proficient video ordering in numerous applications such as multimedia, gaming etc. The application of 'effective key‐frame extraction' can be used in the field of object detection and tracking, and it is found to be helpful in providing some sort of signal to visually impaired persons. In this study, the advantages of Thepade's sorted ternary block truncation coding (TSTBTC) along with binary bat algorithm (BBA) are used to overcome the limitations by extracting an effective key frame from the videos. Static images in the form of frames are extracted from input video database and processed through TSTBTC algorithm for calculating similarity measures amongst two consecutive frames. The input data is processed using four colour spaces, namely red–green–blue colour space, Kekre's LUV colour space, YCbCr colour space, YUV colour space and BBA are used to optimise the threshold value. Furthermore, five similarity measures indices are used to calculate the number of frames, and the result obtained shows that TSTBTC–BBA algorithm deployed in YUV colour space provides better results compared with other existing techniques regarding precision, F‐measure and colour spaces. [ABSTRACT FROM AUTHOR]

Vision-based structural motion estimation methods show great potential in structural health monitoring (SHM). However, available methods cannot automatically estimate and eliminate three-dimensional (3D) camera motion effects caused by environmental factors from the video containing structural motions. This limits further applications of vision-based methods in SHM. To this end, this paper proposed a target-free video measurement method, which is formulated as a feature point congealing problem, and its solution includes two steps. First, the links of feature points from stationary backgrounds are assumed to have consistent and smooth motions, and are detected by a maximum a posteriori (MAP) estimation of the Bayesian model. Second, based on the detected links, the joint homography matrix containing only 3D camera motion effects can be updated iteratively, by minimizing transformation differences between the current frame and other frames. The superiority of the proposed method over traditional methods was validated in case studies of Humen bridge motion estimations. It was shown the proposed method has the best performance in video stabilization and camera motion effects estimation. Moreover, combined with the phase-based method, subpixel small structural motions can be well estimated. • The proposed method can estimate 3D camera motion effects from the video containing structural motions. • When using the proposed method, manual selections and trainings can be avoided. • The proposed method was validated in estimaing 3D camera motion effects imposed with earthquake motions. [ABSTRACT FROM AUTHOR]