Search

This study aims to enable deaf and hard-of-hearing people to communicate with other individuals who know and do not know sign language. The mobile application was developed for video classification by using MediaPipe Library in the study. While doing this, considering the problems that deaf and hearing loss individuals face in Turkey and abroad modelling and training stages were carried out with the English language option. With the real-time translation feature added to the study individuals were provided with instant communication. In this way, communication problems experienced by hearing-impaired individuals will be greatly reduced. Machine learning and Deep learning concepts were investigated in the study. Model creation and training stages were carried out using VGG16, OpenCV, Pandas, Keras, and Os libraries. Due to the low success rate in the model created using VGG16, the MediaPipe library was used in the formation and training stages of the model. The reason for this is that, thanks to the solutions available in the MediaPipe library, it can normalise the coordinates in 3D by marking the regions to be detected in the human body. Being able to extract the coordinates independently of the background and body type in the videos in the dataset increases the success rate of the model in the formation and training stages. As a result of an experiment, the accuracy rate of the deep learning model is 85% and the application can be easily integrated with different languages. It is concluded that deep learning model is more accure than machine learning one and the communication problem faced by hearing-impaired individuals in many countries can be reduced easily. [ABSTRACT FROM AUTHOR]

Understanding road conditions is essential for implementing effective road safety measures and driving solutions. Road situations encompass the day-to-day conditions of roads, including the presence of vehicles and pedestrians. Surveillance cameras strategically placed along streets have been instrumental in monitoring road situations and providing valuable information on pedestrians, moving vehicles, and objects within road environments. However, these video data and information are stored in large volumes, making analysis tedious and time-consuming. Deep learning models are increasingly utilized to monitor vehicles and identify and evaluate road and driving comfort situations. However, the current neural network model requires the recognition of situations using time-series video data. In this paper, we introduced a multi-directional detection model for road situations to uphold high accuracy. Deep learning methods often integrate long short-term memory (LSTM) into long-term recurrent network architectures. This approach effectively combines recurrent neural networks to capture temporal dependencies and convolutional neural networks (CNNs) to extract features from extensive video data. In our proposed method, we form a multi-directional long-term recurrent convolutional network approach with two groups equipped with CNN and two layers of LSTM. Additionally, we compare road situation recognition using convolutional neural networks, long short-term networks, and long-term recurrent convolutional networks. The paper presents a method for detecting and recognizing multi-directional road contexts using a modified LRCN. After balancing the dataset through data augmentation, the number of video files increased, resulting in our model achieving 91% accuracy, a significant improvement from the original dataset. [ABSTRACT FROM AUTHOR]

Taekwondo is a widely practised martial art and an Olympic sport. In Taekwondo, Poomsae movements are essential, as they form the foundation of the sport and are fundamental for success in competitions. The evaluation of Poomsae movements in Taekwondo has been a subjective process, relying heavily on human judgments. This study addresses the above issue by developing a systematic approach to evaluate Poomsae movements using computer vision. A long short-term memory-based (LSTM-based) machine learning (ML) model was developed and evaluated for its effectiveness in Poomsae movement evaluation. The study also aimed to develop this model as an assistant for self-evaluation, that enables Taekwondo players to enhance their skills at their own pace. For this study, a dataset was created specially by recording Poomsae movements of Taekwondo players from the University of Colombo. The technical infrastructure used to capture skeleton point data was cost-effective and easily replicable in other settings. Small video clips containing Taekwondo movements were recorded using a mobile phone camera and the skeleton point data was extracted using the MediaPipe Python library. The model was able to achieve 61% of accuracy when compared with the domain experts' results. Overall, the study successfully achieved its objectives of defining a selfpaced approach to evaluate Poomsae while overcoming human subjectivity otherwise unavoidable in manual evaluation processes. The feedback of domain experts was also considered to finetune the model for better performance. [ABSTRACT FROM AUTHOR]

This paper presents a spatiotemporal deep learning approach for mouse behavioral classification in the home-cage. Using a series of dual-stream architectures with assorted modifications for optimal performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. The dataset in focus is an annotated, publicly available dataset of a singly-housed mouse. We achieved even better classification accuracy by ensembling the best performing models; an Inception-based network and an attention-based network, both of which utilize this feature sharing attribute. Furthermore, we demonstrate through ablation studies that for all models, the feature sharing architectures consistently outperform the conventional dual-stream having standalone streams. In particular, the inception-based architectures showed higher feature sharing gains with their increase in accuracy anywhere between 6.59% and 15.19%. The best-performing models were also further evaluated on other mouse behavioral datasets. [ABSTRACT FROM AUTHOR]

Understanding road conditions is essential for implementing effective road safety measures and driving solutions. Road situations encompass the day-to-day conditions of roads, including the presence of vehicles and pedestrians. Surveillance cameras strategically placed along streets have been instrumental in monitoring road situations and providing valuable information on pedestrians, moving vehicles, and objects within road environments. However, these video data and information are stored in large volumes, making analysis tedious and time-consuming. Deep learning models are increasingly utilized to monitor vehicles and identify and evaluate road and driving comfort situations. However, the current neural network model requires the recognition of situations using time-series video data. In this paper, we introduced a multi-directional detection model for road situations to uphold high accuracy. Deep learning methods often integrate long short-term memory (LSTM) into long-term recurrent network architectures. This approach effectively combines recurrent neural networks to capture temporal dependencies and convolutional neural networks (CNNs) to extract features from extensive video data. In our proposed method, we form a multi-directional long-term recurrent convolutional network approach with two groups equipped with CNN and two layers of LSTM. Additionally, we compare road situation recognition using convolutional neural networks, long short-term networks, and long-term recurrent convolutional networks. The paper presents a method for detecting and recognizing multi-directional road contexts using a modified LRCN. After balancing the dataset through data augmentation, the number of video files increased, resulting in our model achieving 91% accuracy, a significant improvement from the original dataset.

Autism spectrum disorder (ASD), or autism, can be diagnosed based on a lack of behavioral skills and social communication. The most prominent method of diagnosing ASD in children is observing the child's behavior, including some of the signs that the child repeats. Hand flapping is a common stimming behavior in children with ASD. This research paper aims to identify children's abnormal behavior, which might be a sign of autism, using videos recorded in a natural setting during the children's regular activities. Specifically, this study seeks to classify self-stimulatory activities, such as hand flapping, as well as normal behavior in real-time. Two deep learning video classification methods are used to be trained on the publicly available Self-Stimulatory Behavior Dataset (SSBD). The first method is VGG-16-LSTM; VGG-16 to spatial feature extraction and long short-term memory networks (LSTM) for temporal features. The second method is a long-term recurrent convolutional network (LRCN) that learns spatial and temporal features immediately in end-to-end training. The VGG-16-LSTM achieved 0.93% on the testing set, while the LRCN model achieved an accuracy of 0.96% on the testing set. [ABSTRACT FROM AUTHOR]

In sports data analysis and visualization, understanding collective tactical behavior has become an integral part. Interactive and automatic data analysis is instrumental in making use of growing amounts of compound information. In professional team sports, gathering and analyzing sportsperson monitoring data are common practice, intending to evaluate fatigue and succeeding adaptation responses, analyze performance potential, and reduce injury and illness risk. Data visualization technology born in the era of big data analytics provides a good foundation for further developing fitness tools based on artificial intelligence (AI). Hence, this study proposed a video-based effective visualization framework (VEVF) based on artificial intelligence and big data analytics. This study uses the machine learning method to categorize the sports video by extracting both the videos' temporal and spatial features. Our system is based on convolutional neural networks united with temporal pooling layers. The experimental outcomes demonstrate that the recommended VEVF model enhances the accuracy ratio of 98.7%, recall ratio of 94.5%, F1-score ratio of 97.9%, the precision ratio of 96.7%, the error rate of 29.1%, the performance ratio of 95.2%, an efficiency ratio of 96.1% compared to other existing models. [ABSTRACT FROM AUTHOR]

Currently, the volume of sensitive content on the Internet, such as pornography and child pornography, and the amount of time that people spend online (especially children) have led to an increase in the distribution of such content (e.g., images of children being sexually abused, real-time videos of such abuse, grooming activities, etc.). It is therefore essential to have effective IT tools that automate the detection and blocking of this type of material, as manual filtering of huge volumes of data is practically impossible. The goal of this study is to carry out a comprehensive review of different learning strategies for the detection of sensitive content available in the literature, from the most conventional techniques to the most cutting-edge deep learning algorithms, highlighting the strengths and weaknesses of each, as well as the datasets used. The performance and scalability of the different strategies proposed in this work depend on the heterogeneity of the dataset, the feature extraction techniques (hashes, visual, audio, etc.) and the learning algorithms. Finally, new lines of research in sensitive-content detection are presented. [ABSTRACT FROM AUTHOR]

Lung ultrasound is used to detect various artifacts in the lungs that support the diagnosis of different conditions. There is ongoing research to support the automatic detection of such artifacts using machine learning. We propose a solution that uses analytical computer vision methods to detect two types of lung artifacts, namely A- and B-lines. We evaluate the proposed approach on the POCUS dataset and data acquired from a hospital. We show that by using the Fourier transform, we can analyze lung ultrasound images in real-time and classify videos with an accuracy above 70%. We also evaluate the method's applicability for segmentation, showcasing its high success rate for B-lines (89% accuracy) and its shortcomings for A-line detection. We then propose a hybrid solution that uses a combination of neural networks and analytical methods to increase accuracy in horizontal line detection, emphasizing the pleura. [ABSTRACT FROM AUTHOR]

Despite the great impact of lies in human societies and a meager 54% human accuracy for Deception Detection (DD), Machine Learning systems that perform automated DD are still not viable for proper application in real-life settings due to data scarcity. Few publicly available DD datasets exist and the creation of new datasets is hindered by the conceptual distinction between low-stakes and high-stakes lies. Theoretically, the two kinds of lies are so distinct that a dataset of one kind could not be used for applications for the other kind. Even though it is easier to acquire data on low-stakes deception since it can be simulated (faked) in controlled settings, these lies do not hold the same significance or depth as genuine high-stakes lies, which are much harder to obtain and hold the practical interest of automated DD systems. To investigate whether this distinction holds true from a practical perspective, we design several experiments comparing a high-stakes DD dataset and a low-stakes DD dataset evaluating their results on a Deep Learning classifier working exclusively from video data. In our experiments, a network trained in low-stakes lies had better accuracy classifying high-stakes deception than low-stakes, although using low-stakes lies as an augmentation strategy for the high-stakes dataset decreased its accuracy. • There is a clear distinction between low-stakes and high-stakes deception datasets. • Low-stakes lies may be used to classify high-stakes lies. • Combining low-stakes and high-stakes deception data does not yield good results. • Dataset bias is still an issue in video-based deception detection. • Use of automated deception detection in real-life settings is not recommended. [ABSTRACT FROM AUTHOR]

This paper presents a spatiotemporal deep learning approach for mouse behavioral classification in the home-cage. Using a series of dual-stream architectures with assorted modifications for optimal performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. The dataset in focus is an annotated, publicly available dataset of a singly-housed mouse. We achieved even better classification accuracy by ensembling the best performing models; an Inception-based network and an attention-based network, both of which utilize this feature sharing attribute. Furthermore, we demonstrate through ablation studies that for all models, the feature sharing architectures consistently outperform the conventional dual-stream having standalone streams. In particular, the inception-based architectures showed higher feature sharing gains with their increase in accuracy anywhere between 6.59% and 15.19%. The best-performing models were also further evaluated on other mouse behavioral datasets.

Attention mechanisms have significantly boosted the performance of video classification neural networks thanks to the utilization of perspective contexts. However, the current research on video attention generally focuses on adopting a specific aspect of contexts (e.g., channel, spatial/temporal, or global context) to refine the features and neglects their underlying correlation when computing attentions. This leads to incomplete context utilization and hence bears the weakness of limited performance improvement. To tackle the problem, this paper proposes an efficient attention-in-attention (AIA) method for element-wise feature refinement, which investigates the feasibility of inserting the channel context into the spatio-temporal attention learning module, referred to as CinST, and also its reverse variant, referred to as STinC. Specifically, we instantiate the video feature contexts as dynamics aggregated along a specific axis with global average and max pooling operations. The workflow of an AIA module is that the first attention block uses one kind of context information to guide the gating weights calculation of the second attention that targets at the other context. Moreover, all the computational operations in attention units act on the pooled dimension, which results in quite few computational cost increase (< 0.02%). To verify our method, we densely integrate it into two classical video network backbones and conduct extensive experiments on several standard video classification benchmarks. The source code of our AIA is available at https://github.com/haoyanbin918/Attention-in-Attention. [ABSTRACT FROM AUTHOR]

In this paper, a novel video classification method is presented that aims to recognize different categories of third-person videos efficiently. Our motivation is to achieve a light model that could be trained with insufficient training data. With this intuition, the processing of the 3-dimensional video input is broken to 1D in temporal dimension on top of the 2D in spatial. The processes related to 2D spatial frames are being done by utilizing pre-trained networks with no training phase. The only step which involves training is to classify the 1D time series resulted from the description of the 2D signals. As a matter of fact, optical flow images are first calculated from consecutive frames and described by pre-trained CNN networks. Their dimension is then reduced using PCA. By stacking the description vectors beside each other, a multi-channel time series is created for each video. Each channel of the time series represents a specific feature and follows it over time. The main focus of the proposed method is to classify the obtained time series effectively. Towards this, the idea is to let the machine learn temporal features. This is done by training a multi-channel one dimensional Convolutional Neural Network (1D-CNN). The 1D-CNN learns the features along the only temporal dimension. Hence, the number of training parameters decreases significantly which would result in the trainability of the method on even smaller datasets. It is illustrated that the proposed method could reach the state-of-the-art results on two public datasets UCF11, jHMDB and competitive results on HMDB51. [ABSTRACT FROM AUTHOR]

Digital forensics and analysis have emerged as a discipline to fight against cyber and computer-assisted crime. In particular, taking into account the increasing of unconstrained pornographic content over Internet and the spreading cases of Child Sex Abuse material distribution, there is a growing need of efficient computational tools to automatically detect or/and block pornographic videos. The primary objective of this study is to review the different strategies available in the literature for pornography detection in videos and identify research gaps. This survey shows that deep learning based techniques detect videos with sexually explicit content more accurately compared with other conventional detection strategies. The accuracy of the strategies reported in this work, is found to be dependent on features extraction techniques, architecture, and learning algorithms. Finally, further research areas in pornographic video detection are outlined. [ABSTRACT FROM AUTHOR]

Automated event and activity recognition in unconstrained videos has become a societal necessity. In this paper, we address video event classification and analyze the influence of preprocessing through action localization on the classification task. We propose an approach for event classification in videos, that is aided by unsupervised preprocessing through temporal attention and subsequent spatial action-localization at those specific attentive instants of time. The unsupervised temporal attention is achieved through a graph-based algorithm for selection of representative (key) frames. Our spatial action localization technique SALiEnSeA identifies the most-'dynamic' motion patch in each key-frame. It is based on an oil-painting approach of refining and stacking motion components. These focused actions along with spatial and temporal information are fed into three separate deep neural-network pipelines consisting of ResNet50 and LSTM. A multi-tier hierarchical fusion thereby, consolidates frame-level and video-level predictions. The experiment is performed on four benchmark datasets: CCV, KCV, UCF-101 and HMDB-51. The holistically developed solution framework for action localization-aided event classification provides encouraging results. By introducing a separate modality for action-localized SALiEnSeA patches, we get improved video classification performance on top of the traditional modality of RGB frames. This outperforms standard neural-network based approaches as well as state-of-the-art multimodal models in use, for video classification. [ABSTRACT FROM AUTHOR]

Malware is an ever-present problem in the modern era and while detecting malware with AI has grown as a new field of exploration, current methods are not yet mature enough for widespread adoption in terms of speed and performance. Current methods largely focus on viewing malicious assembly as an image for detection, requiring a large amount of preprocessing and making network architectures inflexible. Preprocessing malware images to one size introduces additional time to predict and makes the task of prediction more difficult. We explore a novel method for transforming executable bytecode into a video rather than an image for classification with deep, time-distributed neural networks, achieving up to 98.74% testing accuracy on 9 classes of malware, and up to 99.36% testing accuracy on a balanced set of malicious vs. benign files. The network could also classify all malware in our dataset for a false positive rate of 13%, and was also found successful in classifying only parts of an input, as well as initial success in a 0-day scenario. The network only uses the executable code and no additional information to make predictions. We then explore methods for pruning and quantizing the network so that it may be more feasible for widespread implementation, including a novel pruning method we call Node-Distance pruning. Our model is found to be competitive to current works while remaining fast, lean and flexible.

In this work, we propose 3D Residual Attention Networks (3D RANs) for action recognition, which can learn spatiotemporal representation from videos. The proposed network consists of attention mechanism and 3D ResNets architecture, and it can capture spatiotemporal information in an end-to-end manner. Specifically, we separately add the attention mechanism along channel and spatial domain to each block of 3D ResNets. For each sliced tensor of an intermediate feature map, we sequentially infer channel and spatial attention maps by channel and spatial attention mechanism submodules in each residual unit block, and the attention maps are multiplied to the input feature map to reweight the key features. We validate our network through extensive experiments in UCF-101, HMDB-51 and Kinetics datasets. Our experiments show that the proposed 3D RANs are superior to the state-of-the-art approaches for action recognition, demonstrating the effectiveness of our networks. [ABSTRACT FROM AUTHOR]

Objective: Gastric slow wave abnormalities have been associated with gastric motility disorders. Invasive studies in humans have described normal and abnormal propagation of the slow wave. This study aims to disambiguate the abnormally functioning wave from one of normalcy using multi-electrode abdominal waveforms of the electrogastrogram (EGG). Methods: Human stomach and abdominal models are extracted from computed tomography scans. Normal and abnormal slow waves are simulated along stomach surfaces. Current dipoles at the stomachs surface are propagated to virtual electrodes on the abdomen with a forward model. We establish a deep convolutional neural network (CNN) framework to classify normal and abnormal slow waves from the multi-electrode waveforms. We investigate the effects of non-idealized measurements on performance, including shifted electrode array positioning, smaller array sizes, high body mass index (BMI), and low signal-to-noise ratio (SNR). We compare the performance of our deep CNN to a linear discriminant classifier using wave propagation spatial features. Results: A deep CNN framework demonstrated robust classification, with accuracy above 90% for all SNR above 0 dB, horizontal shifts within 3 cm, vertical shifts within 6 cm, and abdominal tissue depth within 6 cm. The linear discriminant classifier was much more vulnerable to SNR, electrode placement, and BMI. Conclusion: This is the first study to attempt and, moreover, succeed in using a deep CNN to disambiguate normal and abnormal gastric slow wave patterns from high-resolution EGG data. Significance: These findings suggest that multi-electrode cutaneous abdominal recordings have the potential to serve as widely deployable clinical screening tools for gastrointestinal foregut disorders. [ABSTRACT FROM AUTHOR]

Binge drinking is a severe health problem faced by many US colleges and universities. College students often post drinking-related text and images on social media, portraying their alcohol use as socially desirable. In this project, we investigated the feasibility of mining the heterogeneous data (e.g. text, images, and videos) on Facebook to identify drinking-related contents. We manually annotated 4266 posts during 21 October 2011 and 3 November 2014 from "I'm Shmacked" group on Facebook, where 511 posts were drinking-related. Our machine learning models show that by combining heterogeneous data types, we were able to identify drinking-related posts with an F1-score of 0.81. Prediction models built on text data were more reliable compared to those built on image and video data for predicting drinking-related contents. As the first step of our efforts in this direction, this feasibility study showed promise toward unleashing the potential of mining social media to identify students who binge drink. [ABSTRACT FROM AUTHOR]

The integration of artificial intelligence (AI) into human activity recognition (HAR) in smart surveillance systems has the potential to revolutionize behavior monitoring. These systems analyze an individual's physiological or behavioral features to continuously monitor and identify any unusual or suspicious activity in video streams, thereby improving security and surveillance measures. Traditional surveillance systems often rely on manual human monitoring, which is resource-intensive, error-prone, and time-consuming. To address these limitations, computer vision-based behavior biometrics has emerged as a solution for secure video surveillance applications. However, implementing automated HAR in real-world scenarios is challenging due to the diversity of human behavior, complex spatiotemporal patterns, varying viewpoints, and cluttered backgrounds. To tackle these challenges, an AI-based behavior biometrics framework is introduced that is based on a dynamic attention fusion unit (DAFU) followed by a temporal-spatial fusion (TSF) network to effectively recognize human activity in surveillance systems. In the first phase of the proposed framework, a lightweight EfficientNetB0 backbone is enhanced by the DAFU to extract human-centric salient features using a unified channel-spatial attention mechanism. In the second phase, the DAFU features with fixed sequence lengths are passed to the proposed TSF network to capture the temporal, spatial, and behavioral dependencies in video data streams. The integration of Echo-ConvLSTM in the TSF further enhances accuracy and robustness by combining temporal dependencies from the echo state network with spatial and temporal dependencies from the convolutional long short-term memory (ConvLSTM). The proposed AI-based behavior biometrics framework is evaluated using four publicly available HAR datasets (UCF101, HMDB51, UCF50, and YouTube Action), yielding higher accuracies of 98.734%, 80.342%, 98.987%, and 98.927%, which demonstrate the superior performance when compared with state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

We present a new action recognition deep neural network which adaptively learns the best action velocities in addition to the classification. While deep neural networks have reached maturity for image understanding tasks, we are still exploring network topologies and features to handle the richer environment of video clips. Here, we tackle the problem of multiple velocities in action recognition, and provide state-of-the-art results for facial expression recognition, on known and new collected datasets. We further provide the training steps for our semi-supervised network, suited to learn from huge unlabeled datasets with only a fraction of labeled examples. [ABSTRACT FROM AUTHOR]

Videos are inherently multimodal. This paper studies the problem of exploiting the abundant multimodal clues for improved video classification performance. We introduce a novel hybrid deep learning framework that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information, as well as long-range temporal dynamics. More specifically, we utilize three Convolutional Neural Networks (CNNs) operating on appearance, motion, and audio signals to extract their corresponding features. We then employ a feature fusion network to derive a unified representation with an aim to capture the relationships among features. Furthermore, to exploit the long-range temporal dynamics in videos, we apply two long short-term memory (LSTM) networks with extracted appearance and motion features as inputs. Finally, we also propose refining the prediction scores by leveraging contextual relationships among video semantics. The hybrid deep learning framework is able to exploit a comprehensive set of multimodal features for video classification. Through an extensive set of experiments, we demonstrate that: 1) LSTM networks that model sequences in an explicitly recurrent manner are highly complementary to the CNN models; 2) the feature fusion network that produces a fused representation through modeling feature relationships outperforms a large set of alternative fusion strategies; and 3) the semantic context of video classes can help further refine the predictions for improved performance. Experimental results on two challenging benchmarks—the UCF-101 and the Columbia Consumer Videos (CCV)—provide strong quantitative evidence that our framework can produce promising results: ${\text{93.1}\%}$ on the UCF-101 and ${\text{84.5}\%}$ on the CCV, outperforming several competing methods with clear margins. [ABSTRACT FROM AUTHOR]

With the rapid increase in the amount of multimedia data, video classification has become a demanding and challenging research topic. Compared with image classification, video classification requires mapping a video that contains hundreds of frames to semantic tags, which poses many challenges to the direct use of advanced models originally designed for image-oriented tasks. On the other hand, continuous frames in a video also give us more visual clues that we can leverage to achieve better classification. One of the most important clues is the context in the spatiotemporal domain. In this paper, we introduce the multiscale deep alternative neural network (DANN), a novel architecture combining the strengths of both convolutional neural network and recurrent neural networks to achieve a deep network that can collect rich context hierarchies for video classification. In particular, the DANN is stacked with alternative layers, each of which consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as a local feature learner, whereas the latter is used to collect contexts. Compared with popular deep feed-forward neural networks, the DANN learns local features and their contexts from the very beginning. This setting enables preserving context evolutions, which we show to be essential for improving the accuracy of video classification. To release the full potential of the DANN, we develop a deeper version with stochastic-layer skip-connections and construct a multiscale DANN to incorporate contexts at different scales. We show how to apply the multiscale DANN for video classification with carefully designed configurations in terms of both input–output settings and training–testing methods. The DANN is shown to be robust to not only human-centric videos, but also natural videos. As there are few large-scale natural disaster video datasets, we construct a new large-scale one and make it publicly available. Experiments on four datasets show the effectiveness of our method for both human actions and natural events. [ABSTRACT FROM AUTHOR]

In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets. [ABSTRACT FROM AUTHOR]

The significant growth of surveillance camera networks necessitates scalable AI solutions to efficiently analyze the large amount of video data produced by these networks. As a typical analysis performed on surveillance footage, video violence detection has recently received considerable attention. The majority of research has focused on improving existing methods using supervised methods, with little, if any, attention to the semi-supervised learning approaches. In this study, a reinforcement learning model is introduced that can outperform existing models through a semi-supervised approach. The main novelty of the proposed method lies in the introduction of a semi-supervised hard attention mechanism. Using hard attention, the essential regions of videos are identified and separated from the non-informative parts of the data. A model's accuracy is improved by removing redundant data and focusing on useful visual information in a higher resolution. Implementing hard attention mechanisms using semi-supervised reinforcement learning algorithms eliminates the need for attention annotations in video violence datasets, thus making them readily applicable. The proposed model utilizes a pre-trained I3D backbone to accelerate and stabilize the training process. The proposed model achieved state-of-the-art accuracy of 90.4% and 98.7% on RWF and Hockey datasets, respectively. • Videos contain irrelevant information like background and surrounding objects. • Isolating the valuable information in a video increases the classification accuracy. • Reinforcement learning-based hard attention can identify critical visual information. • Hard attention learns violence localization without explicit spatial annotation. [ABSTRACT FROM AUTHOR]

In this paper we propose a multimodal feature learning mechanism based on deep networks (i.e., stacked contractive autoencoders) for video classification. Considering the three modalities in video, i.e., image, audio and text, we first build one Stacked Contractive Autoencoder (SCAE) for each single modality, whose outputs will be joint together and fed into another Multimodal Stacked Contractive Autoencoder (MSCAE). The first stage preserves intra-modality semantic relations and the second stage discovers inter-modality semantic correlations. Experiments on real world dataset demonstrate that the proposed approach achieves better performance compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Video-based computer vision tasks can benefit from estimation of the salient regions and interactions between those regions. Traditionally, this has been done by identifying the object regions in the images by utilizing pre-trained models to perform object detection, object segmentation and/or object pose estimation. Although using pre-trained models is a viable approach, it has several limitations in the need for an exhaustive annotation of object categories, a possible domain gap between datasets and a bias that is typically present in pre-trained models. In this work, we propose to utilize the common rationale that a sequence of video frames capture a set of common objects and interactions between them, thus a notion of co-segmentation between the video frame features may equip the model with the ability to automatically focus on task-specific salient regions and improve the underlying task's performance in an end-to-end manner. In this regard, we propose a generic module called "Co-Segmentation inspired Attention Module" (COSAM) that can be plugged in to any CNN model to promote the notion of co-segmentation based attention among a sequence of video frame features. We show the application of COSAM in three video-based tasks namely: (1) Video-based person re-ID, (2) Video captioning, & (3) Video action classification and demonstrate that COSAM is able to capture the task-specific salient regions in video frames, thus leading to notable performance improvements along with interpretable attention maps for a variety of video-based vision tasks, with possible application to other video-based vision tasks as well. • Exploring the applicability of "Co-segmentation" concept in video-based tasks. • Generic "Co-Segmentation inspired Attention Module" (COSAM) to plug-in to any CNN. • The COSAM module captures task-specific salient regions in an end-to-end manner. • The COSAM module aids interpretability through its spatial attention mask. • Experiments on three video-based tasks show the effectiveness of the COSAM module. [ABSTRACT FROM AUTHOR]

Abstract: This paper introduces a novel recognition framework for human actions using hybrid features. The hybrid features consist of spatio-temporal and local static features extracted using motion-selectivity attribute of 3D dual-tree complex wavelet transform (3D DT-CWT) and affine SIFT local image detector, respectively. The proposed model offers two core advantages: (1) the framework is significantly faster than traditional approaches due to volumetric processing of images as a ‘3D box of data’ instead of a frame by frame analysis, (2) rich representation of human actions in terms of reduction in artifacts in view of the promising properties of our recently designed full symmetry complex filter banks with better directionality and shift-invariance properties. No assumptions about scene background, location, objects of interest, or point of view information are made whereas bidirectional two-dimensional PCA (2D-PCA) is employed for dimensionality reduction which offers enhanced capabilities to preserve structure and correlation amongst neighborhood pixels of a video frame. [Copyright &y& Elsevier]

Accounting for spatial image transformations is a requirement for multimedia problems such as video classification and retrieval, face/object recognition or the creation of image mosaics from video sequences. We analyze a transformation invariant metric recently proposed in the machine learning literature to measure the distance between image manifolds - the tangent distance (TD) - and show that it is closely related to alignment techniques from the motion analysis literature. Exposing these relationships results in benefits for the two domains. On one hand, it allows leveraging on the knowledge acquired in the alignment literature to build better classifiers. On the other, it provides a new interpretation of alignment techniques as one component of a decomposition that has interesting properties for the classification of video. In particular, we embed the TD into a multiresolution framework that makes it significantly less prone to local minima. The new metric - multiresolution tangent distance (MRTD) - can be easily combined with robust estimation procedures, and exhibits significantly higher invariance to image transformations than the TD and the Euclidean distance (ED). For classification, this translates into significant improvements in face recognition accuracy. For video characterization, it leads to a decomposition of image dissimilarity into "differences due to camera motion" plus "differences due to scene activity" that is useful for classification. Experimental results on a movie database indicate that the distance could be used as a basis for the extraction of semantic primitives such as action and romance. [ABSTRACT FROM PUBLISHER]

Today's content-based video retrieval technologies are still far from human's requirements. A fundamental reason is the lack of content representation that is able to bridge the gap between visual features and semantic conception in video. In this paper, we propose a motion pattern descriptor, motion texture that characterizes motion in a generic way. With this representation, we design a semantic classification scheme to effectively map video clips to semantic categories. Support vector machines (SVMs) are used as the classifiers. In addition, this scheme also improves significantly the performance of motion-based shot retrieval due to the comprehensiveness and effectiveness of motion pattern descriptor and the semantic classification capability as shown by experimental evaluations. [ABSTRACT FROM AUTHOR]

• We propose weakly-supervised temporal attention 3D network for human action recognition, called TA3DNet. • TA3DNet consists of two subnetworks: temporal frame selection and weakly supervised temporal attention. • We accelerate 3D convolutional neural networks (3D CNNs) by temporally assigning different importance to each frame • We adopt weakly-supervised learning to successfully train an action recognition model from given labels. From a series of observations, we have inferred that human actions in videos are defined by a set of significant frames. In this paper, we propose a weakly-supervised temporal attention 3D network for human action recognition, called as TA3DNet, to accelerate 3D convolutional neural networks (3D CNNs) by temporally assigning different importance to each frame. First, we obtain short-term frames with long-term connection by regularly or randomly skipping frames to avoid temporal redundancy, and apply 3D convolutional layers to extract features for action recognition. Then, we apply a temporal attention module to assign different weights to each frame. We train the temporal attention module in a weakly-supervised manner that updates weights based on only class labels without event information and extra labels. Thus, TA3DNet reduces the number of input frames and constructs a lightweight network for action recognition. Experimental results demonstrate that TA3DNet achieves high performance on two challenging datasets (UCF101 and HMDB51) and outperforms state-of-the-art methods for action recognition. [ABSTRACT FROM AUTHOR]

Today′s content-based video retrieval technologies are still far from human′s requirements. A fundamental reason is the lack of content representation that is able to bridge the gap between visual features and semantic conception in video. In this paper, we propose a motion pattern descriptor, motion texture that characterizes motion in a generic way. With this representation, we design a semantic classification scheme to effectively map video clips to semantic categories. Support vector machines (SVMs) are used as the classifiers. In addition, this scheme also improves significantly the performance of motion-based shot retrieval due to the comprehensiveness and effectiveness of motion pattern descriptor and the semantic classification capability as shown by experimental evaluations.

Solutions for the quality control of metallic surfaces are proposed. Specifically, we study a deflectometric apparatus based on coaxial structured light and the related algorithmic procedure, which is able to detect the faulty surface of a sample captured by a video sequence. First, by considering the metallic surface a dynamic scene illuminated under different light conditions, we develop the descriptor residuals of linear evolution of light (RLEL) that extracts the defectiveness information starting from the movement of the object without explicitly considering the physical characteristics of the light structure. Then, leveraging on RLEL, we present a hybrid learning (HL) technique capable of overcoming the data-driven approach used in classic deep learning (DL). By exploiting a multisteps training process, we combine the model-based descriptor RLEL and a classical data-driven convolutional neural network (CNN) to obtain an unconventional gray-box CNN, which exceeds the performance of popular DL solutions such as 3-D-inception and 3-D-residual DL networks. Remarkably, HL also shows its validity in comparing the performance of the same network structures trained not in a hybrid way, namely without the injection of the model-based information given by RLEL. [ABSTRACT FROM AUTHOR]

Summary: On the social Web, the amount of video content either originated from wireless devices or previously received from media servers has increased enormously in the recent years. The astounding growth of Web videos has stimulated researchers to propose new strategies to organize them into their respective categories. Because of complex ontology and large variation in content and quality of Web videos, it is difficult to get sufficient, precisely labeled training data, which causes hindrance in automatic video classification. In this paper, we propose a novel content‐ and context‐based Web video classification framework by rendering external support through category discriminative terms (CDTs) and semantic relatedness measure (SRM). Mainly, a three‐step framework is proposed. Firstly, content‐based video classification is proposed, where twofold use of high‐level concept detectors is leveraged to classify Web videos. Initially, category classifiers induced from VIREO‐374 detectors are trained to classify Web videos, and then concept detectors with high confidence for each video are mapped to CDT through SRM‐assisted semantic content fusion function to further boost the category classifiers, which intuitively provide a more robust measure for Web video classification. Secondly, a context‐based video classification is proposed, where twofold use of contextual information is also harnessed. Initially, cosine similarity and then semantic similarity are measured between text features of each video and CDT through vector space model (VSM)‐ and SRM‐assisted semantic context fusion function, respectively. Finally, classification results from content and context are fused to compensate for the shortcomings of each other, which enhance the video classification performance. Experiments on large‐scale video dataset validate the effectiveness of the proposed solution. [ABSTRACT FROM AUTHOR]