Search

Video prediction presents a formidable challenge, requiring effectively processing spatial and temporal information embedded in videos. While recurrent neural network (RNN) and transformer-based models have been extensively explored to address spatial changes over time, recent advancements in convolutional neural networks (CNNs) have yielded high-performance video prediction models. CNN-based models offer advantages over RNN and transformer-based models due to their ease of parallel processing and lower computational complexity, highlighting their significance in practical applications. However, existing CNN-based video prediction models typically treat the spatiotemporal channels of videos similarly to the channel axis of static images. They stack frames in temporal order to construct a spatiotemporal axis and employ standard $1\times 1$ convolution operations. Nevertheless, this approach has its limitations. Applying $1\times 1$ convolution directly to the spatiotemporal axis results in a mixture of temporal and spatial information, which may lead to computational inefficiencies and reduced accuracy. Additionally, this operation needs to improve in processing temporal data. This study introduces a CNN-based time series decomposition model for video prediction. The proposed model first divides the $1\times 1$ convolution operation within the channel aggregation module to independently process the temporal and spatial dimensions. To capture evolving features, the temporal axis is segregated into trend and residual components, followed by applying a time series decomposition forecasting method. To assess the performance of the proposed technique, experiments were conducted using the moving MNIST, KTH, and KITTI-Caltech benchmark datasets. In the experiments on moving MNIST, despite a reduction of approximately 55% in the number of parameters and 37% in computational cost, the proposed method improved accuracy by up to 7% compared to the previous approach.

Temporal reasoning is crucial for action recognition tasks. The previous works use 3D CNNs to jointly capture spatiotemporal information, but it causes a lot of computational costs as well. To improve the above problems, we propose a general channel split spatiotemporal network (CSST-Net) to achieve effective spatiotemporal feature representation learning. The CSST module consists of the grouped spatiotemporal modeling (GSTM) module and the parameter-free feature fusion (PFFF) module. The GSTM module decomposes features into spatial and temporal parts along the channel dimension in parallel, which focuses on spatial and temporal clues, respectively. Meanwhile, we utilize the combination of group-wise convolution and point-wise convolution to reduce the number of parameters in the temporal branch, thus alleviating the overfitting of 3D CNNs. Furthermore, for the problem of spatiotemporal feature fusion, the PFFF module performs the recalibration and fusion of spatial and temporal features by a soft attention mechanism, without introducing extra parameters, thus ensuring the correct network information flow effectively. Finally, extensive experiments on three benchmark databases (Sth-Sth V1, Sth-Sth V2, and Jester) indicate that the proposed CSST-Net can achieve competitive performance compared to existing methods, and significantly reduces the number of parameters and FLOPs of 3D CNNs baseline. [ABSTRACT FROM AUTHOR]

Source acquisition device identification from recorded audio aims to identify the source recording device by analyzing the intrinsic characteristics of audio, which is a challenging problem in audio forensics. In this paper, we propose a spatiotemporal representation learning framework with multi-attention mechanisms to tackle this problem. In the deep feature extraction stage of recording devices, a two-branch network based on residual dense temporal convolution networks (RD-TCNs) and convolutional neural networks (CNNs) is constructed. The spatial probability distribution features of audio signals are employed as inputs to the branch of the CNN for spatial representation learning, and the temporal spectral features of audio signals are fed into the branch of the RD-TCN network for temporal representation learning. This achieves simultaneous learning of long-term and short-term features to obtain an accurate representation of device-related information. In the spatiotemporal feature fusion stage, three attention mechanisms—temporal, spatial, and branch attention mechanisms—are designed to capture spatiotemporal weights and achieve effective deep feature fusion. The proposed framework achieves state-of-the-art performance on the benchmark CCNU_Mobile dataset, reaching an accuracy of 97.6% for the identification of 45 recording devices, with a significant reduction in training time compared to other models. [ABSTRACT FROM AUTHOR]

We developed a self-attention-based spatiotemporal network called a l p h a Beach that uses spatiotemporal representation learning for skillful prediction of shoreline changes multiple days ahead. The proposed model predicts the spatiotemporal position of the shoreline up to seven consecutive days in the future based on hydrodynamic forcing of ocean waves and tide data for the past 30 consecutive days. It is further divided into a l p h a Beach-w/o IC and a l p h a Beach-w/ IC depending on whether or not the beach state of the antecedent historical shoreline information is used as the initial condition. a l p h a Beach-w/o IC, which does not incorporate this information, learns the sequential relationship between hydrodynamic forcing and shoreline to estimate overall trends of shoreline changes including seasonal oscillation from the point in time after model training, given only the ocean waves and tides. a l p h a Beach-w/ IC does incorporate antecedent historical shoreline information to greatly enhance its predictive accuracy of shoreline progradation, retreat, and beach rotation for short-term time scales and for extreme storm events. The proposed model was applied to Tairua Beach, New Zealand, and it demonstrated superior predictive accuracy compared to previous methods and matched current understanding of accretion-dominated and oscillation-dominated shoreline changes. • Self-attention-based spatiotemporal network to predict of shoreline changes • Multiple-days-ahead skillful prediction of spatiotemporal beach position and rotation • Incorporates antecedent shoreline to predict extreme beach progradation and retreat • Plausible framework for encoding domain knowledge of accretion and oscillation • Effective spatiotemporal representation learning to understand shoreline changes [ABSTRACT FROM AUTHOR]

Source acquisition device identification from recorded audio aims to identify the source recording device by analyzing the intrinsic characteristics of audio, which is a challenging problem in audio forensics. In this paper, we propose a spatiotemporal representation learning framework with multi-attention mechanisms to tackle this problem. In the deep feature extraction stage of recording devices, a two-branch network based on residual dense temporal convolution networks (RD-TCNs) and convolutional neural networks (CNNs) is constructed. The spatial probability distribution features of audio signals are employed as inputs to the branch of the CNN for spatial representation learning, and the temporal spectral features of audio signals are fed into the branch of the RD-TCN network for temporal representation learning. This achieves simultaneous learning of long-term and short-term features to obtain an accurate representation of device-related information. In the spatiotemporal feature fusion stage, three attention mechanisms—temporal, spatial, and branch attention mechanisms—are designed to capture spatiotemporal weights and achieve effective deep feature fusion. The proposed framework achieves state-of-the-art performance on the benchmark CCNU_Mobile dataset, reaching an accuracy of 97.6% for the identification of 45 recording devices, with a significant reduction in training time compared to other models.

Video-based anomalous human behavior detection is widely studied in many fields such as security, medical care, education, and energy. However, there are still some open problems in anomalous behavior detection, such as the large and complicated model is difficult to train, the accuracy of anomalous behavior detection is not high enough and the speed is not fast enough. A spatiotemporal representation learning model is proposed in this paper. Firstly, the spatial-temporal features of the video are extracted by the constructed multi-scale 3D convolutional neural network. Then the scene background is modeled by the high-dimensional mixed Gaussian model and used for anomaly detection. Finally, the accurate position of anomalous behavior in the video data is achieved by calculating the position of the last output feature, that is, the position of the receptive field. The proposed model does not require specific training. Moreover, the proposed method has the advantages of high versatility, fast calculation speed and high detection accuracy. We validated the proposed algorithm on two representative surveillance scene datasets, the Subway and the UCSDSped2. Results show that proposed algorithm has achieved the detection rate of 18 FPS under the condition of common computing resources, and meet the real-time requirements. Moreover, compared the similar methods, the proposed method has achieved the competitive results in both frame-level accuracy and pixel-level accuracy.

Accurate predictions of extreme water temperatures are criticalto understanding the variability of the marine environment and reducing marine disasters maximized by global warming. In this study, we propose a two-pathway framework with separated spatial and temporal encoders for accurate prediction of water temperature, especially extremely high water temperature, through effective spatiotemporal representation learning. The spatial and temporal encoder networks based on the Transformer's self-attention mechanism performs the task of predicting the water temperature time series at the 16 coastal locations around the Korean Peninsula for the seven consecutive days ahead at daily intervals with various combinations of patch embedding methods, positional embedding for spatial features. Comparative experiments with conventional deep convolutional and recurrent networks are also conducted for comparison. By comparing and assessing these results, the proposed two-pathway framework can improve the predictability of extremely high coastal water temperature by better capturing spatiotemporal interrelationships and long-range dependencies from open ocean and regional sea, and further determines the optimal architectural details of self-attention-based spatial and temporal encoders. Furthermore, to examine the explainability of the proposed model and its consistency with domain knowledge, spatial and temporal attention maps are visualized and analyzed that represents weights for spatiotemporal input sequences that are more relevant to predict for future predictions. • Development of two-pathway spatiotemporal representation learning framework for consecutive multi-step-ahead SST prediction. • Spatiotemporal representation learning to capture the temporally correlated regional to local spatial dependencies. • Notable predictability of extreme coastal SST for consecutive 7-day ahead spatiotemporal forecasts. • Visualization of attention map to secure domain knowledge-based validity and explainability of model predictions. • Discovering spatiotemporal teleconnections of SST variability and its consistency with ocean dynamics. [ABSTRACT FROM AUTHOR]

Action recognition is a significant and challenging topic in the field of sensor and computer vision. Two-stream convolutional neural networks (CNNs) and 3D CNNs are two mainstream deep learning architectures for video action recognition. To combine them into one framework to further improve performance, we proposed a novel deep network, named the spatiotemporal interaction residual network with pseudo3D (STINP). The STINP possesses three advantages. First, the STINP consists of two branches constructed based on residual networks (ResNets) to simultaneously learn the spatial and temporal information of the video. Second, the STINP integrates the pseudo3D block into residual units for building the spatial branch, which ensures that the spatial branch can not only learn the appearance feature of the objects and scene in the video, but also capture the potential interaction information among the consecutive frames. Finally, the STINP adopts a simple but effective multiplication operation to fuse the spatial branch and temporal branch, which guarantees that the learned spatial and temporal representation can interact with each other during the entire process of training the STINP. Experiments were implemented on two classic action recognition datasets, UCF101 and HMDB51. The experimental results show that our proposed STINP can provide better performance for video recognition than other state-of-the-art algorithms.

Action recognition is a significant and challenging topic in the field of sensor and computer vision. Two-stream convolutional neural networks (CNNs) and 3D CNNs are two mainstream deep learning architectures for video action recognition. To combine them into one framework to further improve performance, we proposed a novel deep network, named the spatiotemporal interaction residual network with pseudo3D (STINP). The STINP possesses three advantages. First, the STINP consists of two branches constructed based on residual networks (ResNets) to simultaneously learn the spatial and temporal information of the video. Second, the STINP integrates the pseudo3D block into residual units for building the spatial branch, which ensures that the spatial branch can not only learn the appearance feature of the objects and scene in the video, but also capture the potential interaction information among the consecutive frames. Finally, the STINP adopts a simple but effective multiplication operation to fuse the spatial branch and temporal branch, which guarantees that the learned spatial and temporal representation can interact with each other during the entire process of training the STINP. Experiments were implemented on two classic action recognition datasets, UCF101 and HMDB51. The experimental results show that our proposed STINP can provide better performance for video recognition than other state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]