Search

Video prediction presents a formidable challenge, requiring effectively processing spatial and temporal information embedded in videos. While recurrent neural network (RNN) and transformer-based models have been extensively explored to address spatial changes over time, recent advancements in convolutional neural networks (CNNs) have yielded high-performance video prediction models. CNN-based models offer advantages over RNN and transformer-based models due to their ease of parallel processing and lower computational complexity, highlighting their significance in practical applications. However, existing CNN-based video prediction models typically treat the spatiotemporal channels of videos similarly to the channel axis of static images. They stack frames in temporal order to construct a spatiotemporal axis and employ standard $1\times 1$ convolution operations. Nevertheless, this approach has its limitations. Applying $1\times 1$ convolution directly to the spatiotemporal axis results in a mixture of temporal and spatial information, which may lead to computational inefficiencies and reduced accuracy. Additionally, this operation needs to improve in processing temporal data. This study introduces a CNN-based time series decomposition model for video prediction. The proposed model first divides the $1\times 1$ convolution operation within the channel aggregation module to independently process the temporal and spatial dimensions. To capture evolving features, the temporal axis is segregated into trend and residual components, followed by applying a time series decomposition forecasting method. To assess the performance of the proposed technique, experiments were conducted using the moving MNIST, KTH, and KITTI-Caltech benchmark datasets. In the experiments on moving MNIST, despite a reduction of approximately 55% in the number of parameters and 37% in computational cost, the proposed method improved accuracy by up to 7% compared to the previous approach.

We developed a self-attention-based spatiotemporal network called a l p h a Beach that uses spatiotemporal representation learning for skillful prediction of shoreline changes multiple days ahead. The proposed model predicts the spatiotemporal position of the shoreline up to seven consecutive days in the future based on hydrodynamic forcing of ocean waves and tide data for the past 30 consecutive days. It is further divided into a l p h a Beach-w/o IC and a l p h a Beach-w/ IC depending on whether or not the beach state of the antecedent historical shoreline information is used as the initial condition. a l p h a Beach-w/o IC, which does not incorporate this information, learns the sequential relationship between hydrodynamic forcing and shoreline to estimate overall trends of shoreline changes including seasonal oscillation from the point in time after model training, given only the ocean waves and tides. a l p h a Beach-w/ IC does incorporate antecedent historical shoreline information to greatly enhance its predictive accuracy of shoreline progradation, retreat, and beach rotation for short-term time scales and for extreme storm events. The proposed model was applied to Tairua Beach, New Zealand, and it demonstrated superior predictive accuracy compared to previous methods and matched current understanding of accretion-dominated and oscillation-dominated shoreline changes. • Self-attention-based spatiotemporal network to predict of shoreline changes • Multiple-days-ahead skillful prediction of spatiotemporal beach position and rotation • Incorporates antecedent shoreline to predict extreme beach progradation and retreat • Plausible framework for encoding domain knowledge of accretion and oscillation • Effective spatiotemporal representation learning to understand shoreline changes [ABSTRACT FROM AUTHOR]

Accurate predictions of extreme water temperatures are criticalto understanding the variability of the marine environment and reducing marine disasters maximized by global warming. In this study, we propose a two-pathway framework with separated spatial and temporal encoders for accurate prediction of water temperature, especially extremely high water temperature, through effective spatiotemporal representation learning. The spatial and temporal encoder networks based on the Transformer's self-attention mechanism performs the task of predicting the water temperature time series at the 16 coastal locations around the Korean Peninsula for the seven consecutive days ahead at daily intervals with various combinations of patch embedding methods, positional embedding for spatial features. Comparative experiments with conventional deep convolutional and recurrent networks are also conducted for comparison. By comparing and assessing these results, the proposed two-pathway framework can improve the predictability of extremely high coastal water temperature by better capturing spatiotemporal interrelationships and long-range dependencies from open ocean and regional sea, and further determines the optimal architectural details of self-attention-based spatial and temporal encoders. Furthermore, to examine the explainability of the proposed model and its consistency with domain knowledge, spatial and temporal attention maps are visualized and analyzed that represents weights for spatiotemporal input sequences that are more relevant to predict for future predictions. • Development of two-pathway spatiotemporal representation learning framework for consecutive multi-step-ahead SST prediction. • Spatiotemporal representation learning to capture the temporally correlated regional to local spatial dependencies. • Notable predictability of extreme coastal SST for consecutive 7-day ahead spatiotemporal forecasts. • Visualization of attention map to secure domain knowledge-based validity and explainability of model predictions. • Discovering spatiotemporal teleconnections of SST variability and its consistency with ocean dynamics. [ABSTRACT FROM AUTHOR]