Search

The task of human action recognition (HAR) can be found in many computer vision practical applications. Various data modalities have been considered for solving this task, including joint-based skeletal representations which are suitable for real-time applications on platforms with limited computational resources. We propose a spatio-temporal neural network that uses handcrafted geometric features to classify human actions from video data. The proposed deep neural network architecture combines graph convolutional and temporal convolutional layers. The experiments performed on public HAR datasets show that our model obtains results similar to other state-of-the-art methods but has a lower inference time while offering the possibility to obtain an explanation for the classified action. [ABSTRACT FROM AUTHOR]

In this article, a new self-supervised strategy for learning meaningful representations of complex optical satellite image time series (SITS) is presented. The methodology proposed, named Unet-BERT spAtio-temporal Representation eNcoder (U-BARN), exploits irregularly sampled SITS. The designed architecture allows learning rich and discriminative features from unlabeled data, enhancing the synergy between the spatio-spectral and the temporal dimensions. To train on unlabeled data, a time-series reconstruction pretext task inspired by the BERT strategy but adapted to SITS is proposed. A Sentinel-2 large-scale unlabeled dataset is used to pretrain U-BARN. During the pretraining, U-BARN processes annual time series composed of a maximum of 100 dates. To demonstrate its feature learning capability, representations of SITS encoded by U-BARN are then fed into a shallow classifier to generate semantic segmentation maps. Experimental results are conducted on a labeled crop dataset (PASTIS) as well as a dense land cover dataset (MultiSenGE). Two ways of exploiting U-BARN pretraining are considered: either U-BARN weights are frozen or fine-tuned. The obtained results demonstrate that representations of SITS given by the frozen U-BARN are more efficient for land cover and crop classification than those of a supervised-trained linear layer. Then, we observe that fine-tuning boosts U-BARN performances on MultiSenGE dataset. In addition, we observe on PASTIS, in scenarios with scarce reference data that the fine-tuning brings a significative performance gain compared to fully supervised approaches. We also investigate the influence of the percentage of elements masked during pretraining on the quality of the SITS representation. Eventually, semantic segmentation performances show that the fully supervised U-BARN architecture reaches better performances than the spatio-temporal baseline (U-TAE) on both downstream tasks: crop and dense land cover segmentation.

In this paper, the overall architecture design of the platform is proposed, and the data architecture and business architecture of the platform are discussed. On this basis, the logistics path planning method of digital collaboration is proposed, including the aspects of path demand splitting and distribution vehicle spatio-temporal network, and the demand splittable logistics path planning model is considered. Finally, the simulation of the workshop logistics dynamic scheduling example is designed, and the dynamic scheduling optimization results are analyzed and verified. The results show that: through optimization, its time penalty cost f1 = 801.38, the total distance traveled by the delivery trolley f2 = 1709m, and the total distance of empty load is 692.15 m, and the dispatching scheme increases the distance of empty load by 59 m . The research results of this paper can provide a reference for digital management and collaboration platform for civil aircraft manufacturing enterprises and provide a reference for research in related fields.

Remote photoplethysmography (rPPG), which measures human heart rate without physical contact with the skin, has become active research in recent years. Neural networks have been introduced into rPPG for accurate pulse measurement and have achieved overwhelming results. However, there is a lack of in-depth analysis of key components of neural networks exhibiting a crucial impact on pulse extraction from video. In this paper, we present a network with attention and spatio-temporal convolutional block (ASTNet), exploiting the impact of key factors including different spatio-temporal convolutions, attention mechanism, the number of convolutional layers, and receptive field sizes. The novel attention module named spatio-temporal-channel (STC) attention is designed to jointly learn weights in spatial, temporal, and channel dimensions in a more efficient way. Extensive experiments have been conducted on two uncompressed datasets and one compressed dataset. Results show that ASTNet outperforms state-of-the-art methods in accuracy and computational time. Specifically, networks with larger receptive field sizes and more spatio-temporal blocks generally achieve better performance. Networks with pseudo 3D convolution outperform those with convolutional 3D in static videos, and the opposite is true in motion videos. The results exhibit a similar tendency both on uncompressed and compressed datasets. The proposed method improves the performance of pulse signal compared to PhysNet (the second-best approach in the compared methods), with the signal-to-noise ratio increased by 7.03%, 10.19%, 4.79%, the mean absolute error decreased by 17.95%, 14.17%, 22.76%, and the root-mean-square error decreased by 21.43%, 2.73%, 25.43%, on the PURE, Self-rPPG, and COHFACE datasets, respectively. [ABSTRACT FROM AUTHOR]

The environment and development are major issues of general concern. After much suffering from the harm of environmental pollution, human beings began to pay attention to environmental protection and started to carry out pollutant prediction research. A large number of air pollutant predictions have tried to predict pollutants by revealing their evolution patterns, emphasizing the fitting analysis of time series but ignoring the spatial transmission effect of adjacent areas, leading to low prediction accuracy. To solve this problem, we propose a time series prediction network with the self-optimization ability of a spatio-temporal graph neural network (BGGRU) to mine the changing pattern of the time series and the spatial propagation effect. The proposed network includes spatial and temporal modules. The spatial module uses a graph sampling and aggregation network (GraphSAGE) in order to extract the spatial information of the data. The temporal module uses a Bayesian graph gated recurrent unit (BGraphGRU), which applies a graph network to the gated recurrent unit (GRU) so as to fit the data's temporal information. In addition, this study used Bayesian optimization to solve the problem of the model's inaccuracy caused by inappropriate hyperparameters of the model. The high accuracy of the proposed method was verified by the actual PM2.5 data of Beijing, China, which provided an effective method for predicting the PM2.5 concentration. [ABSTRACT FROM AUTHOR]

Traffic flow forecasting is of great significance to urban traffic control and public safety applications. The key challenge of traffic flow forecasting is how to capture the complex correlation of different time levels and learn time dependence. Some external information is closely related to traffic flow, such as accidental traffic accidents, weather, and Point of Interests (PoI) information. This paper proposes a deep learning-based model, called AttDeepSTN+, which is used to predict the inflow and outflow of each area of the entire city. Specifically, AttDeepSTN+ uses the structure of interactive attention and convolution to model the temporal closeness, trend, and periodicity of crowd flow, in the interactive attention layer, learn the importance of closeness to periodicity and trend respectively to model the long-term dependence of time, and then use feature fusion to capture complex correlations at different levels, thereby reducing model prediction accuracy. In addition, PoI information are combined with time factors to express the influence of location attributes on crowd flow, to learn prior knowledge of crowd flow. Finally, a new fusion mechanism is used to fuse the attention layer modules and PoI information and other information together into the module to capture the complex correlation between different levels of features, to predict the final crowd flow in each area, and further improve the prediction accuracy of the model. The New York City crowd flow experiment shows that the model is better than the current state-of-the-art baseline. [ABSTRACT FROM AUTHOR]

Multivariate time series prediction, especially in early fault warning for chemical processes, poses significant challenges. The advent of graph neural network (GNN) method has made breakthroughs in this domain by enabling the processing of topological data. However, the traditional methods suffer from the issue of over-smoothing and inability to capture intricate multi-scale spatio-temporal dependencies. Additionally, the existing graph structures fall short in describing the complex spatial relationships among multi-stage sensors, impeding their adaptability to dynamically evolving chain reaction scenarios. To alleviate these limitations, a novel Dynamic spatio-Temporal Network for early fault warning in chemical processes, named DSTN for short, is proposed in this paper. We extract the spatial and temporal features of the time series by the designed dynamic GNN and the improved Transformer network. Then, we integrate the spatio-temporal features through the residual network. DSTN has the following advantages: (1) A one-dimensional convolutional neural network is seamlessly incorporated into the Transformer architecture for bolstering its capacity to discern both global and local features within time series. (2) The continuous sliding window and mutual information methods are employed to construct a dynamic topology graph, and a K-order adjacency matrix is designed to rectify the inefficiencies in learning weights associated with convolution kernel parameters. (3) Multiple spatio-temporal modules interconnected via residual connection to adaptively fuse multi-scale features. Experimental results demonstrate that our proposed DSTN method outperforms existing methods in terms of both performance and interpretability in early fault warning of chemical processes. [Display omitted] • One-dimension convolution enhances Transformer's ability to learn global local feature weights. • The K-order adjacency matrix and sliding window are used to construct higher-order dynamic graphs. • The residual connects spatial and temporal features to improve the learning ability of the model. • The effectiveness and practicability of method are verified by two chemical process datasets. [ABSTRACT FROM AUTHOR]

Origin-Destination (OD) prediction which aims to predict the number of passenger's travel demands from one region to another, is critically important to many real applications including intelligent transportation systems and public safety. The challenges of this problem lie in both the dynamic patterns of the human mobility data and data sparsity in issue in some regions. Thus it is difficult to model the complex spatio-temporal correlations of the human mobility data to predict the OD of their trips. Meanwhile, the crowd flows in different regions of a city and the context features (e.g. holiday, weather and POIs) are potentially useful to alleviate the data sparsity issue and improve the OD prediction, but are largely ignored by existing works. In this paper, we propose a deep spatio-temporal framework which named Auxiliary-tasks Enhanced Spatio-Temporal Network (AEST) to more effectively address the OD prediction problem. AEST trains a model to conduct OD inference via learning crowd flow and external data as auxiliary task. The novel Hierarchical Convolutional LSTM (HC-LSTM) Network is proposed which combines CNN, GCN and LSTM to effectively capture spatiao-temporal correlations. In addition, we design a Contextual Network (ContextNet) which learns representations of contextual information to assist OD prediction. We conduct extensive experiments over bike and taxicab trip datasets in New York. The results show that our method is superior to the state-of-art approaches. [ABSTRACT FROM AUTHOR]

The environment and development are major issues of general concern. After much suffering from the harm of environmental pollution, human beings began to pay attention to environmental protection and started to carry out pollutant prediction research. A large number of air pollutant predictions have tried to predict pollutants by revealing their evolution patterns, emphasizing the fitting analysis of time series but ignoring the spatial transmission effect of adjacent areas, leading to low prediction accuracy. To solve this problem, we propose a time series prediction network with the self-optimization ability of a spatio-temporal graph neural network (BGGRU) to mine the changing pattern of the time series and the spatial propagation effect. The proposed network includes spatial and temporal modules. The spatial module uses a graph sampling and aggregation network (GraphSAGE) in order to extract the spatial information of the data. The temporal module uses a Bayesian graph gated recurrent unit (BGraphGRU), which applies a graph network to the gated recurrent unit (GRU) so as to fit the data’s temporal information. In addition, this study used Bayesian optimization to solve the problem of the model’s inaccuracy caused by inappropriate hyperparameters of the model. The high accuracy of the proposed method was verified by the actual PM2.5 data of Beijing, China, which provided an effective method for predicting the PM2.5 concentration.

• A behavioural multiparametric based pain assessment system is proposed. • Spatio-temporal features are used for facial expression based pain assessment system. • To learn body movement dynamics FCN based stacked BiLSTM model is implemented. • VGGish-SVM based model is proposed for sound based pain assessment system. • Decision level fusion is implemented to improve the performance of the system. • Introduced a new mutiparametric pain dataset which is collected from the hospital. Existing approaches for pain assessment are mostly based on facial expressions which cannot fully describe the expressiveness of pain. Although few researchers have also implemented multimodal approaches, they have not considered body movement for pain assessment. Besides, assessing pain in an unconstrained setting is challenging due to occlusions and poor illumination. Hence, to overcome these challenges we present a multi-stream framework for behavioural multiparametric pain assessment. To detect pain from face images a domain adaptation technique with a stacked BiLSTM network is implemented for joint spatio-temporal modelling. Further, a FCN-based-BiLSTM model is implemented to learn pain-related body dynamics. To learn pain-related features from pain sound VGG-based features are extracted and then a linear model is used for pain classification. Finally, a decision-level fusion approach is implemented to learn jointly from all pain classifiers. The proposed pain assessment system has achieved 92 % of accuracy after evaluating on self-created pain dataset. [ABSTRACT FROM AUTHOR]

We investigate the structural properties of a spatio-temporal network of earthquake events that incorporates magnitude information between the connected events. The network creates temporally directed links from an origin event towards a later event if it breaks the record closest distance from the origin among all the events in the catalog so far. Additionally, the links are conditionally classified based on the magnitude difference between connected events: 'up' ('down') connections point from a weaker (stronger) to a stronger (weaker) event. Using earthquake records from the Philippines from 1973 to 2012 and southern California from 1982 to 2012, we observe that the out-degree distributions show slight deviations from the corresponding Poisson distribution of the same mean. The space and time separations of connected earthquakes both show power-law regimes, suggesting spatio-temporal (self-)organization. More importantly, the conditional distributions of 'up' and 'down' connections in space, time, and network structure point to a higher likelihood of a stronger event triggering a nearby weaker event for the first few connections, as in the case of aftershocks. The results are captured by a sandpile-based model where a small but finite probability of preferentially targeting the most susceptible grid site is introduced. Our analysis, coupled with the discrete model analog, provides a quantitative picture of the spatio-temporal and magnitude organization of seismicity beyond just the successive events. The technique may be extended to further characterize similar long-period earthquake records to yield a more complete picture of the underlying processes involved in seismicity. [ABSTRACT FROM AUTHOR]

In this paper, we propose a spatio-temporal contextual network, STC-Flow, for optical flow estimation. Unlike previous optical flow estimation approaches with local pyramid feature extraction and multi-level correlation, we propose a contextual relation exploration architecture by capturing rich long-range dependencies in spatial and temporal dimensions. Specifically, STC-Flow contains three key context modules, i.e. , pyramidal spatial context module, temporal context correlation module and recurrent residual contextual upsampling module for the effect of feature extraction, correlation, and flow reconstruction, respectively. Experimental results demonstrate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on Sintel and KITTI datasets. • A general contextual attention framework of multi-level features & complicated fusion. • STC-Flow to capture rich spatio-temporal long-range dependencies for flow estimation. • PSC, TCC & RRCU for the effect of feature extraction, correlation, and reconstruction. • Promising performance in Sintel & KITTI for two-frame based optical flow estimation. [ABSTRACT FROM AUTHOR]

This study presents a methodology for estimating passenger's spatio-temporal trajectory with personalization and timeliness by using incomplete Wi-Fi probe data in urban rail transit network. Unlike the automatic fare collection data that only records passenger's entries and exits, the Wi-Fi probe data can capture more detailed passenger movements, such as riding a train or waiting on a platform. However, the estimation of spatio-temporal trajectories remains as a challenging task because a few unfavorable situations could result into deficient data. To address this problem, we first describe the Wi-Fi probe data and summarize their common defects. Then, the n -gram method is developed to infer missing spatio-temporal location information. Next, an estimation algorithm is designed to generate feasible spatio-temporal trajectories for each individual passenger by integrating multiple data sources, i.e., urban rail transit network topology, Wi-Fi probe data, train schedules, etc. This proposed method is tested on both simulated data in blind experiments and real-world data from a complex urban rail transit network. The results of case study show that 93% of passengers' unique physical routes can be estimated. Then, for 80% of passengers, the number of feasible spatio-temporal trajectories can be reduced to one or two. Potential applications of the trajectory estimation approach are also identified. • This study extends personalization and timeliness into the SST estimation. • A new algorithm is developed by integrating structurally heterogeneous data sources. • The estimated trajectory describes detailed information both on-board and at stations. • Real-world analyses are conducted for the URT network with the largest route length in China. • This is the first attempt to find out all the routes and trains at the network level. [ABSTRACT FROM AUTHOR]

Video super-resolution has become an emerging topic in the field of machine learning. The generative adversarial network is a framework that is widely used to develop solutions for low-resolution videos. Video surveillance using closed-circuit television (CCTV) is significant in every field, all over the world. A common problem with CCTV videos is sudden video loss or poor quality. In this paper, we propose a generative adversarial network that implements spatio-temporal generators and discriminators to enhance real-time low-resolution CCTV videos to high-resolution. The proposed model considers both foreground and background motion of a CCTV video and effectively models the spatial and temporal consistency from low-resolution video frames to generate high-resolution videos. Quantitative and qualitative experiments on benchmark datasets, including Kinetics-700, UCF101, HMDB51 and IITH_Helmet2, showed that our model outperforms the existing GAN models for video super-resolution. [ABSTRACT FROM AUTHOR]

Network ecology is an emerging field that allows researchers to conceptualize and analyse ecological networks and their dynamics. Here, we focus on the dynamics of ecological networks in response to environmental changes. Specifically, we formalize how network topologies constrain the dynamics of ecological systems into a unifying framework in network ecology that we refer to as the 'ecological network dynamics framework'. This framework stresses that the interplay between species interaction networks and the spatial layout of habitat patches is key to identifying which network properties (number and weights of nodes and links) and trade-offs among them are needed to maintain species interactions in dynamic landscapes. We conclude that to be functional, ecological networks should be scaled according to species dispersal abilities in response to landscape heterogeneity. Determining how such effective ecological networks change through space and time can help reveal their complex dynamics in a changing world.