Search

Deep probabilistic forecasting is gaining attention across numerous applications. From weather prognosis, electricity consumption estimation, traffic flow prediction, to autonomous vehicle trajectory prediction. However, existing approaches focus on improving on average metrics without addressing the long-tailed distribution of errors. This thesis identifies long tail behavior in the error distribution of state-of-the-art deep learning methods for probabilistic forecasting. We analyze potential sources and explanations for this behavior. Further, we present two loss augmentation methods to mitigate tailedness pf error distributions: Pareto Loss and Kurtosis Loss. Both methods modify the loss using the concept of moments to penalize higher loss samples. Kurtosis Loss uses a symmetric measure, the fourth moment, while Pareto Loss uses an asymmetric measure of right-tailedness and models loss using a Generalized Pareto Distribution (GPD). We demonstrate the performance of our methods on several real-world datasets, including time series and spatiotemporal trajectories, achieving significant improvements on tail error metrics, while maintaining and even improving upon average error metrics.

In hopes of lessening the reliance on fossil fuels, Plug-in Hybrid Electric Vehicles (PHEVs) have become an attractive option as an alternative fuel vehicle due to their larger electric motors and energy storage systems (ESS). PHEVs can propel themself relying solely on their internal combustion engine (ICE), electric motor (EM), and or a hybrid of both. To improve their fuel efficiency, many studies have been done to investigate the use of a priori route information to optimize the use of a PHEV’s ICE and EM. This study introduces a real-time machine learning application of a control strategy known as Trajectory Forecasting (TF). TF takes a priori knowledge of a PHEV’s pre-planned route to determine when the vehicle will use its different forms of propulsion in the form of propulsion mode scheduling. However, it assumes constant route data such as traffic and resulting driving speed for its scheduling to be applicable. To automatically account for changing traffic as well as choose better alternative routes, this study looks at the use of a Convolutional Neural Network (CNN) to simulate a PHEV’s operation along available routes beforehand according to the rules of TF to choose a route that best satisfies a driver’s want, better fuel efficiency and possibly lower emissions. This new real-time TF-based machine learning control strategy is evaluated and compared to common PHEV control strategies such as Charge Sustaining (CS) and Charge Depletion (CD) using National Renewable Energy Laboratory’s vehicle simulator ADVISOR. Results show possible increases in Mpgge from 1.72%-130%, decreases in emitted hydrocarbons (HC), carbon monoxide (CO), and nitrous oxides (NOx) from 0.05%-70%, and 0.05%-90.35% reduction in gasoline consumption depending on overall route length and PHEV configuration.

Since the past few decades, human trajectory forecasting has been a field of active research owing to its numerous real-world applications: evacuation situation analysis, deployment of intelligent transport systems, traffic operations, to name a few. Early works handcrafted this representation based on domain knowledge. However, social interactions in crowded environments are not only diverse but often subtle. Recently, deep learning methods have outperformed their handcrafted counterparts, as they learned about human-human interactions in a more generic data-driven fashion. In this work, we present an in-depth analysis of existing deep learning-based methods for modelling social interactions. We propose two knowledge-based data-driven methods to effectively capture these social interactions. To objectively compare the performance of these interaction-based forecasting models, we develop a large scale interaction-centric benchmark TrajNet++, a significant yet missing component in the field of human trajectory forecasting. We propose novel performance metrics that evaluate the ability of a model to output socially acceptable trajectories. Experiments on TrajNet++ validate the need for our proposed metrics, and our method outperforms competitive baselines on both real-world and synthetic datasets., IEEE format, Layer-wise Relevance Propagation added

In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents — a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https://github.com/MIPT-Oulu/CLIMAT.

Pedestrian motion behavior involves a combination of individual goals and social interactions with other agents. In this article, we present an asymmetrical bidirectional recurrent neural network architecture called U-RNN to encode pedestrian trajectories and evaluate its relevance to replace LSTMs for various forecasting models. Experimental results on the Trajnet++ benchmark show that the U-LSTM variant yields better results regarding every available metrics (ADE, FDE, Collision rate) than common trajectory encoders for a variety of approaches and interaction modules, suggesting that the proposed approach is a viable alternative to the de facto sequence encoding RNNs. Our implementation of the asymmetrical Bi-RNNs for the Trajnet++ benchmark is available at: github.com/JosephGesnouin/Asymmetrical-Bi-RNNs-to-encode-pedestrian-trajectories, Comment: 7 pages

Human Trajectory Prediction (HTP) has gained much momentum in the last years and many solutions have been proposed to solve it. Proper benchmarking being a key issue for comparing methods, this paper addresses the question of evaluating how complex is a given dataset with respect to the prediction problem. For assessing a dataset complexity, we define a series of indicators around three concepts: Trajectory predictability; Trajectory regularity; Context complexity. We compare the most common datasets used in HTP in the light of these indicators and discuss what this may imply on benchmarking of HTP algorithms. Our source code is released on Github., Comment: ACCV2020

Anticipating human motion in crowded scenarios is essential for developing intelligent transportation systems, social-aware robots and advanced video surveillance applications. A key component of this task is represented by the inherently multi-modal nature of human paths which makes socially acceptable multiple futures when human interactions are involved. To this end, we propose a generative architecture for multi-future trajectory predictions based on Conditional Variational Recurrent Neural Networks (C-VRNNs). Conditioning mainly relies on prior belief maps, representing most likely moving directions and forcing the model to consider past observed dynamics in generating future positions. Human interactions are modeled with a graph-based attention mechanism enabling an online attentive hidden state refinement of the recurrent estimation. To corroborate our model, we perform extensive experiments on publicly-available datasets (e.g., ETH/UCY, Stanford Drone Dataset, STATS SportVU NBA, Intersection Drone Dataset and TrajNet++) and demonstrate its effectiveness in crowded scenes compared to several state-of-the-art methods., Accepted at Computer Vision and Image Understanding (CVIU)

In this work, we explore the correlation between people trajectories and their head orientations. We argue that people trajectory and head pose forecasting can be modelled as a joint problem. Recent approaches on trajectory forecasting leverage short-term trajectories (aka tracklets) of pedestrians to predict their future paths. In addition, sociological cues, such as expected destination or pedestrian interaction, are often combined with tracklets. In this paper, we propose MiXing-LSTM (MX-LSTM) to capture the interplay between positions and head orientations (vislets) thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. We additionally exploit the head orientations as a proxy for the visual attention, when modeling social interactions. MX-LSTM predicts future pedestrians location and head pose, increasing the standard capabilities of the current approaches on long-term trajectory forecasting. Compared to the state-of-the-art, our approach shows better performances on an extensive set of public benchmarks. MX-LSTM is particularly effective when people move slowly, i.e. the most challenging scenario for all other models. The proposed approach also allows for accurate predictions on a longer time horizon., Comment: Accepted at IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2019. arXiv admin note: text overlap with arXiv:1805.00652

In this paper we show the importance of the head pose estimation in the task of trajectory forecasting. This cue, when produced by an oracle and injected in a novel socially-based energy minimization approach, allows to get state-of-the-art performances on four different forecasting benchmarks, without relying on additional information such as expected destination and desired speed, which are supposed to be know beforehand for most of the current forecasting techniques. Our approach uses the head pose estimation for two aims: 1) to define a view frustum of attention, highlighting the people a given subject is more interested about, in order to avoid collisions; 2) to give a shorttime estimation of what would be the desired destination point. Moreover, we show that when the head pose estimation is given by a real detector, though the performance decreases, it still remains at the level of the top score forecasting systems.

Spatio-temporal data is a commonly used source of information. Using machine learning to analyse this kind of data can lead to many interesting and useful insights. In this thesis project, a novel public transportation spatio-temporal dataset is explored and analysed. The dataset contains 282 GB of positional events, spanning two weeks of time, from all public transportation vehicles in Östergötland county, Sweden. From the data exploration, three high-level problems are formulated: bus stop detection, GPS variance estimation, and arrival time prediction, also called trajectory forecasting. The bus stop detection problem is briefly discussed and solutions are proposed. Gaussian process regression is an effective method for solving regression problems. The GPS variance estimation problem is solved via the use of a mixture of Gaussian processes. A mixture of Gaussian processes is also used to predict the arrival time for public transportation buses. The arrival time prediction is from one bus stop to the next, not for the whole trajectory. The result from the arrival time prediction is a distribution of arrival times, which can easily be applied to determine the earliest and latest expected arrival to the next bus stop, alongside the most probable arrival time. The naïve arrival time prediction model implemented has a root mean square error of 5 to 19 seconds. In general, the absolute error of the prediction model decreases over time in each respective segment. The results from the GPS variance estimation problem is a model which can compare the variance for different environments along the route on a given trajectory.

Human trajectory forecasting in crowds presents the challenges of modelling social interactions and outputting collision-free multimodal distribution. Following the success of Social Generative Adversarial Networks (SGAN), recent works propose various GAN-based designs to better model human motion in crowds. Despite superior performance in reducing distance-based metrics, current networks fail to output socially acceptable trajectories, as evidenced by high collisions in model predictions. To counter this, we introduce SGANv2: an improved safety-compliant SGAN architecture equipped with spatio-temporal interaction modelling and a transformer-based discriminator. The spatio-temporal modelling ability helps to learn the human social interactions better while the transformer-based discriminator design improves temporal sequence modelling. Additionally, SGANv2 utilizes the learned discriminator even at test-time via a collaborative sampling strategy that not only refines the colliding trajectories but also prevents mode collapse, a common phenomenon in GAN training. Through extensive experimentation on multiple real-world and synthetic datasets, we demonstrate the efficacy of SGANv2 to provide socially-compliant multimodal trajectories.

The ability to forecast human motion, called ``human trajectory forecasting", is a critical requirement for mobility applications such as autonomous driving and robot navigation. Humans plan their path taking into account what might happen in the future. Similarly, the decision-making algorithm of autonomous systems should predict how their environment will evolve in the future. This thesis focuses on developing deep learning methods for forecasting human motion. In the first part of this thesis, we tackle the fundamental challenges of social interaction modelling and multimodality. Social interactions dictate how the motion of a human is affected by others. Current deep learning methods often struggle to model these interactions between trajectory sequences. To promote interaction-awareness in forecasting models, we develop a training paradigm that explicitly focuses on samples that undergo interactions and incorporates model uncertainty. Furthermore, we build a taxonomy of existing interaction encoders and propose an optimal design that is robust to the real-world noise. In addition to modelling interactions, a good trajectory forecasting model must account for the multimodal nature of the prediction, i.e., the possibility of having multiple plausible futures given the past observations. To tackle multimodality, we present a socially-aware generative adversarial network that leverages recent advances in sequence modelling, and has the ability to model the temporal evolution of social interactions. Furthermore, we develop a collaborative sampling technique that refines the bad generated predictions at test time. In the second part of this thesis, we focus on two challenges specific to the real-world deployment of forecasting models: interpretability and adaptability. While neural networks have the capacity to learn complex interactions, it is difficult to understand the reason behind their predictions. Thus, we develop a framework that combines the interpretability of the classical models with the predictive power of neural networks. With regards to adaptability, existing deep forecasting models suffer from inferior performance when they encounter novel scenarios. We develop a strategy to adapt a pre-trained forecasting model to a target domain using limited samples. In particular, we introduce motion style adapters that identify and adjust the target domain-specific features. Throughout this thesis, experiments on synthetic and real-world forecasting datasets validate the effectiveness of our proposed methods.

Despite promising progress in reinforcement learning (RL), developing algorithms for autonomous driving (AD) remains challenging: one of the critical issues being the absence of an open-source platform capable of training and effectively validating the RL policies on real-world data. We propose DriverGym, an open-source OpenAI Gym-compatible environment specifically tailored for developing RL algorithms for autonomous driving. DriverGym provides access to more than 1000 hours of expert logged data and also supports reactive and data-driven agent behavior. The performance of an RL policy can be easily validated on real-world data using our extensive and flexible closed-loop evaluation protocol. In this work, we also provide behavior cloning baselines using supervised learning and RL, trained in DriverGym. We make DriverGym code, as well as all the baselines publicly available to further stimulate development from the community., Accepted to NeurIPS 2021 ML4AD Workshop

Human ability to foresee the near future plays a key role in everyone's life to prevent potentially dangerous situations. To be able to make predictions is crucial when people have to interact with the surrounding environment. Modeling such capability can lead to the design of automated warning systems and provide moving robots with an intelligent way of interaction with changing situation. In this work we focus on a typical urban human-scene where we aim at predicting an agent's behavior using a stochastic model. In this approach, we fuse the various factors that would contribute to a human motion in different contexts. Our method uses previously observed trajectories to build point-wise circular distributions that after combination, provide a statistical smooth prediction towards the most likely areas. More specifically, a ray-launching procedure, based on a semantic segmentation, gives a coarse scene representation for collision avoidance; a nearly-constant velocity dynamic model smooths the acceleration progression and knowledge of the agent's destination may further steer the path prediction. Experimental results in structured scenes, validate the effectiveness of the method in predicting paths in comparison to actual trajectories.

Human trajectory forecasting in crowds, at its core, is a sequence prediction problem with specific challenges of capturing inter-sequence dependencies (social interactions) and consequently predicting socially-compliant multimodal distributions. In recent years, neural network-based methods have been shown to outperform hand-crafted methods on distance-based metrics. However, these data-driven methods still suffer from one crucial limitation: lack of interpretability. To overcome this limitation, we leverage the power of discrete choice models to learn interpretable rule-based intents, and subsequently utilise the expressibility of neural networks to model scene-specific residual. Extensive experimentation on the interaction-centric benchmark TrajNet++ demonstrates the effectiveness of our proposed architecture to explain its predictions without compromising the accuracy., To appear in Computer Vision and Pattern Recognition (CVPR) 2021