Search

Understanding the decision-making process of drivers is one of the keys to ensuring road safety. While the driver intent and the resulting ego-motion trajectory are valuable in developing driver-assistance systems, existing methods mostly focus on the motions of other vehicles. In contrast, we focus on inferring the ego trajectory of a driver's vehicle using their gaze data. For this purpose, we first collect a new dataset, GEM, which contains high-fidelity ego-motion videos paired with drivers' eye-tracking data and GPS coordinates. Next, we develop G-MEMP, a novel multimodal ego-trajectory prediction network that combines GPS and video input with gaze data. We also propose a new metric called Path Complexity Index (PCI) to measure the trajectory complexity. We perform extensive evaluations of the proposed method on both GEM and DR(eye)VE, an existing benchmark dataset. The results show that G-MEMP significantly outperforms state-of-the-art methods in both benchmarks. Furthermore, ablation studies demonstrate over 20% improvement in average displacement using gaze data, particularly in challenging driving scenarios with a high PCI. The data, code, and models can be found at https://eth-ait.github.io/g-memp/.

The task of predicting 3D eye gaze from eye images can be performed either by (a) end-to-end learning for image-to-gaze mapping or by (b) fitting a 3D eye model onto images. The former case requires 3D gaze labels, while the latter requires eye semantics or landmarks to facilitate the model fitting. Although obtaining eye semantics and landmarks is relatively easy, fitting an accurate 3D eye model on them remains to be very challenging due to its ill-posed nature in general. On the other hand, obtaining large-scale 3D gaze data is cumbersome due to the required hardware setups and computational demands. In this work, we propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. The proposed method combines the best of both worlds by leveraging large amounts of weak annotations--which are easy to obtain, and only a few 3D gaze vectors--which alleviate the difficulty of fitting 3D eye models on the semantic segmentation of eye images. Thus, the eye gaze vectors, used in the model fitting, are directly supervised using the few-shot gaze labels. Additionally, we propose a transformer-based network architecture, that serves as a solid baseline for our improvements. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 degrees lower angular gaze error over the baseline, when only 0.05% 3D annotations of the training images are used. The source code is available at https://github.com/dimitris-christodoulou57/Model-aware_3D_Eye_Gaze., Comment: Accepted to ISMAR2023 as a poster paper

Novel view synthesis and 3D modeling using implicit neural field representation are shown to be very effective for calibrated multi-view cameras. Such representations are known to benefit from additional geometric and semantic supervision. Most existing methods that exploit additional supervision require dense pixel-wise labels or localized scene priors. These methods cannot benefit from high-level vague scene priors provided in terms of scenes' descriptions. In this work, we aim to leverage the geometric prior of Manhattan scenes to improve the implicit neural radiance field representations. More precisely, we assume that only the knowledge of the indoor scene (under investigation) being Manhattan is known -- with no additional information whatsoever -- with an unknown Manhattan coordinate frame. Such high-level prior is used to self-supervise the surface normals derived explicitly in the implicit neural fields. Our modeling allows us to cluster the derived normals and exploit their orthogonality constraints for self-supervision. Our exhaustive experiments on datasets of diverse indoor scenes demonstrate the significant benefit of the proposed method over the established baselines. The source code is available at https://github.com/nikola3794/normal-clustering-nerf., Comment: Paper accepted to ICCV23

We study delayed loss of stability in a class of fast-slow systems with two fast variables and one slow one, where the linearisation of the fast vector field along a one-dimensional critical manifold has two real eigenvalues which intersect before the accumulated contraction and expansion are balanced along any individual eigendirection. That interplay between eigenvalues and eigendirections renders the use of known entry-exit relations unsuitable for calculating the point at which trajectories exit neighbourhoods of the given manifold. We illustrate the various qualitative scenarios that are possible in the class of systems considered here, and we propose novel formulae for the entry-exit functions that underlie the phenomenon of delayed loss of stability therein.

We study a singularly perturbed fast-slow system of two partial differential equations (PDEs) of reaction-diffusion type on a bounded domain. We assume that the reaction terms in the fast variable contain a fold singularity, whereas the slow variable assumes the role of a dynamic bifurcation parameter, thus extending the classical analysis of a fast-slow dynamic fold bifurcation to an infinite-dimensional setting. Our approach combines a spectral Galerkin discretisation with techniques from Geometric Singular Perturbation Theory (GSPT) which are applied to the resulting high-dimensional systems of ordinary differential equations (ODEs). In particular, we show the existence of invariant manifolds away from the fold singularity, while the dynamics in a neighbourhood of the singularity is described by geometric desingularisation, via the blow-up technique. Finally, we relate the Galerkin manifolds that are obtained after the discretisation to the invariant manifolds which exist in the phase space of the original system of PDEs., Comment: 41 pages with 9 figures. Corresponding author: Thomas Zacharis

We study the three-timescale dynamics of a model that describes the El Ni\~no Southern Oscillation (ENSO) phenomenon, which was proposed in [A. Roberts, J. Guckenheimer, E. Widiasih, A. Timmermann, and C. K. Jones, Mixed-mode oscillations of El Ni\~no--Southern Oscillation, Journal of the Atmospheric Sciences, 73 (2016), pp. 1755--1766]. While ENSO phenomena are inherently characterised by the presence of multiple distinct timescales, the above model has previously been studied in a two-timescale context only. Here, we uncover the geometric mechanisms that are responsible for complex oscillatory dynamics in a three-timescale regime, and we demonstrate that the system exhibits a variety of qualitatively different behaviours, such as mixed-mode oscillation (MMO) with ``plateaus" -- trajectories where epochs of quiescence alternate with dramatic excursions -- and relaxation oscillation. The latter, although emergent also in the two-timescale context in appropriate parameter regimes, had not been documented previously. Moreover, we show that these mechanisms are relevant to models from other fields of ecological and population dynamics, as the underlying geometry is similar to the unfolding of Rosenzweig--MacArthur-type models in three dimensions.

One popular group of defense techniques against adversarial attacks is based on injecting stochastic noise into the network. The main source of robustness of such stochastic defenses however is often due to the obfuscation of the gradients, offering a false sense of security. Since most of the popular adversarial attacks are optimization-based, obfuscated gradients reduce their attacking ability, while the model is still susceptible to stronger or specifically tailored adversarial attacks. Recently, five characteristics have been identified, which are commonly observed when the improvement in robustness is mainly caused by gradient obfuscation. It has since become a trend to use these five characteristics as a sufficient test, to determine whether or not gradient obfuscation is the main source of robustness. However, these characteristics do not perfectly characterize all existing cases of gradient obfuscation, and therefore can not serve as a basis for a conclusive test. In this work, we present a counterexample, showing this test is not sufficient for concluding that gradient obfuscation is not the main cause of improvements in robustness.

In most scenarios, conditional image generation can be thought of as an inversion of the image understanding process. Since generic image understanding involves solving multiple tasks, it is natural to aim at generating images via multi-conditioning. However, multi-conditional image generation is a very challenging problem due to the heterogeneity and the sparsity of the (in practice) available conditioning labels. In this work, we propose a novel neural architecture to address the problem of heterogeneity and sparsity of the spatially multi-conditional labels. Our choice of spatial conditioning, such as by semantics and depth, is driven by the promise it holds for better control of the image generation process. The proposed method uses a transformer-like architecture operating pixel-wise, which receives the available labels as input tokens to merge them in a learned homogeneous space of labels. The merged labels are then used for image generation via conditional generative adversarial training. In this process, the sparsity of the labels is handled by simply dropping the input tokens corresponding to the missing labels at the desired locations, thanks to the proposed pixel-wise operating architecture. Our experiments on three benchmark datasets demonstrate the clear superiority of our method over the state-of-the-art and compared baselines. The source code will be made publicly available.

We present a novel and global three-dimensional reduction of a non-dimensionalised version of the four-dimensional Hodgkin-Huxley equations [J. Rubin and M. Wechselberger, Giant squid--hidden canard: the 3D geometry of the Hodgkin-Huxley model, Biological Cybernetics, 97 (2007), pp. 5--32] that is based on geometric singular perturbation theory (GSPT). We investigate the dynamics of the resulting three-dimensional system in two parameter regimes in which the flow evolves on three distinct timescales. Specifically, we demonstrate that the system exhibits bifurcations of oscillatory dynamics and complex mixed-mode oscillations (MMOs), in accordance with the geometric mechanisms introduced in [P. Kaklamanos, N. Popovi\'c, and K. U. Kristiansen, Bifurcations of mixed--mode oscillations in three--timescale systems: An extended prototypical example, Chaos: An Interdisciplinary Journal of Nonlinear Science, 32 (2022), p. 013108], and we classify the various firing patterns in terms of the external applied current. While such patterns have been documented in [S. Doi, S. Nabetani, and S. Kumagai, Complex nonlinear dynamics of the Hodgkin-Huxley equations induced by time scale changes, Biological Cybernetics, 85 (2001), pp. 51--64] for the multiple-timescale Hodgkin-Huxley equations, we elucidate the geometry that underlies the transitions between them, which had not been previously emphasised.

We introduce token-consistent stochastic layers in vision transformers, without causing any severe drop in performance. The added stochasticity improves network calibration, robustness and strengthens privacy. We use linear layers with token-consistent stochastic parameters inside the multilayer perceptron blocks, without altering the architecture of the transformer. The stochastic parameters are sampled from the uniform distribution, both during training and inference. The applied linear operations preserve the topological structure, formed by the set of tokens passing through the shared multilayer perceptron. This operation encourages the learning of the recognition task to rely on the topological structures of the tokens, instead of their values, which in turn offers the desired robustness and privacy of the visual features. The effectiveness of the token-consistent stochasticity is demonstrated on three different applications, namely, network calibration, adversarial robustness, and feature privacy, by boosting the performance of the respective established baselines., Comment: This article is under consideration at the Computer Vision and Image Understanding journal

This paper investigates the role of global context for crowd counting. Specifically, a pure transformer is used to extract features with global information from overlapping image patches. Inspired by classification, we add a context token to the input sequence, to facilitate information exchange with tokens corresponding to image patches throughout transformer layers. Due to the fact that transformers do not explicitly model the tried-and-true channel-wise interactions, we propose a token-attention module (TAM) to recalibrate encoded features through channel-wise attention informed by the context token. Beyond that, it is adopted to predict the total person count of the image through regression-token module (RTM). Extensive experiments on various datasets, including ShanghaiTech, UCF-QNRF, JHU-CROWD++ and NWPU, demonstrate that the proposed context extraction techniques can significantly improve the performance over the baselines., Comment: Accepted by Machine Intelligence Research (MIR)

Mixed-mode oscillations (MMOs) are complex oscillatory patterns in which large-amplitude relaxation oscillations (LAOs) alternate with small-amplitude oscillations (SAOs). MMOs are found in singularly perturbed systems of ordinary differential equations of slow-fast type, and are typically related to the presence of so-called folded singularities and the corresponding canard trajectories in such systems. Here, we introduce a canonical family of three-dimensional slow-fast systems that exhibit MMOs which are induced by relaxation-type dynamics, and which are hence based on a "jump mechanism", rather than on a more standard canard mechanism. In particular, we establish a correspondence between that family and a class of associated one-dimensional piecewise affine maps (PAMs) which exhibit MMOs with the same signature. Finally, we give a preliminary classification of admissible mixed-mode signatures, and we illustrate our findings with numerical examples., Comment: 34 pages total, 11 figures, 2 tables

We define the concept of CompositeTasking as the fusion of multiple, spatially distributed tasks, for various aspects of image understanding. Learning to perform spatially distributed tasks is motivated by the frequent availability of only sparse labels across tasks, and the desire for a compact multi-tasking network. To facilitate CompositeTasking, we introduce a novel task conditioning model -- a single encoder-decoder network that performs multiple, spatially varying tasks at once. The proposed network takes an image and a set of pixel-wise dense task requests as inputs, and performs the requested prediction task for each pixel. Moreover, we also learn the composition of tasks that needs to be performed according to some CompositeTasking rules, which includes the decision of where to apply which task. It not only offers us a compact network for multi-tasking, but also allows for task-editing. Another strength of the proposed method is demonstrated by only having to supply sparse supervision per task. The obtained results are on par with our baselines that use dense supervision and a multi-headed multi-tasking design. The source code will be made publicly available at www.github.com/nikola3794/composite-tasking.

We study a class of multi-parameter three-dimensional systems of ordinary differential equations that exhibit dynamics on three distinct timescales. We apply geometric singular perturbation theory to explore the dependence of the geometry of these systems on their parameters, with a focus on mixed-mode oscillations (MMOs) and their bifurcations. In particular, we uncover a novel geometric mechanism that encodes the transition from MMOs with single epochs of small-amplitude oscillations (SAOs) to those with double-epoch SAOs. We identify a relatively simple prototypical three-timescale system that realises our mechanism, featuring a one-dimensional $S$-shaped supercritical manifold that is embedded into a two-dimensional $S$-shaped critical manifold in a symmetric fashion. We show that the Koper model from chemical kinetics is merely a particular realisation of that prototypical system for a specific choice of parameters; in particular, we explain the robust occurrence of mixed-mode dynamics with double epochs of SAOs therein. Finally, we argue that our geometric mechanism can elucidate the mixed-mode dynamics of more complicated systems with a similar underlying geometry, such as of a three-dimensional, three-timescale reduction of the Hodgkin-Huxley equations from mathematical neuroscience.

Micro-Electro Mechanical Systems (MEMS) are defined as very small structures that combine electrical and mechanical components on a common substrate. Here, the electrostatic-elastic case is considered, where an elastic membrane is allowed to deflect above a ground plate under the action of an electric potential, whose strength is proportional to a parameter $\lambda$. Such devices are commonly described by a parabolic partial differential equation that contains a singular nonlinear source term. The singularity in that term corresponds to the so-called "touchdown" phenomenon, where the membrane establishes contact with the ground plate. Touchdown is known to imply the non-existence of steady state solutions and blow-up of solutions in finite time. We study a recently proposed extension of that canonical model, where such singularities are avoided due to the introduction of a regularizing term involving a small "regularization" parameter $\varepsilon$. Methods from dynamical systems and geometric singular perturbation theory, in particular the desingularization technique known as "blow-up", allow for a precise description of steady-state solutions of the regularized model, as well as for a detailed resolution of the resulting bifurcation diagram. The interplay between the two main model parameters $\varepsilon$ and $\lambda$ is emphasized; in particular, the focus is on the singular limit as both parameters tend to zero.

Few analytical methods exist for quantitative studies of large fluctuations in stochastic systems. In this article, we develop a simple diagrammatic approach to the Chemical Master Equation that allows us to calculate multi-time correlation functions which are accurate to a any desired order in van Kampen's system size expansion. Specifically, we present a set of Feynman rules from which this diagrammatic perturbation expansion can be constructed algorithmically. We then apply the methodology to derive in closed form the leading order corrections to the linear noise approximation of the intrinsic noise power spectrum for general biochemical reaction networks. Finally, we illustrate our results by describing noise-induced oscillations in the Brusselator reaction scheme which are not captured by the common linear noise approximation., Comment: 25 pages, 5 figures, under review, Journal of Physics A