Search

In this work, we study amodal video instance segmentation for automated driving. Previous works perform amodal video instance segmentation relying on methods trained on entirely labeled video data with techniques borrowed from standard video instance segmentation. Such amodally labeled video data is difficult and expensive to obtain and the resulting methods suffer from a trade-off between instance segmentation and tracking performance. To largely solve this issue, we propose to study the application of foundation models for this task. More precisely, we exploit the extensive knowledge of the Segment Anything Model (SAM), while fine-tuning it to the amodal instance segmentation task. Given an initial video instance segmentation, we sample points from the visible masks to prompt our amodal SAM. We use a point memory to store those points. If a previously observed instance is not predicted in a following frame, we retrieve its most recent points from the point memory and use a point tracking method to follow those points to the current frame, together with the corresponding last amodal instance mask. This way, while basing our method on an amodal instance segmentation, we nevertheless obtain video-level amodal instance segmentation results. Our resulting S-AModal method achieves state-of-the-art results in amodal video instance segmentation while resolving the need for amodal video-based labels. Code for S-AModal is available at https://github.com/ifnspaml/S-AModal., Comment: accepted at ECCV VCAD Workshop 2024

Recently, BigVGAN has emerged as high-performance speech vocoder. Its sequence-to-sequence-based synthesis, however, prohibits usage in low-latency conversational applications. Our work addresses this shortcoming in three steps. First, we introduce low latency into BigVGAN via implementing causal convolutions, yielding decreased performance. Second, to regain performance, we propose a teacher-student transfer learning scheme to distill the high-delay non-causal BigVGAN into our low-latency causal vocoder. Third, taking advantage of a self-supervised learning (SSL) model, in our case wav2vec 2.0, we align its encoder speech representations extracted from our low-latency causal vocoder to the ground truth ones. In speaker-independent settings, both proposed training schemes notably elevate the performance of our low-latency vocoder, closing up to the original high-delay BigVGAN. At only 21% higher complexity, our best small causal vocoder achieves 3.96 PESQ and 1.25 MCD, excelling even the original small non-causal BigVGAN (3.64 PESQ) by 0.32 PESQ and 0.1 MCD points, respectively., Comment: Accepted at IWAENC 2024

Training a linear classifier or lightweight model on top of pretrained vision model outputs, so-called 'frozen features', leads to impressive performance on a number of downstream few-shot tasks. Currently, frozen features are not modified during training. On the other hand, when networks are trained directly on images, data augmentation is a standard recipe that improves performance with no substantial overhead. In this paper, we conduct an extensive pilot study on few-shot image classification that explores applying data augmentations in the frozen feature space, dubbed 'frozen feature augmentation (FroFA)', covering twenty augmentations in total. Our study demonstrates that adopting a deceptively simple pointwise FroFA, such as brightness, can improve few-shot performance consistently across three network architectures, three large pretraining datasets, and eight transfer datasets., Comment: CVPR 2024 (18 pages, main paper + supplementary material)

Hypothesis:Diffusion in confinement is an important fundamental problem with significant implications for applications of supported liquid phases. However, resolving the spatially dependent diffusion coefficient, parallel and perpendicular to interfaces, has been a standing issue and for objects of nanometric size, which structurally fluctuate on a similar time scale as they diffuse, no methodology has been established so far. We hypothesise that the complex, coupled dynamics can be captured and analysed by using a model built on the $2$-dimensional Smoluchowski equation and systematic coarse-graining. Methods and simulations: For large, flexible species, a universal approach is offered that does not make any assumptions about the separation of time scales between translation and other degrees of freedom. The method is validated on Molecular Dynamics simulations of bulk systems of a family of ionic liquids with increasing cation sizes where internal degrees of freedom have little to major effects. Findings: After validation on bulk liquids, where we provide an interpretation of two diffusion constants for each species found experimentally, we clearly demonstrate the anisotropic nature of diffusion coefficients at interfaces. Spatial variations in the diffusivities relate to interface-induced structuring of the ionic liquids. Notably, the length scales in strongly confined ionic liquids vary consistently but differently at the solid-liquid and liquid-vapour interfaces., Comment: 21 pages, 14 figures, 5 tables, pdflatex, submitted to JCIS; This is a follow-up to "Anisotropic molecular diffusion in confinement I: Transport of small particles in potential and density gradients" due to a split of arXiv:archive/2212.09545

The Stokes-Einstein-Sutherland (SES) equation is at the foundation of statistical physics, relating a particle's diffusion coefficient and size with the fluid viscosity, temperature and the boundary condition for the particle-solvent interface. It is assumed that it relies on the separation of scales between the particle and the solvent, hence it is expected to break down for diffusive transport on the molecular scale. This assumption is however challenged by a number of experimental studies showing a remarkably small, if any, violation, while simulations systematically report the opposite. To understand these discrepancies, analytical ultracentrifugation experiments are combined with molecular simulations, both performed at unprecedented accuracies, to study the transport of buckminsterfullerene C60 in toluene at infinite dilution. This system is demonstrated to clearly violate the conditions of slow momentum relaxation. Yet, through a linear response to a constant force, the SES equation can be recovered in the long time limit with no more than 4% uncertainty both in experiments and in simulations. This nonetheless requires partial slip on the particle interface, extracted consistently from all the data. These results, thus, resolve a long-standing discussion on the validity and limits of the SES equation at the molecular scale., Comment: The data that support the findings of this study are openly available in Zenodo at http://doi.org/10.5281/zenodo.8281244, reference number 8281244

Hypothesis: Diffusion in confinement is an important fundamental problem with significant implications for applications of supported liquid phases. However, resolving the spatially dependent diffusion coefficient, parallel and perpendicular to interfaces, has been a standing issue. In the vicinity of interfaces, density fluctuations as a consequence of layering locally impose statistical drift, which impedes the analysis of spatially dependent diffusion coefficients even further. We hypothesise, that we can derive a model to spatially resolve interface-perpendicular diffusion coefficients based on local lifetime statistics with an extension to explicitly account for the effect of local drift using the Smoluchowski equation, that allows us to resolve anisotropic and spatially dependent diffusivity landscapes at interfaces. Methods and simulations: An analytic relation between local crossing times in system slices and diffusivity as well as an explicit term for calculating drift-induced systematic errors is presented. The method is validated on Molecular Dynamics simulations of bulk water and applied to simulations of water in slit pores. Findings: After validation on bulk liquids, we clearly demonstrate the anisotropic nature of diffusion coefficients at interfaces. Significant spatial variations in the diffusivities correlate with interface-induced structuring but cannot be solely attributed to the drift induced by local density fluctuations., Comment: 13 pages, 6 figures, submitted to JCIS, PDFLaTex; updated for submission to new Journal and split in two due to length; second accompanying publication: "Anisotropic molecular diffusion in confinement II: A model for structurally complex particles applied to transport in thin ionic liquid films" (both due to a split of arXiv:archive/2212.09545 )

Accurate modeling of diffusive transport of nanoparticles across nanopores is a particularly challenging problem. The reason is that for such narrow pores the large surface-to-volume ratio amplifies the relevance of the nanoscopic details and of the effective interactions at the interface with pore walls. Close to the pore wall, there is no clear separation between the length scales associated with molecular interactions, layering of the solvent at the interface with the pore and the particle size. Therefore, the standard hydrodynamic arguments may not apply and alternative solutions to determining average transport coefficients need to be developed. We here address this problem by offering a multiscale ansatz that uses effective potentials determined from molecular dynamics simulations to parametrise a four state stochastic model for the positional configuration of the particle in the pore. This is in turn combined with diffusivities in the centre of the pore and at the pore wall to calculate the average diffusion constant. We apply this model to the diffusion of fullerenes in a toluene filled slit nanopore and calculate the mean diffusion coefficient as a function of the pore size. We show that the accuracy of our model is affected by the partial slip of the toluene on the pore wall.

While deep neural networks (DNNs) achieve impressive performance on environment perception tasks, their sensitivity to adversarial perturbations limits their use in practical applications. In this paper, we (i) propose a novel adversarial perturbation detection scheme based on multi-task perception of complex vision tasks (i.e., depth estimation and semantic segmentation). Specifically, adversarial perturbations are detected by inconsistencies between extracted edges of the input image, the depth output, and the segmentation output. To further improve this technique, we (ii) develop a novel edge consistency loss between all three modalities, thereby improving their initial consistency which in turn supports our detection scheme. We verify our detection scheme's effectiveness by employing various known attacks and image noises. In addition, we (iii) develop a multi-task adversarial attack, aiming at fooling both tasks as well as our detection scheme. Experimental evaluation on the Cityscapes and KITTI datasets shows that under an assumption of a 5% false positive rate up to 100% of images are correctly detected as adversarially perturbed, depending on the strength of the perturbation. Code is available at https://github.com/ifnspaml/AdvAttackDet. A short video at https://youtu.be/KKa6gOyWmH4 provides qualitative results., Comment: Accepted at IROS 2022

The use of deep neural networks (DNNs) in safety-critical applications like mobile health and autonomous driving is challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability to problems with malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from safety concerns. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. This work provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our paper addresses both machine learning experts and safety engineers: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern ML methods. We moreover hope that our contribution fuels discussions on desiderata for ML systems and strategies on how to propel existing approaches accordingly., Comment: 94 pages

In this work we address the task of observing the performance of a semantic segmentation deep neural network (DNN) during online operation, i.e., during inference, which is of high importance in safety-critical applications such as autonomous driving. Here, many high-level decisions rely on such DNNs, which are usually evaluated offline, while their performance in online operation remains unknown. To solve this problem, we propose an improved online performance prediction scheme, building on a recently proposed concept of predicting the primary semantic segmentation task's performance. This can be achieved by evaluating the auxiliary task of monocular depth estimation with a measurement supplied by a LiDAR sensor and a subsequent regression to the semantic segmentation performance. In particular, we propose (i) sequential training methods for both tasks in a multi-task training setup, (ii) to share the encoder as well as parts of the decoder between both task's networks for improved efficiency, and (iii) a temporal statistics aggregation method, which significantly reduces the performance prediction error at the cost of a small algorithmic latency. Evaluation on the KITTI dataset shows that all three aspects improve the performance prediction compared to previous approaches., Comment: Accepted to CVPR SAIAD Workshop

Enabling autonomous driving (AD) can be considered one of the biggest challenges in today's technology. AD is a complex task accomplished by several functionalities, with environment perception being one of its core functions. Environment perception is usually performed by combining the semantic information captured by several sensors, i.e., lidar or camera. The semantic information from the respective sensor can be extracted by using convolutional neural networks (CNNs) for dense prediction. In the past, CNNs constantly showed state-of-the-art performance on several vision-related tasks, such as semantic segmentation of traffic scenes using nothing but the red-green-blue (RGB) images provided by a camera. Although CNNs obtain state-of-the-art performance on clean images, almost imperceptible changes to the input, referred to as adversarial perturbations, may lead to fatal deception. The goal of this article is to illuminate the vulnerability aspects of CNNs used for semantic segmentation with respect to adversarial attacks, and share insights into some of the existing known adversarial defense strategies. We aim to clarify the advantages and disadvantages associated with applying CNNs for environment perception in AD to serve as a motivation for future research in this field., Comment: IEEE Signal Processing Magazine (Volume: 38, Issue: 1, Jan. 2021), pp. 42 - 52

Despite recent advancements, deep neural networks are not robust against adversarial perturbations. Many of the proposed adversarial defense approaches use computationally expensive training mechanisms that do not scale to complex real-world tasks such as semantic segmentation, and offer only marginal improvements. In addition, fundamental questions on the nature of adversarial perturbations and their relation to the network architecture are largely understudied. In this work, we study the adversarial problem from a frequency domain perspective. More specifically, we analyze discrete Fourier transform (DFT) spectra of several adversarial images and report two major findings: First, there exists a strong connection between a model architecture and the nature of adversarial perturbations that can be observed and addressed in the frequency domain. Second, the observed frequency patterns are largely image- and attack-type independent, which is important for the practical impact of any defense making use of such patterns. Motivated by these findings, we additionally propose an adversarial defense method based on the well-known Wiener filters that captures and suppresses adversarial frequencies in a data-driven manner. Our proposed method not only generalizes across unseen attacks but also beats five existing state-of-the-art methods across two models in a variety of attack settings., Comment: Accepted by The International Joint Conference on Neural Network (IJCNN) 2021

Deep neural networks tend to be vulnerable to adversarial perturbations, which by adding to a natural image can fool a respective model with high confidence. Recently, the existence of image-agnostic perturbations, also known as universal adversarial perturbations (UAPs), were discovered. However, existing UAPs still lack a sufficiently high fooling rate, when being applied to an unknown target model. In this paper, we propose a novel deep learning technique for generating more transferable UAPs. We utilize a perturbation generator and some given pretrained networks so-called source models to generate UAPs using the ImageNet dataset. Due to the similar feature representation of various model architectures in the first layer, we propose a loss formulation that focuses on the adversarial energy only in the respective first layer of the source models. This supports the transferability of our generated UAPs to any other target model. We further empirically analyze our generated UAPs and demonstrate that these perturbations generalize very well towards different target models. Surpassing the current state of the art in both, fooling rate and model-transferability, we can show the superiority of our proposed approach. Using our generated non-targeted UAPs, we obtain an average fooling rate of 93.36% on the source models (state of the art: 82.16%). Generating our UAPs on the deep ResNet-152, we obtain about a 12% absolute fooling rate advantage vs. cutting-edge methods on VGG-16 and VGG-19 target models.

While neural networks trained for semantic segmentation are essential for perception in autonomous driving, most current algorithms assume a fixed number of classes, presenting a major limitation when developing new autonomous driving systems with the need of additional classes. In this paper we present a technique implementing class-incremental learning for semantic segmentation without using the labeled data the model was initially trained on. Previous approaches still either rely on labels for both old and new classes, or fail to properly distinguish between them. We show how to overcome these problems with a novel class-incremental learning technique, which nonetheless requires labels only for the new classes. Specifically, (i) we introduce a new loss function that neither relies on old data nor on old labels, (ii) we show how new classes can be integrated in a modular fashion into pretrained semantic segmentation models, and finally (iii) we re-implement previous approaches in a unified setting to compare them to ours. We evaluate our method on the Cityscapes dataset, where we exceed the mIoU performance of all baselines by 3.5% absolute reaching a result, which is only 2.2% absolute below the upper performance limit of single-stage training, relying on all data and labels simultaneously., Comment: ITSC 2020 Conference Paper

While current approaches for neural network training often aim at improving performance, less focus is put on training methods aiming at robustness towards varying noise conditions or directed attacks by adversarial examples. In this paper, we propose to improve robustness by a multi-task training, which extends supervised semantic segmentation by a self-supervised monocular depth estimation on unlabeled videos. This additional task is only performed during training to improve the semantic segmentation model's robustness at test time under several input perturbations. Moreover, we even find that our joint training approach also improves the performance of the model on the original (supervised) semantic segmentation task. Our evaluation exhibits a particular novelty in that it allows to mutually compare the effect of input noises and adversarial attacks on the robustness of the semantic segmentation. We show the effectiveness of our method on the Cityscapes dataset, where our multi-task training approach consistently outperforms the single-task semantic segmentation baseline in terms of both robustness vs. noise and in terms of adversarial attacks, without the need for depth labels in training., Comment: CVPR 2020 Workshop on Safe Artificial Intelligence for Automated Driving

Hypothesis A prototypical modelling approach is required for a full characterisation of the static and equilibrium dynamical properties of confined ionic liquids (ILs), in order to gain predictive power of properties that are difficult to extract from experiments. Such a protocol needs to be constructed by benchmarking molecular dynamics simulations against available experiments. Simulations We perform an in-depth study of [C2Mim][NTf2] in bulk, at the vacuum and at hydroxylated alumina surface. Using the charge methods CHelpG, RESP-HF and RESP-B3LYP with charge scaling factors 1.0, 0.9 and 0.85, we search for an optimum non-polarizable force field by benchmarking against self-diffusion coefficients, surface tension, X-ray reflectivity data, and structural data. Findings Benchmarking, which relies on establishing the significance of an appropriate size of the model systems and the length of the simulations, yields RESP-HF/0.9 as the best suited force field for this IL overall. A complete and accurate characterisation of the spatially-dependent internal configurational space and orientation of IL molecules relative to the solid and vacuum interfaces is obtained. Furthermore, the density and mobility of IL ions in the plane parallel and normal to the interfaces is evaluated and the correlation between the stratification and dynamics in the interfacial layers is detectable deep into the films., Comment: 14 pages, 9 figures in main text and 14 figures in Supporting Information

The progress in autonomous driving is also due to the increased availability of vast amounts of training data for the underlying machine learning approaches. Machine learning systems are generally known to lack robustness, e.g., if the training data did rarely or not at all cover critical situations. The challenging task of corner case detection in video, which is also somehow related to unusual event or anomaly detection, aims at detecting these unusual situations, which could become critical, and to communicate this to the autonomous driving system (online use case). Such a system, however, could be also used in offline mode to screen vast amounts of data and select only the relevant situations for storing and (re)training machine learning algorithms. So far, the approaches for corner case detection have been limited to videos recorded from a fixed camera, mostly for security surveillance. In this paper, we provide a formal definition of a corner case and propose a system framework for both the online and the offline use case that can handle video signals from front cameras of a naturally moving vehicle and can output a corner case score.

The high amount of sensors required for autonomous driving poses enormous challenges on the capacity of automotive bus systems. There is a need to understand tradeoffs between bitrate and perception performance. In this paper, we compare the image compression standards JPEG, JPEG2000, and WebP to a modern encoder/decoder image compression approach based on generative adversarial networks (GANs). We evaluate both the pure compression performance using typical metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and others, but also the performance of a subsequent perception function, namely a semantic segmentation (characterized by the mean intersection over union (mIoU) measure). Not surprisingly, for all investigated compression methods, a higher bitrate means better results in all investigated quality metrics. Interestingly, however, we show that the semantic segmentation mIoU of the GAN autoencoder in the highly relevant low-bitrate regime (at 0.0625 bit/pixel) is better by 3.9% absolute than JPEG2000, although the latter still is considerably better in terms of PSNR (5.91 dB difference). This effect can greatly be enlarged by training the semantic segmentation model with images originating from the decoder, so that the mIoU using the segmentation model trained by GAN reconstructions exceeds the use of the model trained with original images by almost 20% absolute. We conclude that distributed perception in future autonomous driving will most probably not provide a solution to the automotive bus capacity bottleneck by using standard compression schemes such as JPEG2000, but requires modern coding approaches, with the GAN encoder/decoder method being a promising candidate.

Dynamic structuring of water is a key player in a large class of processes underlying biochemical and technological developments today, the latter often involving electric fields. However, the anisotropic coupling between the water structure and the field has not been understood on a molecular level so far. Here we perform extensive molecular dynamics simulations to explore the influence of an externally imposed electric field on liquid water under ambient conditions. Using self-developed analysis tools and rigorous statistical analysis, we unambiguously show that water hydration shells break into subcompartments, which were hitherto not observed due to radial averaging. The shape of subcompartments is sensitive to the field magnitude, and affects excitations of the hydrogen bond network including the femtosecond stretching and the sub-picosecond restructuring of hydrogen bonds. Furthermore, by analysing the reorientational dynamics of water molecules, we ascertain the existence of cooperative excitations of small water clusters. Enabled by the interplay between hydrogen bonding, and the coupling of water dipoles to the field, these coordinated motions, occurring on the picosecond time scale, are associated with fluctuations between torque-free states of water dipoles. We show that unlike the coupling between translation and reorientation of water molecules, which takes place on even longer time scales, these coordinated motions are the key for understanding the emergent anisotropy of diffusion and viscosity of water. Particular effort is invested to provide an analysis that allows for future experimental validation.