Search

In this paper, we present a novel visual servoing (VS) approach based on latent Denoising Diffusion Probabilistic Models (DDPMs). Opposite to classical VS methods, the proposed approach allows reaching the desired target view, even when the target is initially not visible. This is possible thanks to the learning of a latent representation that the DDPM uses for planning and a dataset of trajectories encompassing target-invisible initial views. The latent representation is learned using a Cross-Modal Variational Autoencoder, and used to estimate the return for conditioning the trajectory generation of the DDPM. Given the current image, the DDPM generates trajectories in the latent space driving the robotic platform to the desired visual target. The approach is applicable to any velocity-based controlled platform. We test our method with simulated and real-world experiments using generic multi-rotor Uncrewed Aerial Vehicles (UAVs). A video of our experiments can be found at https://youtu.be/yu-aTxqceOA.

We introduce a general class of autoregressive models for studying the dynamic of multivariate binary time series with stationary exogenous covariates. Using a high-level set of assumptions, we show that existence of a stationary path for such models is almost automatic and does not require parameter restrictions when the noise term is not compactly supported. We then study in details statistical inference in a dynamic version of a multivariate probit type model, as a particular case of our general construction. To avoid a complex likelihood optimization, we combine pseudo-likelihood and pairwise likelihood methods for which asymptotic results are obtained for a single path analysis and also for panel data, using ergodic theorems for multi-indexed partial sums. The latter scenario is particularly important for analyzing absence-presence of species in Ecology, a field where data are often collected from surveys at various locations. Our results also give a theoretical background for such models which are often used by the practitioners but without a probabilistic framework.

Knowledge of the nucleosynthetic isotope composition of the outermost protoplanetary disk is critical to understand the formation and early dynamical evolution of the Solar System. We report the discovery of outer disk material preserved in a pristine meteorite based on its chemical composition, organic-rich petrology, and 15N-rich, deuterium-rich, and 16O-poor isotope signatures. We infer that this outer disk material originated in the comet-forming region. The nucleosynthetic Fe, Mg, Si and Cr compositions of this material reveal that, contrary to current belief, the isotope signature of the comet-forming region is ubiquitous amongst outer Solar System bodies, possibly reflecting an important planetary building block in the outer Solar System. This nucleosynthetic component represents fresh material added to the outer disk by late accretion streamers connected to the ambient molecular cloud. Our results show that most Solar System carbonaceous asteroids accreted material from the comet-forming region, a signature lacking in the terrestrial planet region., Comment: Accepted manuscript, pre-print version

Vision-Language Models seamlessly discriminate among arbitrary semantic categories, yet they still suffer from poor generalization when presented with challenging examples. For this reason, Episodic Test-Time Adaptation (TTA) strategies have recently emerged as powerful techniques to adapt VLMs in the presence of a single unlabeled image. The recent literature on TTA is dominated by the paradigm of prompt tuning by Marginal Entropy Minimization, which, relying on online backpropagation, inevitably slows down inference while increasing memory. In this work, we theoretically investigate the properties of this approach and unveil that a surprisingly strong TTA method lies dormant and hidden within it. We term this approach ZERO (TTA with "zero" temperature), whose design is both incredibly effective and frustratingly simple: augment N times, predict, retain the most confident predictions, and marginalize after setting the Softmax temperature to zero. Remarkably, ZERO requires a single batched forward pass through the vision encoder only and no backward passes. We thoroughly evaluate our approach following the experimental protocol established in the literature and show that ZERO largely surpasses or compares favorably w.r.t. the state-of-the-art while being almost 10x faster and 13x more memory-friendly than standard Test-Time Prompt Tuning. Thanks to its simplicity and comparatively negligible computation, ZERO can serve as a strong baseline for future work in this field. The code is available at https://github.com/FarinaMatteo/zero., Comment: Preprint. Work in progress

The sixth generation (6G) mobile communication networks are expected to intelligently integrate into various aspects of modern digital society, including smart cities, homes, health-care, transportation, and factories. While offering a multitude of services, it is likely that societies become increasingly reliant on 6G infrastructure. Any disruption to these digital services, whether due to human or technical failures, natural disasters, or terrorism, would significantly impact citizens' daily lives. Hence, 6G networks need not only to provide high-performance services but also to be resilient in maintaining essential services in the face of potentially unknown challenges. This paper provides a general review of the state of the art on resilient systems, definitions, concepts, and approaches. Moreover, it introduces a comprehensive concept, i.e., resilience-by-design (RBD), in three different levels for designing resilient 6G communication networks, summarizing our initial studies within the German Open6GHub project. First, we outline the general RBD enabling principles and discuss their related sub-categories. Next, adopting an interdisciplinary approach, we propose to embed these principles across all 6G layers/perspectives including electronics, physical channel, network components and functions, networks, services, and cross-layer and cross-infrastructure considerations and discuss their challenges. We further elaborate the RBD principles and their realizations along with several 6G use-cases. The paper is concluded by presenting a comprehensive list of open problems for future research on 6G resilience.

Berry picking has long-standing traditions in Finland, yet it is challenging and can potentially be dangerous. The integration of drones equipped with advanced imaging techniques represents a transformative leap forward, optimising harvests and promising sustainable practices. We propose WildBe, the first image dataset of wild berries captured in peatlands and under the canopy of Finnish forests using drones. Unlike previous and related datasets, WildBe includes new varieties of berries, such as bilberries, cloudberries, lingonberries, and crowberries, captured under severe light variations and in cluttered environments. WildBe features 3,516 images, including a total of 18,468 annotated bounding boxes. We carry out a comprehensive analysis of WildBe using six popular object detectors, assessing their effectiveness in berry detection across different forest regions and camera types. We will release WildBe publicly.

On 24 September 2023, the NASA OSIRIS-REx mission dropped a capsule to Earth containing approximately 120 g of pristine carbonaceous regolith from Bennu. We describe the delivery and initial allocation of this asteroid sample and introduce its bulk physical, chemical, and mineralogical properties from early analyses. The regolith is very dark overall, with higher-reflectance inclusions and particles interspersed. Particle sizes range from sub-micron dust to a stone about 3.5 cm long. Millimeter-scale and larger stones typically have hummocky or angular morphologies. A subset of the stones appears mottled by brighter material that occurs as veins and crusts. Hummocky stones have the lowest densities and mottled stones have the highest. Remote sensing of the surface of Bennu detected hydrated phyllosilicates, magnetite, organic compounds, carbonates, and scarce anhydrous silicates, all of which the sample confirms. We also find sulfides, presolar grains, and, less expectedly, Na-rich phosphates, as well as other trace phases. The sample composition and mineralogy indicate substantial aqueous alteration and resemble those of Ryugu and the most chemically primitive, low-petrologic-type carbonaceous chondrites. Nevertheless, we find distinct hydrogen, nitrogen, and oxygen isotopic compositions, and some of the material we analyzed is enriched in fluid-mobile elements. Our findings underscore the value of sample return, especially for low-density material that may not readily survive atmospheric entry, and lay the groundwork for more comprehensive analyses., Comment: 73 pages, 22 figures

Accurate trajectory forecasting is crucial for the performance of various systems, such as advanced driver-assistance systems and self-driving vehicles. These forecasts allow to anticipate events leading to collisions and, therefore, to mitigate them. Deep Neural Networks have excelled in motion forecasting, but issues like overconfidence and uncertainty quantification persist. Deep Ensembles address these concerns, yet applying them to multimodal distributions remains challenging. In this paper, we propose a novel approach named Hierarchical Light Transformer Ensembles (HLT-Ens), aimed at efficiently training an ensemble of Transformer architectures using a novel hierarchical loss function. HLT-Ens leverages grouped fully connected layers, inspired by grouped convolution techniques, to capture multimodal distributions, effectively. Through extensive experimentation, we demonstrate that HLT-Ens achieves state-of-the-art performance levels, offering a promising avenue for improving trajectory forecasting techniques.

In this work, we present a model-based optimal boundary control design for an aerial robotic system composed of a quadrotor carrying a flexible cable. The whole system is modeled by partial differential equations (PDEs) combined with boundary conditions described by ordinary differential equations (ODEs). The proper orthogonal decomposition (POD) method is adopted to project the original infinite-dimensional system on a subspace spanned by orthogonal basis functions. Based on the reduced order model, nonlinear model predictive control (NMPC) is implemented online to realize shape trajectory tracking of the flexible cable in an optimal predictive fashion. The proposed reduced modeling and optimal control paradigms are numerically verified against an accurate high-dimensional FDM-based model in different scenarios and the controller's superior performance is shown compared to an optimally tuned PID controller.

It is shown that higher degree exact differential forms on compact Riemannian $n$-manifolds possess continuous primitives whose uniform norm is controlled by their $L^n$ norm. A contact sub-Riemannian analogue is proven, with differential forms replaced with Rumin differential forms.

Label smoothing (LS) is a popular regularisation method for training deep neural network classifiers due to its effectiveness in improving test accuracy and its simplicity in implementation. "Hard" one-hot labels are "smoothed" by uniformly distributing probability mass to other classes, reducing overfitting. In this work, we reveal that LS negatively affects selective classification (SC) - where the aim is to reject misclassifications using a model's predictive uncertainty. We first demonstrate empirically across a range of tasks and architectures that LS leads to a consistent degradation in SC. We then explain this by analysing logit-level gradients, showing that LS exacerbates overconfidence and underconfidence by regularising the max logit more when the probability of error is low, and less when the probability of error is high. This elucidates previously reported experimental results where strong classifiers underperform in SC. We then demonstrate the empirical effectiveness of logit normalisation for recovering lost SC performance caused by LS. Furthermore, based on our gradient analysis, we explain why such normalisation is effective. We will release our code shortly.

In this letter, we address the problem of re-targeting a commercial under-actuated robotic system to a higher dimensional output task. Commercial platforms are equipped with an on-board low-level internal controller that provides the user some virtual references inputs and cannot be replaced. Such controller may control in practice only a subset of the total degrees-of-freedom of the system. Our primary objective is to augment these systems by introducing supplementary inputs, attaining increased output capability. Integrating such additional inputs into the control framework introduces a layer of complexity, raising questions about the compatibility and cohesiveness of the overall system. We derive necessary and sufficient conditions under which it is possible to extend the controlled outputs by adding extra inputs when one is forced to keep the internal controller unmodified. We underscore the efficacy and universality of our proposed methodology through the presentation of two relevant examples.

This work answers positively the question whether non-stop flights are possible for maintaining constant the pose of cable-suspended objects. Such a counterintuitive answer paves the way for a paradigm shift where energetically efficient fixed-wing flying carriers can replace the inefficient multirotor carriers that have been used so far in precise cooperative cable-suspended aerial manipulation. First, we show that one or two flying carriers alone cannot perform non-stop flights while maintaining a constant pose of the suspended object. Instead, we prove that three flying carriers can achieve this task provided that the orientation of the load at the equilibrium is such that the components of the cable forces that balance the external force (typically gravity) do not belong to the plane of the cable anchoring points on the load. Numerical tests are presented in support of the analytical results.

Scaffolds, also called sidewalk sheds, are intended to be temporary structures to protect pedestrians from construction and repair hazards. However, some sidewalk sheds are left up for years. Long-term scaffolding becomes eyesores, creates accessibility issues on sidewalks, and gives cover to illicit activity. Today, there are over 8,000 active permits for scaffolds in NYC; the more problematic scaffolds are likely expired or unpermitted. This research uses computer vision on street-level imagery to develop a longitudinal map of scaffolding throughout the city. Using a dataset of 29,156,833 dashcam images taken between August 2023 and January 2024, we develop an algorithm to track the presence of scaffolding over time. We also design and implement methods to match detected scaffolds to reported locations of active scaffolding permits, enabling the identification of sidewalk sheds without corresponding permits. We identify 850,766 images of scaffolding, tagging 5,156 active sidewalk sheds and estimating 529 unpermitted sheds. We discuss the implications of an in-the-wild scaffolding classifier for urban tech, innovations to governmental inspection processes, and out-of-distribution evaluations outside of New York City.

We provide a complete description of the set of all solutions to an autoregressive law of motion in a finite-dimensional complex vector space. Every solution is shown to be the sum of three parts, each corresponding to a directed flow of time. One part flows forward from the arbitrarily distant past; one flows backward from the arbitrarily distant future; and one flows outward from time zero. The three parts are obtained by applying three complementary spectral projections to the solution, these corresponding to a separation of the eigenvalues of the autoregressive operator according to whether they are inside, outside or on the unit circle. We provide a finite-dimensional parametrization of the set of all solutions.

In this work, we prove a two-scale homogenization result for a set of diffusion-coagulation Smoluchowski-type equations with transmission boundary conditions. This system is meant to describe the aggregation and diffusion of pathological tau proteins in the cerebral tissue, a process associated with the onset and evolution of a large variety of tauopathies (such as Alzheimer's disease). We prove the existence, positivity and boundedness of solutions to the model equations derived at the microscale (that is the scale of single neurons). Then, we study the convergence of the homogenization process to the solution of a macro-model asymptotically consistent with the microscopic one.

Large-scale infrastructures are prone to deterioration due to age, environmental influences, and heavy usage. Ensuring their safety through regular inspections and maintenance is crucial to prevent incidents that can significantly affect public safety and the environment. This is especially pertinent in the context of electrical power networks, which, while essential for energy provision, can also be sources of forest fires. Intelligent drones have the potential to revolutionize inspection and maintenance, eliminating the risks for human operators, increasing productivity, reducing inspection time, and improving data collection quality. However, most of the current methods and technologies in aerial robotics have been trialed primarily in indoor testbeds or outdoor settings under strictly controlled conditions, always within the line of sight of human operators. Additionally, these methods and technologies have typically been evaluated in isolation, lacking comprehensive integration. This paper introduces the first autonomous system that combines various innovative aerial robots. This system is designed for extended-range inspections beyond the visual line of sight, features aerial manipulators for maintenance tasks, and includes support mechanisms for human operators working at elevated heights. The paper further discusses the successful validation of this system on numerous electrical power lines, with aerial robots executing flights over 10 kilometers away from their ground control stations.

Deep Neural Networks (DNNs) are powerful tools for various computer vision tasks, yet they often struggle with reliable uncertainty quantification - a critical requirement for real-world applications. Bayesian Neural Networks (BNN) are equipped for uncertainty estimation but cannot scale to large DNNs that are highly unstable to train. To address this challenge, we introduce the Adaptable Bayesian Neural Network (ABNN), a simple and scalable strategy to seamlessly transform DNNs into BNNs in a post-hoc manner with minimal computational and training overheads. ABNN preserves the main predictive properties of DNNs while enhancing their uncertainty quantification abilities through simple BNN adaptation layers (attached to normalization layers) and a few fine-tuning steps on pre-trained models. We conduct extensive experiments across multiple datasets for image classification and semantic segmentation tasks, and our results demonstrate that ABNN achieves state-of-the-art performance without the computational budget typically associated with ensemble methods.

In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLIP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron is linked to a specific word. Observing that this latent space can be modeled with simple distributions, we use a Mixture of Gaussians (MoG) formalism to enhance the interpretability of this latent space. Then, we introduce CLIP-QDA, a classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both local and global explanations. These explanations come from the inner design of our architecture, our work is part of a new family of greybox models, combining performances of opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP-QDA achieves similar accuracy with state-of-the-art methods CBMs. Our explanations compete with existing XAI methods while being faster to compute.

Sensing plays a crucial role in autonomous and assisted vehicular driving, as well as in the operation of autonomous drones. The traditional segregation of communication and onboard sensing systems in mobility applications is due to be merged using Joint Communication and Sensing (JCAS) in the development of the 6G mobile radio standard. The integration of JCAS functions into the future road traffic landscape introduces novel challenges for the design of the 6G system architecture. Special emphasis will be placed on facilitating direct communication between road users and aerial drones. In various mobility scenarios, diverse levels of integration will be explored, ranging from leveraging communication capabilities to coordinate different radars to achieving deep integration through a unified waveform. In this paper, we have identified use cases and derive five higher-level Tech Cases (TCs). Technical and functional requirements for the 6G system architecture for a device-oriented JCAS approach will be extracted from the TCs and used to conceptualize the architectural views., Comment: 6 pages, 3 figures, 4th IEEE Symposium on Joint Communication and Sensing

AcubeSAT is an open-source CubeSat mission aiming to explore the effects of microgravity and radiation on eukaryotic cells using a compact microfluidic LoC platform. It is developed by SpaceDot, a volunteer, interdisciplinary student team at the Aristotle University of Thessaloniki and supported by the "Fly Your Satellite! 3" program of the ESA Education Office. The scientific data of the mission is comprised of microscope images captured through the on-board integrated camera setup. As the total size of the payload data is expected to be close to 2GB over 12 months, a fast and efficient downlink fulfilling the restrictive power, cost and complexity budgets is required. Currently, there is no open-source communications system design which fully supports these specific constraints, so we opted to develop our own solutions. The antenna system underwent multiple iterations as the design matured, a process highly aided by the feedback received from the ESA experts. The final communications system configuration consists of an S-band microstrip antenna operating at 2.4GHz and a UHF deployable antenna, for the payload data and TM&TC respectively, both in-house designed. In this paper, we will present AcubeSAT's antenna system iterations that span over 3 years, as well as the rationale and analysis results behind each. The development decisions will be highlighted throughout the paper in an effort to aid in the future development of such a low-cost CubeSat mission communications system., Comment: 74th International Astronautical Congress

The distribution of the weights of modern deep neural networks (DNNs) - crucial for uncertainty quantification and robustness - is an eminently complex object due to its extremely high dimensionality. This paper proposes one of the first large-scale explorations of the posterior distribution of deep Bayesian Neural Networks (BNNs), expanding its study to real-world vision tasks and architectures. Specifically, we investigate the optimal approach for approximating the posterior, analyze the connection between posterior quality and uncertainty quantification, delve into the impact of modes on the posterior, and explore methods for visualizing the posterior. Moreover, we uncover weight-space symmetries as a critical aspect for understanding the posterior. To this extent, we develop an in-depth assessment of the impact of both permutation and scaling symmetries that tend to obfuscate the Bayesian posterior. While the first type of transformation is known for duplicating modes, we explore the relationship between the latter and L2 regularization, challenging previous misconceptions. Finally, to help the community improve our understanding of the Bayesian posterior, we will shortly release the first large-scale checkpoint dataset, including thousands of real-world models and our codes.

Detecting out-of-distribution (OOD) data is a critical challenge in machine learning due to model overconfidence, often without awareness of their epistemological limits. We hypothesize that ``neural collapse'', a phenomenon affecting in-distribution data for models trained beyond loss convergence, also influences OOD data. To benefit from this interplay, we introduce NECO, a novel post-hoc method for OOD detection, which leverages the geometric properties of ``neural collapse'' and of principal component spaces to identify OOD data. Our extensive experiments demonstrate that NECO achieves state-of-the-art results on both small and large-scale OOD detection tasks while exhibiting strong generalization capabilities across different network architectures. Furthermore, we provide a theoretical explanation for the effectiveness of our method in OOD detection. Code is available at https://gitlab.com/drti/neco, Comment: Accepted to ICLR2024

This paper presents a new approach for training two-stage object detection ensemble models, more specifically, Faster R-CNN models to estimate uncertainty. We propose training one Region Proposal Network(RPN) and multiple Fast R-CNN prediction heads is all you need to build a robust deep ensemble network for estimating uncertainty in object detection. We present this approach and provide experiments to show that this approach is much faster than the naive method of fully training all $n$ models in an ensemble. We also estimate the uncertainty by measuring this ensemble model's Expected Calibration Error (ECE). We then further compare the performance of this model with that of Gaussian YOLOv3, a variant of YOLOv3 that models uncertainty using predicted bounding box coordinates. The source code is released at \url{https://github.com/Akola-Mbey-Denis/EfficientEnsemble}

The capacity of a modern deep learning system to determine if a sample falls within its realm of knowledge is fundamental and important. In this paper, we offer insights and analyses of recent state-of-the-art out-of-distribution (OOD) detection methods - extremely simple activation shaping (ASH). We demonstrate that activation pruning has a detrimental effect on OOD detection, while activation scaling enhances it. Moreover, we propose SCALE, a simple yet effective post-hoc network enhancement method for OOD detection, which attains state-of-the-art OOD detection performance without compromising in-distribution (ID) accuracy. By integrating scaling concepts into the training process to capture a sample's ID characteristics, we propose Intermediate Tensor SHaping (ISH), a lightweight method for training time OOD detection enhancement. We achieve AUROC scores of +1.85\% for near-OOD and +0.74\% for far-OOD datasets on the OpenOOD v1.5 ImageNet-1K benchmark. Our code and models are available at https://github.com/kai422/SCALE.

This paper outlines the winning solutions employed in addressing the MUAD uncertainty quantification challenge held at ICCV 2023. The challenge was centered around semantic segmentation in urban environments, with a particular focus on natural adversarial scenarios. The report presents the results of 19 submitted entries, with numerous techniques drawing inspiration from cutting-edge uncertainty quantification methodologies presented at prominent conferences in the fields of computer vision and machine learning and journals over the past few years. Within this document, the challenge is introduced, shedding light on its purpose and objectives, which primarily revolved around enhancing the robustness of semantic segmentation in urban scenes under varying natural adversarial conditions. The report then delves into the top-performing solutions. Moreover, the document aims to provide a comprehensive overview of the diverse solutions deployed by all participants. By doing so, it seeks to offer readers a deeper insight into the array of strategies that can be leveraged to effectively handle the inherent uncertainties associated with autonomous driving and semantic segmentation, especially within urban environments., Comment: 11 pages, 4 figures, accepted at ICCV 2023 UNCV workshop

Current deep neural networks (DNNs) for autonomous driving computer vision are typically trained on specific datasets that only involve a single type of data and urban scenes. Consequently, these models struggle to handle new objects, noise, nighttime conditions, and diverse scenarios, which is essential for safety-critical applications. Despite ongoing efforts to enhance the resilience of computer vision DNNs, progress has been sluggish, partly due to the absence of benchmarks featuring multiple modalities. We introduce a novel and versatile dataset named InfraParis that supports multiple tasks across three modalities: RGB, depth, and infrared. We assess various state-of-the-art baseline techniques, encompassing models for the tasks of semantic segmentation, object detection, and depth estimation. More visualizations and the download link for InfraParis are available at \href{https://ensta-u2is.github.io/infraParis/}{https://ensta-u2is.github.io/infraParis/}., Comment: 15 pages, 7 figures. Accepted at WACV 2024

This paper examines the robustness of a multi-modal computer vision model, CLIP (Contrastive Language-Image Pretraining), in the context of unsupervised learning. The main objective is twofold: first, to evaluate the robustness of CLIP, and second, to explore strategies for augmenting its robustness. To achieve this, we introduce a novel approach named LP-CLIP. This technique involves the distillation of CLIP features through the incorporation of a linear probing layer positioned atop its encoding structure. This newly added layer is trained utilizing pseudo-labels produced by CLIP, coupled with a self-training strategy. The LP-CLIP technique offers a promising approach to enhance the robustness of CLIP without the need for annotations. By leveraging a simple linear probing layer, we aim to improve the model's ability to withstand various uncertainties and challenges commonly encountered in real-world scenarios. Importantly, our approach does not rely on annotated data, which makes it particularly valuable in situations where labeled data might be scarce or costly to obtain. Our proposed approach increases the robustness of CLIP with SOTA results compared to supervised technique on various datasets.

Uncertainty quantification is critical for deploying deep neural networks (DNNs) in real-world applications. An Auxiliary Uncertainty Estimator (AuxUE) is one of the most effective means to estimate the uncertainty of the main task prediction without modifying the main task model. To be considered robust, an AuxUE must be capable of maintaining its performance and triggering higher uncertainties while encountering Out-of-Distribution (OOD) inputs, i.e., to provide robust aleatoric and epistemic uncertainty. However, for vision regression tasks, current AuxUE designs are mainly adopted for aleatoric uncertainty estimates, and AuxUE robustness has not been explored. In this work, we propose a generalized AuxUE scheme for more robust uncertainty quantification on regression tasks. Concretely, to achieve a more robust aleatoric uncertainty estimation, different distribution assumptions are considered for heteroscedastic noise, and Laplace distribution is finally chosen to approximate the prediction error. For epistemic uncertainty, we propose a novel solution named Discretization-Induced Dirichlet pOsterior (DIDO), which models the Dirichlet posterior on the discretized prediction error. Extensive experiments on age estimation, monocular depth estimation, and super-resolution tasks show that our proposed method can provide robust uncertainty estimates in the face of noisy inputs and that it can be scalable to both image-level and pixel-wise tasks. Code is available at https://github.com/ENSTA-U2IS/DIDO ., Comment: 23 pages with main paper and supplymentary material. Accepted at AAAI 2024

Semantic segmentation methods have advanced significantly. Still, their robustness to real-world perturbations and object types not seen during training remains a challenge, particularly in safety-critical applications. We propose a novel approach to improve the robustness of semantic segmentation techniques by leveraging the synergy between label-to-image generators and image-to-label segmentation models. Specifically, we design Robusta, a novel robust conditional generative adversarial network to generate realistic and plausible perturbed images that can be used to train reliable segmentation models. We conduct in-depth studies of the proposed generative model, assess the performance and robustness of the downstream segmentation network, and demonstrate that our approach can significantly enhance the robustness in the face of real-world perturbations, distribution shifts, and out-of-distribution samples. Our results suggest that this approach could be valuable in safety-critical applications, where the reliability of perception modules such as semantic segmentation is of utmost importance and comes with a limited computational budget in inference. We release our code at https://github.com/ENSTA-U2IS-AI/robusta., Comment: Published as a conference paper at WACV 2024