Search

Pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive zero-shot classification capabilities with free-form prompts and even show some generalization in specialized domains. However, their performance on satellite imagery is limited due to the underrepresentation of such data in their training sets, which predominantly consist of ground-level images. Existing prompting techniques for satellite imagery are often restricted to generic phrases like a satellite image of ..., limiting their effectiveness for zero-shot land-use and land-cover (LULC) mapping. To address these challenges, we introduce SenCLIP, which transfers CLIPs representation to Sentinel-2 imagery by leveraging a large dataset of Sentinel-2 images paired with geotagged ground-level photos from across Europe. We evaluate SenCLIP alongside other SOTA remote sensing VLMs on zero-shot LULC mapping tasks using the EuroSAT and BigEarthNet datasets with both aerial and ground-level prompting styles. Our approach, which aligns ground-level representations with satellite imagery, demonstrates significant improvements in classification accuracy across both prompt styles, opening new possibilities for applying free-form textual descriptions in zero-shot LULC mapping., Comment: Accepted at WACV'25

Cross-modal knowledge distillation (CMKD) refers to the scenario in which a learning framework must handle training and test data that exhibit a modality mismatch, more precisely, training and test data do not cover the same set of data modalities. Traditional approaches for CMKD are based on a teacher/student paradigm where a teacher is trained on multi-modal data with the aim to successively distill knowledge from a multi-modal teacher to a single-modal student. Despite the widespread adoption of such paradigm, recent research has highlighted its inherent limitations in the context of cross-modal knowledge transfer.Taking a step beyond the teacher/student paradigm, here we introduce a new framework for cross-modal knowledge distillation, named DisCoM-KD (Disentanglement-learning based Cross-Modal Knowledge Distillation), that explicitly models different types of per-modality information with the aim to transfer knowledge from multi-modal data to a single-modal classifier. To this end, DisCoM-KD effectively combines disentanglement representation learning with adversarial domain adaptation to simultaneously extract, foreach modality, domain-invariant, domain-informative and domain-irrelevant features according to a specific downstream task. Unlike the traditional teacher/student paradigm, our framework simultaneously learns all single-modal classifiers, eliminating the need to learn each student model separately as well as the teacher classifier. We evaluated DisCoM-KD on three standard multi-modal benchmarks and compared its behaviourwith recent SOTA knowledge distillation frameworks. The findings clearly demonstrate the effectiveness of DisCoM-KD over competitors considering mismatch scenarios involving both overlapping and non-overlapping modalities. These results offer insights to reconsider the traditional paradigm for distilling information from multi-modal data to single-modal neural networks.

Computer vision methods that explicitly detect object parts and reason on them are a step towards inherently interpretable models. Existing approaches that perform part discovery driven by a fine-grained classification task make very restrictive assumptions on the geometric properties of the discovered parts; they should be small and compact. Although this prior is useful in some cases, in this paper we show that pre-trained transformer-based vision models, such as self-supervised DINOv2 ViT, enable the relaxation of these constraints. In particular, we find that a total variation (TV) prior, which allows for multiple connected components of any size, substantially outperforms previous work. We test our approach on three fine-grained classification benchmarks: CUB, PartImageNet and Oxford Flowers, and compare our results to previously published methods as well as a re-implementation of the state-of-the-art method PDiscoNet with a transformer-based backbone. We consistently obtain substantial improvements across the board, both on part discovery metrics and the downstream classification task, showing that the strong inductive biases in self-supervised ViT models require to rethink the geometric priors that can be used for unsupervised part discovery., Comment: Accepted as a main conference paper at the European Conference of Computer Vision (ECCV) 2024

Semi-supervised domain adaptation methods leverage information from a source labelled domain with the goal of generalizing over a scarcely labelled target domain. While this setting already poses challenges due to potential distribution shifts between domains, an even more complex scenario arises when source and target data differs in modality representation (e.g. they are acquired by sensors with different characteristics). For instance, in remote sensing, images may be collected via various acquisition modes (e.g. optical or radar), different spectral characteristics (e.g. RGB or multi-spectral) and spatial resolutions. Such a setting is denoted as Semi-Supervised Heterogeneous Domain Adaptation (SSHDA) and it exhibits an even more severe distribution shift due to modality heterogeneity across domains.To cope with the challenging SSHDA setting, here we introduce SHeDD (Semi-supervised Heterogeneous Domain Adaptation via Disentanglement) an end-to-end neural framework tailored to learning a target domain classifier by leveraging both labelled and unlabelled data from heterogeneous data sources. SHeDD is designed to effectively disentangle domain-invariant representations, relevant for the downstream task, from domain-specific information, that can hinder the cross-modality transfer. Additionally, SHeDD adopts an augmentation-based consistency regularization mechanism that takes advantages of reliable pseudo-labels on the unlabelled target samples to further boost its generalization ability on the target domain. Empirical evaluations on two remote sensing benchmarks, encompassing heterogeneous data in terms of acquisition modes and spectral/spatial resolutions, demonstrate the quality of SHeDD compared to both baseline and state-of-the-art competing approaches. Our code is publicly available here: https://github.com/tanodino/SSHDA/

Recently, there has been increasing interest in multimodal applications that integrate text with other modalities, such as images, audio and video, to facilitate natural language interactions with multimodal AI systems. While applications involving standard modalities have been extensively explored, there is still a lack of investigation into specific data modalities such as remote sensing (RS) data. Despite the numerous potential applications of RS data, including environmental protection, disaster monitoring and land planning, available solutions are predominantly focused on specific tasks like classification, captioning and retrieval. These solutions often overlook the unique characteristics of RS data, such as its capability to systematically provide information on the same geographical areas over time. This ability enables continuous monitoring of changes in the underlying landscape. To address this gap, we propose a novel foundation model for bi-temporal RS image pairs, in the context of change detection analysis, leveraging Contrastive Learning and the LEVIR-CC dataset for both captioning and text-image retrieval. By jointly training a contrastive encoder and captioning decoder, our model add text-image retrieval capabilities, in the context of bi-temporal change detection, while maintaining captioning performances that are comparable to the state of the art. We release the source code and pretrained weights at: https://github.com/rogerferrod/RSICRC.

Timely up-to-date land use/land cover (LULC) maps play a pivotal role in supporting agricultural territory management, environmental monitoring and facilitating well-informed and sustainable decision-making. Typically, when creating a land cover (LC) map, precise ground truth data is collected through time-consuming and expensive field campaigns. This data is then utilized in conjunction with satellite image time series (SITS) through advanced machine learning algorithms to get the final map. Unfortunately, each time this process is repeated (e.g., annually over a region to estimate agricultural production or potential biodiversity loss), new ground truth data must be collected, leading to the complete disregard of previously gathered reference data despite the substantial financial and time investment they have required. How to make value of historical data, from the same or similar study sites, to enhance the current LULC mapping process constitutes a significant challenge that could enable the financial and human-resource efforts invested in previous data campaigns to be valued again. Aiming to tackle this important challenge, we here propose a deep learning framework based on recent advances in domain adaptation and generalization to combine remote sensing and reference data coming from two different domains (e.g. historical data and fresh ones) to ameliorate the current LC mapping process. Our approach, namely REFeD (data Reuse with Effective Feature Disentanglement for land cover mapping), leverages a disentanglement strategy, based on contrastive learning, where invariant and specific per-domain features are derived to recover the intrinsic information related to the downstream LC mapping task and alleviate possible distribution shifts between domains. Additionally, REFeD is equipped with an effective supervision scheme where feature disentanglement is further enforced via multiple levels of supervision at different granularities. The experimental assessment over two study areas covering extremely diverse and contrasted landscapes, namely Koumbia (located in the West-Africa region, in Burkina Faso) and Centre Val de Loire (located in centre Europe, France), underlines the quality of our framework and the obtained findings demonstrate that out-of-year information coming from the same (or similar) study site, at different periods of time, can constitute a valuable additional source of information to enhance the LC mapping process.

The Black Grouse (Lyrurus tetrix) is an emblematic alpine species with high conservation importance. The population size of these mountain bird tends to decline on the reference sites and shows differences according to changes in local landscape characteristics. Habitat changes are at the centre of the identified pressures impacting part or all of its life cycle, according to experts. Hence, an approach to monitor population dynamics, is trough modelling the favourable habitats of Black Grouse breeding (nesting sites). Then, coupling modelling with multi-source remote sensing data (medium and very high spatial resolution), allowed the implementation of a spatial distribution model of the species. Indeed, the extraction of variables from remote sensing helped to describe the area studied at appropriate spatial and temporal scales: horizontal and vertical structure (heterogeneity), functioning (vegetation indices), phenology (seasonal or inter-annual dynamics) and biodiversity. An annual time series of radiometric indices (NDVI, NDWI, BI {\ldots}) from Sentinel-2 has made it possible to generate Dynamic Habitat Indices (DHIs) to derive phenological indications on the nature and dynamics of natural habitats. In addition, very high resolution images (SPOT6) provided access to the fine structure of natural habitats, i.e. the vertical and horizontal organisation by states identified as elementary (mineral, herbaceous, low and high woody). Indeed, one of the essential limiting factors for brood rearing is the presence of a well-developed herbaceous or ericaceous stratum in the northern Alps and larch forests in the southern region. A deep learning model was used to classify elementary strata. Finally, Biomod2 R platform, using an ensemble approach, was applied to model, the favourable habitat of Black Grouse reproduction. Of all the models, Random Forest and Extreme Boosted Gradient are the best performing, with TSS and ROC scores close to 1. For the SDM, we selected only Random Forest models (ensemble modelling) because of their low susceptibility to overfitting and coherent predictions (after comparing model predictions).In this ensemble model, the most important explanatory variables are altitude, the proportion of heathland, and the DHI (NDVI Max and NDWI Max). Results from the habitat model can be used as an operational tool for monitoring forest landscape shifts and changes. In addition, to delimiting potential areas to protect the species habitat, which constitute a valuable decision-making tool for conservation management of mountain open forest., Comment: in French language

Deep learning algorithms have recently gained significant attention due to their impressive performance. However, their high complexity and un-interpretable mode of operation hinders their confident deployment in real-world safety-critical tasks. This work targets ante hoc interpretability, and specifically Concept Bottleneck Models (CBMs). Our goal is to design a framework that admits a highly interpretable decision making process with respect to human understandable concepts, on two levels of granularity. To this end, we propose a novel two-level concept discovery formulation leveraging: (i) recent advances in vision-language models, and (ii) an innovative formulation for coarse-to-fine concept selection via data-driven and sparsity-inducing Bayesian arguments. Within this framework, concept information does not solely rely on the similarity between the whole image and general unstructured concepts; instead, we introduce the notion of concept hierarchy to uncover and exploit more granular concept information residing in patch-specific regions of the image scene. As we experimentally show, the proposed construction not only outperforms recent CBM approaches, but also yields a principled framework towards interpetability.

Fine-grained classification often requires recognizing specific object parts, such as beak shape and wing patterns for birds. Encouraging a fine-grained classification model to first detect such parts and then using them to infer the class could help us gauge whether the model is indeed looking at the right details better than with interpretability methods that provide a single attribution map. We propose PDiscoNet to discover object parts by using only image-level class labels along with priors encouraging the parts to be: discriminative, compact, distinct from each other, equivariant to rigid transforms, and active in at least some of the images. In addition to using the appropriate losses to encode these priors, we propose to use part-dropout, where full part feature vectors are dropped at once to prevent a single part from dominating in the classification, and part feature vector modulation, which makes the information coming from each part distinct from the perspective of the classifier. Our results on CUB, CelebA, and PartImageNet show that the proposed method provides substantially better part discovery performance than previous methods while not requiring any additional hyper-parameter tuning and without penalizing the classification performance. The code is available at https://github.com/robertdvdk/part_detection., Comment: 9 pages, 8 figures, ICCV

Models for fine-grained image classification tasks, where the difference between some classes can be extremely subtle and the number of samples per class tends to be low, are particularly prone to picking up background-related biases and demand robust methods to handle potential examples with out-of-distribution (OOD) backgrounds. To gain deeper insights into this critical problem, our research investigates the impact of background-induced bias on fine-grained image classification, evaluating standard backbone models such as Convolutional Neural Network (CNN) and Vision Transformers (ViT). We explore two masking strategies to mitigate background-induced bias: Early masking, which removes background information at the (input) image level, and late masking, which selectively masks high-level spatial features corresponding to the background. Extensive experiments assess the behavior of CNN and ViT models under different masking strategies, with a focus on their generalization to OOD backgrounds. The obtained findings demonstrate that both proposed strategies enhance OOD performance compared to the baseline models, with early masking consistently exhibiting the best OOD performance. Notably, a ViT variant employing GAP-Pooled Patch token-based classification combined with early masking achieves the highest OOD robustness., Comment: Accepted at the 2023 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) on Out Of Distribution Generalization in Computer Vision (OOD-CV)

The recent mass adoption of DNNs, even in safety-critical scenarios, has shifted the focus of the research community towards the creation of inherently intrepretable models. Concept Bottleneck Models (CBMs) constitute a popular approach where hidden layers are tied to human understandable concepts allowing for investigation and correction of the network's decisions. However, CBMs usually suffer from: (i) performance degradation and (ii) lower interpretability than intended due to the sheer amount of concepts contributing to each decision. In this work, we propose a simple yet highly intuitive interpretable framework based on Contrastive Language Image models and a single sparse linear layer. In stark contrast to related approaches, the sparsity in our framework is achieved via principled Bayesian arguments by inferring concept presence via a data-driven Bernoulli distribution. As we experimentally show, our framework not only outperforms recent CBM approaches accuracy-wise, but it also yields high per example concept sparsity, facilitating the individual investigation of the emerging concepts., Comment: Accepted @ ICCVW CLVL 2023

The Transiting Exoplanet Survey Satellite (TESS) mission is providing the scientific community with millions of light curves of stars spread across the whole sky. Since 2018 the telescope has detected thousands of planet candidates that need to be meticulously scrutinized before being considered amenable targets for follow-up programs. We present the second catalog of the Plant Patrol citizen science project containing 999 uniformly-vetted exoplanet candidates within the TESS ExoFOP archive. The catalog was produced by fully exploiting the power of the Citizen Science Planet Patrol project. We vetted TESS Objects of Interest (TOIs) based on the results of Discovery And Vetting of Exoplanets DAVE pipeline. We also implemented the Automatic Disposition Generator, a custom procedure aimed at generating the final classification for each TOI that was vetted by at least three vetters. The majority of the candidates in our catalog, $752$ TOIs, passed the vetting process and were labelled as planet candidates. We ruled out $142$ candidates as false positives and flagged $105$ as potential false positives. Our final dispositions and comments for all the planet candidates are provided as a publicly available supplementary table., Comment: 17 pages, 11 figures, 5 tables. Accepted for publication on MNRAS

Counterfactual explanations are an emerging tool to enhance interpretability of deep learning models. Given a sample, these methods seek to find and display to the user similar samples across the decision boundary. In this paper, we propose a generative adversarial counterfactual approach for satellite image time series in a multi-class setting for the land cover classification task. One of the distinctive features of the proposed approach is the lack of prior assumption on the targeted class for a given counterfactual explanation. This inherent flexibility allows for the discovery of interesting information on the relationship between land cover classes. The other feature consists of encouraging the counterfactual to differ from the original sample only in a small and compact temporal segment. These time-contiguous perturbations allow for a much sparser and, thus, interpretable solution. Furthermore, plausibility/realism of the generated counterfactual explanations is enforced via the proposed adversarial learning strategy.

We statistically validated a sample of hot Neptune candidates applying a two-step vetting technique using DAVE and TRICERATOPS. We performed a systematic validation of 250 transit-like events in the Transiting Exoplanet Survey Satellite (TESS) archive in the parameter region defined by $P\leq 4$ d and $3R_\oplus\leq R\leq 5R_\oplus$. Through our analysis, we identified 18 hot Neptune-sized candidates, with a false positive probability $<50\%$. Nine of these planet candidates still need to be confirmed. For each of the nine targets we retrieved the stellar parameters using ARIADNE and derived constraints on the planetary parameters by fitting the lightcurves with the juliet package. Within this sample of nine candidates, we statistically validated (i.e, with false positive probability < $0.3\%$) two systems (TOI-277 b and TOI-1288 b) by re-processing the candidates with TRICERATOPS along with follow-up observations. These new validated exoplanets expand the known hot Neptunes population and are high-priority targets for future radial velocities follow-up., Comment: 24 pages, 20 figures. Accepted for publication on MNRAS

We report the Transiting Exoplanet Survey Satellite (TESS) discovery of a three-planet system around the bright Sun-like star HD~22946(V=8.3 mag),also known as TIC~100990000, located 63 parsecs away.The system was observed by TESS in Sectors 3, 4, 30 and 31 and two planet candidates, labelled TESS Objects of Interest (TOIs) 411.01 (planet $c$) and 411.02 (planet $b$), were identified on orbits of 9.57 and 4.04 days, respectively. In this work, we validate the two planets and recover an additional single transit-like signal in the light curve, which suggests the presence of a third transiting planet with a longer period of about 46 days.We assess the veracity of the TESS transit signals and use follow-up imaging and time series photometry to rule out false positive scenarios, including unresolved binary systems, nearby eclipsing binaries or background/foreground stars contaminating the light curves. Parallax measurements from Gaia EDR3, together with broad-band photometry and spectroscopic follow-up by TFOP allowed us to constrain the stellar parameters of TOI-411, including its radius of$1.157\pm0.025R_\odot$. Adopting this value, we determined the radii for the three exoplanet candidates and found that planet $b$ is a super-Earth, with a radius of $1.72\pm0.10R_\oplus$, while planet $c$ and $d$ are sub-Neptunian planets, with radii of$2.74\pm0.14R_\oplus$ and $3.23\pm0.19R_\oplus$ respectively. By using dynamical simulations, we assessed the stability of the system and evaluated the possibility of the presence of other undetected, non-transiting planets by investigating its dynamical packing. We find that the system is dynamically stable and potentially unpacked, with enough space to host at least one more planet between $c$ and $d$.(Abridged), Comment: 21 pages, 12 figures. Accepted for publication on A&A

The Transiting Exoplanet Survey Satellite (TESS) has detected thousands of exoplanet candidates since 2018, most of which have yet to be confirmed. A key step in the confirmation process of these candidates is ruling out false positives through vetting. Vetting also eases the burden on follow-up observations, provides input for demographics studies, and facilitates training machine learning algorithms. Here we present the TESS Triple-9 (TT9) catalog -- a uniformly-vetted catalog containing dispositions for 999 exoplanet candidates listed on ExoFOP-TESS, known as TESS Objects of Interest (TOIs). The TT9 was produced using the Discovery And Vetting of Exoplanets pipeline, DAVE, and utilizing the power of citizen science as part of the Planet Patrol project. More than 70% of the TOIs listed in the TT9 pass our diagnostic tests, and are thus marked as true planetary candidates. We flagged 144 candidates as false positives, and identified 146 as potential false positives. At the time of writing, the TT9 catalog contains ~20% of the entire ExoFOP-TESS TOIs list, demonstrates the synergy between automated tools and citizen science, and represents the first stage of our efforts to vet all TOIs. The DAVE generated results are publicly available on ExoFOP-TESS., Comment: 15 pages, 11 figures. Accepted on MNRAS

Oxygenic photosynthesis is the most important biochemical process in Earth biosphere and likely very common on other habitable terrestrial planets, given the general availability of its input chemical ingredients and of light as source of energy. It is therefore important to evaluate the effective possibility of oxygenic photosynthesis on planets around stars as a function of their spectral type and the planet-star separation. We aim at estimating the photon flux, the exergy and the exergetic efficiency of the radiation in the wavelength range useful for the oxygenic photosynthesis as a function of the host star effective temperature and planet-star separation. We compute analytically these quantities and compare our results with the estimates for the small sample of known Earth-like planets. We find that exergy is an increasing function of the star effective temperature, within the range 2600-7200 K. It depends both on the star-planet separation and the star effective temperature. Biospheres on exoplanets around cool stars might be generally light-limited. So far, we have not observed terrestrial planets comparable to Earth in terms of useful photon flux, exergy and exergetic efficiency., Comment: Accepted for publication on MNRAS

Deep Neural Networks (DNNs) are getting increasing attention to deal with Land Cover Classification (LCC) relying on Satellite Image Time Series (SITS). Though high performances can be achieved, the rationale of a prediction yielded by a DNN often remains unclear. An architecture expressing predictions with respect to input channels is thus proposed in this paper. It relies on convolutional layers and an attention mechanism weighting the importance of each channel in the final classification decision. The correlation between channels is taken into account to set up shared kernels and lower model complexity. Experiments based on a Sentinel-2 SITS show promising results.

Nowadays, modern Earth Observation systems continuously collect massive amounts of satellite information. The unprecedented possibility to acquire high resolution Satellite Image Time Series (SITS) data (series of images with high revisit time period on the same geographical area) is opening new opportunities to monitor the different aspects of the Earth Surface but, at the same time, it is raising up new challenges in term of suitable methods to analyze and exploit such huge amount of rich and complex image data. One of the main task associated to SITS data analysis is related to land cover mapping where satellite data are exploited via learning methods to recover the Earth Surface status aka the corresponding land cover classes. Due to operational constraints, the collected label information, on which machine learning strategies are trained, is often limited in volume and obtained at coarse granularity carrying out inexact and weak knowledge that can affect the whole process. To cope with such issues, in the context of object-based SITS land cover mapping, we propose a new deep learning framework, named TASSEL (aTtentive weAkly Supervised Satellite image time sEries cLassifier), that is able to intelligently exploit the weak supervision provided by the coarse granularity labels. Furthermore, our framework also produces an additional side-information that supports the model interpretability with the aim to make the black box gray. Such side-information allows to associate spatial interpretation to the model decision via visual inspection., Comment: Under submission to Elsevier journal

Nowadays, there is a general agreement on the need to better characterize agricultural monitoring systems in response to the global changes. Timely and accurate land use/land cover mapping can support this vision by providing useful information at fine scale. Here, a deep learning approach is proposed to deal with multi-source land cover mapping at object level. The approach is based on an extension of Recurrent Neural Network enriched via an attention mechanism dedicated to multi-temporal data context. Moreover, a new hierarchical pretraining strategy designed to exploit specific domain knowledge available under hierarchical relationships within land cover classes is introduced. Experiments carried out on the Reunion island - a french overseas department - demonstrate the significance of the proposal compared to remote sensing standard approaches for land cover mapping., Comment: Paper presented at the ICLR 2020 Workshop on Computer Vision for Agriculture (CV4A)