Search

We describe the software package $\texttt{ESpRESSO}$ - [E]xtragalactic [Sp]ectroscopic [R]oman [E]mulator and [S]imulator of [S]ynthetic [O]bjects, created to emulate the slitless spectroscopic observing modes of the Nancy Grace Roman Space Telescope (Roman) Wide Field Instrument (WFI). We combine archival Hubble Space Telescope (HST) imaging data of comparable spatial resolution with model spectral energy distributions to create a data-cube of flux density as a function of position and wavelength. This data-cube is used for simulating a nine detector grism observation, producing a crowded background scene which model field angle dependent optical distortions expected for the grism. We also demonstrate the ability to inject custom sources using the described tools and pipelines. In addition, we show that spectral features such as emission line pairs are unlikely to be mistaken as off order contaminating features and vice versa. Our result is a simulation suite of half of the eighteen detector array, with a realistic background scene and injected Ly$\alpha$ emitter (LAE) galaxies, realized at 25 position angles (PAs), 12 with analogous positive and negative dithers, Using an exposure time of 10ks per PA, the full PA set can be used as a mock deep Roman grism survey with high (synthetic) LAE completeness for developing future spectral data analysis tools., Comment: 15 pages, 10 figures, 1 supplementary figure, 1 table. Submitted to MNRAS. Comments are welcome!

Generalized Category Discovery (GCD) aims to cluster unlabeled images into known and novel categories using labeled images from known classes. To address the challenge of transferring features from known to unknown classes while mitigating model bias, we introduce GraphVL, a novel approach for vision-language modeling in GCD, leveraging CLIP. Our method integrates a graph convolutional network (GCN) with CLIP's text encoder to preserve class neighborhood structure. We also employ a lightweight visual projector for image data, ensuring discriminative features through margin-based contrastive losses for image-text mapping. This neighborhood preservation criterion effectively regulates the semantic space, making it less sensitive to known classes. Additionally, we learn textual prompts from known classes and align them to create a more contextually meaningful semantic feature space for the GCN layer using a contextual similarity loss. Finally, we represent unlabeled samples based on their semantic distance to class prompts from the GCN, enabling semi-supervised clustering for class discovery and minimizing errors. Our experiments on seven benchmark datasets consistently demonstrate the superiority of GraphVL when integrated with the CLIP backbone., Comment: Accepted in ACM ICVGIP 2024

Multi-Target Domain Adaptation (MTDA) entails learning domain-invariant information from a single source domain and applying it to multiple unlabeled target domains. Yet, existing MTDA methods predominantly focus on addressing domain shifts within visual features, often overlooking semantic features and struggling to handle unknown classes, resulting in what is known as Open-Set (OS) MTDA. While large-scale vision-language foundation models like CLIP show promise, their potential for MTDA remains largely unexplored. This paper introduces COSMo, a novel method that learns domain-agnostic prompts through source domain-guided prompt learning to tackle the MTDA problem in the prompt space. By leveraging a domain-specific bias network and separate prompts for known and unknown classes, COSMo effectively adapts across domain and class shifts. To the best of our knowledge, COSMo is the first method to address Open-Set Multi-Target DA (OSMTDA), offering a more realistic representation of real-world scenarios and addressing the challenges of both open-set and multi-target DA. COSMo demonstrates an average improvement of $5.1\%$ across three challenging datasets: Mini-DomainNet, Office-31, and Office-Home, compared to other related DA methods adapted to operate within the OSMTDA setting. Code is available at: https://github.com/munish30monga/COSMo, Comment: Accepted in BMVC 2024

We address the challenges inherent in sketch-based image retrieval (SBIR) across various settings, including zero-shot SBIR, generalized zero-shot SBIR, and fine-grained zero-shot SBIR, by leveraging the vision-language foundation model CLIP. While recent endeavors have employed CLIP to enhance SBIR, these approaches predominantly follow uni-modal prompt processing and overlook to exploit CLIP's integrated visual and textual capabilities fully. To bridge this gap, we introduce SpLIP, a novel multi-modal prompt learning scheme designed to operate effectively with frozen CLIP backbones. We diverge from existing multi-modal prompting methods that treat visual and textual prompts independently or integrate them in a limited fashion, leading to suboptimal generalization. SpLIP implements a bi-directional prompt-sharing strategy that enables mutual knowledge exchange between CLIP's visual and textual encoders, fostering a more cohesive and synergistic prompt processing mechanism that significantly reduces the semantic gap between the sketch and photo embeddings. In addition to pioneering multi-modal prompt learning, we propose two innovative strategies for further refining the embedding space. The first is an adaptive margin generation for the sketch-photo triplet loss, regulated by CLIP's class textual embeddings. The second introduces a novel task, termed conditional cross-modal jigsaw, aimed at enhancing fine-grained sketch-photo alignment by implicitly modeling sketches' viable patch arrangement using knowledge of unshuffled photos. Our comprehensive experimental evaluations across multiple benchmarks demonstrate the superior performance of SpLIP in all three SBIR scenarios. Project page: https://mainaksingha01.github.io/SpLIP/ ., Comment: Accepted in ECCV 2024

We searched the Chandra and XMM archives for observations of 900 green pea galaxies to find AGN signatures. Green peas are low-mass galaxies with prominent emission lines, similar in size and star formation rate to high-redshift dwarf galaxies. Of the 29 observations found, 9 show X-ray detections with $S/N>3$. The 2-10 keV X-ray luminosity for these 9 sources exceeds $10^{40}~\mathrm{erg~s}^{-1}$, with 2 sources exceeding $10^{41}~\mathrm{erg~s}^{-1}$, suggesting the presence of intermediate-mass black holes (IMBH) or low-luminosity AGN (LLAGN) with BH masses between $100-10^6M_\mathrm{\odot}$. All X-ray detected sources (plus 6 additional sources) show He~II$\lambda4686$ emission and a broad component of the H$\alpha$ emission line, indicating winds. The line widths of the broad H$\alpha$ and He II$\lambda4686$ emitting gas clouds are weakly correlated ($R^{2}=0.15$), suggesting He II$\lambda4686$ emission is inconsistent with winds from super-Eddington accretors. However, the ratio of X-ray luminosity to star formation rate shows an anti-correlation with metallicity in 5 out of 9 X-ray detected sources, implying ultraluminous X-ray sources are key contributors to the observed X-ray luminosity. This could be due to super-Eddington accretors or IMBH. The X-ray emission is much higher than that produced by Wolf-Rayet stars and supernovae-driven winds. Thus, the X-ray luminosity in these 9 sources can only be explained by black holes with masses over $100~M_\mathrm{\odot}$. Our findings suggest the presence of LLAGN in these galaxies, with broad H$\alpha$ line widths implying BH masses of $10^4-10^6M_\mathrm{\odot}$. Given Green Peas' role as significant Lyman Continuum leakers, LLAGN in these galaxies could have contributed significantly to cosmic reionization., Comment: Submitted to ApJ. 17 pages, 11 figures and 3 tables. Comments welcome

In Generalized Category Discovery (GCD), we cluster unlabeled samples of known and novel classes, leveraging a training dataset of known classes. A salient challenge arises due to domain shifts between these datasets. To address this, we present a novel setting: Across Domain Generalized Category Discovery (AD-GCD) and bring forth CDAD-NET (Class Discoverer Across Domains) as a remedy. CDAD-NET is architected to synchronize potential known class samples across both the labeled (source) and unlabeled (target) datasets, while emphasizing the distinct categorization of the target data. To facilitate this, we propose an entropy-driven adversarial learning strategy that accounts for the distance distributions of target samples relative to source-domain class prototypes. Parallelly, the discriminative nature of the shared space is upheld through a fusion of three metric learning objectives. In the source domain, our focus is on refining the proximity between samples and their affiliated class prototypes, while in the target domain, we integrate a neighborhood-centric contrastive learning mechanism, enriched with an adept neighborsmining approach. To further accentuate the nuanced feature interrelation among semantically aligned images, we champion the concept of conditional image inpainting, underscoring the premise that semantically analogous images prove more efficacious to the task than their disjointed counterparts. Experimentally, CDAD-NET eclipses existing literature with a performance increment of 8-15% on three AD-GCD benchmarks we present., Comment: Accepted in L3D-IVU, CVPR Workshop, 2024

We delve into Open Domain Generalization (ODG), marked by domain and category shifts between training's labeled source and testing's unlabeled target domains. Existing solutions to ODG face limitations due to constrained generalizations of traditional CNN backbones and errors in detecting target open samples in the absence of prior knowledge. Addressing these pitfalls, we introduce ODG-CLIP, harnessing the semantic prowess of the vision-language model, CLIP. Our framework brings forth three primary innovations: Firstly, distinct from prevailing paradigms, we conceptualize ODG as a multi-class classification challenge encompassing both known and novel categories. Central to our approach is modeling a unique prompt tailored for detecting unknown class samples, and to train this, we employ a readily accessible stable diffusion model, elegantly generating proxy images for the open class. Secondly, aiming for domain-tailored classification (prompt) weights while ensuring a balance of precision and simplicity, we devise a novel visual stylecentric prompt learning mechanism. Finally, we infuse images with class-discriminative knowledge derived from the prompt space to augment the fidelity of CLIP's visual embeddings. We introduce a novel objective to safeguard the continuity of this infused semantic intel across domains, especially for the shared classes. Through rigorous testing on diverse datasets, covering closed and open-set DG contexts, ODG-CLIP demonstrates clear supremacy, consistently outpacing peers with performance boosts between 8%-16%. Code will be available at https://github.com/mainaksingha01/ODG-CLIP., Comment: Accepted in CVPR 2024

Recognition of remote sensing (RS) or aerial images is currently of great interest, and advancements in deep learning algorithms added flavor to it in recent years. Occlusion, intra-class variance, lighting, etc., might arise while training neural networks using unimodal RS visual input. Even though joint training of audio-visual modalities improves classification performance in a low-data regime, it has yet to be thoroughly investigated in the RS domain. Here, we aim to solve a novel problem where both the audio and visual modalities are present during the meta-training of a few-shot learning (FSL) classifier; however, one of the modalities might be missing during the meta-testing stage. This problem formulation is pertinent in the RS domain, given the difficulties in data acquisition or sensor malfunctioning. To mitigate, we propose a novel few-shot generative framework, Hallucinated Audio-Visual Embeddings-Network (HAVE-Net), to meta-train cross-modal features from limited unimodal data. Precisely, these hallucinated features are meta-learned from base classes and used for few-shot classification on novel classes during the inference phase. The experimental results on the benchmark ADVANCE and AudioSetZSL datasets show that our hallucinated modality augmentation strategy for few-shot classification outperforms the classifier performance trained with the real multimodal information at least by 0.8-2%., Comment: 8 Page, 2 Figures, 2 Tables, Accepted in Adapting to Change: Reliable Multimodal Learning Across Domains Workshop, ECML PKDD 2023

Large-scale foundation models, such as CLIP, have demonstrated remarkable success in visual recognition tasks by embedding images in a semantically rich space. Self-supervised learning (SSL) has also shown promise in improving visual recognition by learning invariant features. However, the combination of CLIP with SSL is found to face challenges due to the multi-task framework that blends CLIP's contrastive loss and SSL's loss, including difficulties with loss weighting and inconsistency among different views of images in CLIP's output space. To overcome these challenges, we propose a prompt learning-based model called GOPro, which is a unified framework that ensures similarity between various augmented views of input images in a shared image-text embedding space, using a pair of learnable image and text projectors atop CLIP, to promote invariance and generalizability. To automatically learn such prompts, we leverage the visual content and style primitives extracted from pre-trained CLIP and adapt them to the target task. In addition to CLIP's cross-domain contrastive loss, we introduce a visual contrastive loss and a novel prompt consistency loss, considering the different views of the images. GOPro is trained end-to-end on all three loss objectives, combining the strengths of CLIP and SSL in a principled manner. Empirical evaluations demonstrate that GOPro outperforms the state-of-the-art prompting techniques on three challenging domain generalization tasks across multiple benchmarks by a significant margin. Our code is available at https://github.com/mainaksingha01/GOPro., Comment: Accepted at BMVC 2023

Although deep learning models have shown impressive performance on supervised learning tasks, they often struggle to generalize well when the training (source) and test (target) domains differ. Unsupervised domain adaptation (DA) has emerged as a popular solution to this problem. However, current DA techniques rely on visual backbones, which may lack semantic richness. Despite the potential of large-scale vision-language foundation models like CLIP, their effectiveness for DA has yet to be fully explored. To address this gap, we introduce \textsc{AD-CLIP}, a domain-agnostic prompt learning strategy for CLIP that aims to solve the DA problem in the prompt space. We leverage the frozen vision backbone of CLIP to extract both image style (domain) and content information, which we apply to learn prompt tokens. Our prompts are designed to be domain-invariant and class-generalizable, by conditioning prompt learning on image style and content features simultaneously. We use standard supervised contrastive learning in the source domain, while proposing an entropy minimization strategy to align domains in the embedding space given the target domain data. We also consider a scenario where only target domain samples are available during testing, without any source domain data, and propose a cross-domain style mapping network to hallucinate domain-agnostic tokens. Our extensive experiments on three benchmark DA datasets demonstrate the effectiveness of \textsc{AD-CLIP} compared to existing literature. Code is available at \url{https://github.com/mainaksingha01/AD-CLIP}, Comment: 10 pages, 8 figures, 4 tables. Accepted at OOD-CV, ICCV Workshop, 2023

Although double-peaked narrow emission-line galaxies have been studied extensively in the past years, only a few are reported with the green pea galaxies (GPs). Here we present our discovery of five GPs with double-peaked narrow [OIII] emission lines, referred to as DPGPs, selected from the LAMOST and SDSS spectroscopic surveys. We find that these five DPGPs have blueshifted narrow components more prominent than the redshifted components, with velocity offsets of [OIII]$\lambda$5007 lines ranging from 306 to 518 $\rm km\, s^{-1}$ and full widths at half maximums (FWHMs) of individual components ranging from 263 to 441 $\rm km\, s^{-1}$. By analyzing the spectra and the spectral energy distributions (SEDs), we find that they have larger metallicities and stellar masses compared with other GPs. The H$\alpha$ line width, emission-line diagnostic, mid-infrared color, radio emission, and SED fitting provide evidence of the AGN activities in these DPGPs. They have the same spectral properties of Type 2 quasars. Furthermore, we discuss the possible nature of the double-peaked narrow emission-line profiles of these DPGPs and find that they are more likely to be dual AGN. These DPGP galaxies are ideal laboratories for exploring the growth mode of AGN in the extremely luminous emission-line galaxies, the co-evolution between AGN and host galaxies, and the evolution of high-redshift galaxies in the early Universe., Comment: 13 pages, 8 figures, 3 tables; Accepted for publication in MNRAS

In recent years, the success of large-scale vision-language models (VLMs) such as CLIP has led to their increased usage in various computer vision tasks. These models enable zero-shot inference through carefully crafted instructional text prompts without task-specific supervision. However, the potential of VLMs for generalization tasks in remote sensing (RS) has not been fully realized. To address this research gap, we propose a novel image-conditioned prompt learning strategy called the Visual Attention Parameterized Prompts Learning Network (APPLeNet). APPLeNet emphasizes the importance of multi-scale feature learning in RS scene classification and disentangles visual style and content primitives for domain generalization tasks. To achieve this, APPLeNet combines visual content features obtained from different layers of the vision encoder and style properties obtained from feature statistics of domain-specific batches. An attention-driven injection module is further introduced to generate visual tokens from this information. We also introduce an anti-correlation regularizer to ensure discrimination among the token embeddings, as this visual information is combined with the textual tokens. To validate APPLeNet, we curated four available RS benchmarks and introduced experimental protocols and datasets for three domain generalization tasks. Our results consistently outperform the relevant literature and code is available at https://github.com/mainaksingha01/APPLeNet, Comment: 11 Pages, 6 figures, 8 tables, Accepted in Earth Vision (CVPR 2023)

Large-scale foundation models (e.g., CLIP) have shown promising zero-shot generalization performance on downstream tasks by leveraging carefully designed language prompts. However, despite their success, most prompt learning techniques tend to underperform in the presence of domain shift. Our study addresses this problem and, to improve CLIP's generalization ability across domains, proposes \textsc{StyLIP}, a novel approach for Domain Generalization (DG) based on a domain-agnostic prompt learning strategy. In the absence of explicit domain knowledge, we aim to disentangle the visual style and the content information extracted from the pre-trained CLIP in the prompts so they can be effortlessly adapted to novel domains during inference. Furthermore, we consider a set of style projectors to learn the prompt tokens directly from these multi-scale style features, and the generated prompt embeddings are later fused with the multi-scale visual features learned through a content projector. The projectors are contrastively trained, given CLIP's frozen vision and text encoders. We present extensive experiments in five different DG settings on multiple benchmarks, demonstrating that \textsc{StyLIP} consistently outperforms the relevant state-of-the-art methods., Comment: 23 pages, 7 figures, 9 tables

The interaction between Active Galactic Nuclei (AGN) and their host galaxies is scarcely resolved. Narrow-line Seyfert 1 (NLS1) galaxies are believed to represent AGN at early stages of their evolution and allow to observe AGN feeding and feedback processes at high accretion rates. We apply a spectroastrometric analysis to VLT MUSE NFM-AO observations of Mrk 1044, a nearby super-Eddington accreting NLS1. This allows us to map two ionised gas outflows traced by [O$\,$III] which have velocities of $-560\pm20\,{\rm km\:s}^{-1}$ and $-144 \pm 5 \,{\rm km\:s}^{-1}$. Both outflows are spatially unresolved and located close to the galaxy nucleus ($<1\,{\rm pc}$). They have gas densities higher than $10^5\,{\rm cm}^{-3}$, which implies that the BPT diagnostic cannot be used to constrain the underlying ionisation mechanism. We explore whether an expanding shell model can describe the velocity structure of Mrk 1044's unresolved multi-phase outflow. A kinematic analysis suggests that significant turbulence may be present in the ISM around the nucleus, which may lead to a condensation rain, potentially explaining the efficient feeding of Mrk 1044's AGN. We identify an additional ionised gas outflowing component that is spatially resolved. It has a velocity of $-211 \pm 22 \,{\rm km\:s}^{-1}$ and a projected size of $4.6 \pm 0.6 \,{\rm pc}$. Within the innermost 0.5" (160$\,{\rm pc}$) around the nucleus we detect modest star formation hidden by the beam-smeared emission from the outflow, which suggests that Mrk 1044's AGN phase set on recently. We estimate that the multi-phase outflow has been launched $<10^4 \,{\rm yrs}$ ago. It carries enough mass and energy to impact the host galaxy star formation on different spatial scales, highlighting the complexity of the AGN feeding and feedback cycle in its early stages., Comment: 17 pages. 12 figures. Re-submitted to A&A after minor revision. Comments welcome