Search

We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality geometrically consistent and photo-realistic mirror reflections given an input image and a mask depicting the mirror region. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike., Comment: Project Page: https://val.cds.iisc.ac.in/reflecting-reality.github.io/

Recently, we have seen a surge of personalization methods for text-to-image (T2I) diffusion models to learn a concept using a few images. Existing approaches, when used for face personalization, suffer to achieve convincing inversion with identity preservation and rely on semantic text-based editing of the generated face. However, a more fine-grained control is desired for facial attribute editing, which is challenging to achieve solely with text prompts. In contrast, StyleGAN models learn a rich face prior and enable smooth control towards fine-grained attribute editing by latent manipulation. This work uses the disentangled $\mathcal{W+}$ space of StyleGANs to condition the T2I model. This approach allows us to precisely manipulate facial attributes, such as smoothly introducing a smile, while preserving the existing coarse text-based control inherent in T2I models. To enable conditioning of the T2I model on the $\mathcal{W+}$ space, we train a latent mapper to translate latent codes from $\mathcal{W+}$ to the token embedding space of the T2I model. The proposed approach excels in the precise inversion of face images with attribute preservation and facilitates continuous control for fine-grained attribute editing. Furthermore, our approach can be readily extended to generate compositions involving multiple individuals. We perform extensive experiments to validate our method for face personalization and fine-grained attribute editing., Comment: ECCV 2024, Project page: https://rishubhpar.github.io/PreciseControl.home/

For a given scene, humans can easily reason for the locations and pose to place objects. Designing a computational model to reason about these affordances poses a significant challenge, mirroring the intuitive reasoning abilities of humans. This work tackles the problem of realistic human insertion in a given background scene termed as \textbf{Semantic Human Placement}. This task is extremely challenging given the diverse backgrounds, scale, and pose of the generated person and, finally, the identity preservation of the person. We divide the problem into the following two stages \textbf{i)} learning \textit{semantic masks} using text guidance for localizing regions in the image to place humans and \textbf{ii)} subject-conditioned inpainting to place a given subject adhering to the scene affordance within the \textit{semantic masks}. For learning semantic masks, we leverage rich object-scene priors learned from the text-to-image generative models and optimize a novel parameterization of the semantic mask, eliminating the need for large-scale training. To the best of our knowledge, we are the first ones to provide an effective solution for realistic human placements in diverse real-world scenes. The proposed method can generate highly realistic scene compositions while preserving the background and subject identity. Further, we present results for several downstream tasks - scene hallucination from a single or multiple generated persons and text-based attribute editing. With extensive comparisons against strong baselines, we show the superiority of our method in realistic human placement., Comment: ECCV 2024, Project Page: https://rishubhpar.github.io/Text2Place/

Supernova remnants (SNRs) are believed to be capable of accelerating cosmic rays (CRs) to PeV energies. SNR G106.3+2.7 is a prime PeVatron candidate. It is formed by a head region, where the pulsar J2229+6114 and its boomerang-shaped pulsar wind nebula are located, and a tail region containing SN ejecta. The lack of observed gamma ray emission from the two regions of this SNR has made it difficult to assess which region would be responsible for the PeV CRs. We aim to characterize the very-high-energy (VHE, 0.1-100 TeV) gamma ray emission from SNR G106.3+2.7 by determining the morphology and spectral energy distribution of the region. This is accomplished using 2565 days of data and improved reconstruction algorithms from the HAWC Observatory. We also explore possible gamma ray production mechanisms for different energy ranges. Using a multi-source fitting procedure based on a maximum-likelihood estimation method, we evaluate the complex nature of this region. We determine the morphology, spectrum, and energy range for the source found in the region. Molecular cloud information is also used to create a template and evaluate the HAWC gamma ray spectral properties at ultra-high-energies (UHE, >56 TeV). This will help probe the hadronic nature of the highest-energy emission from the region. We resolve one extended source coincident with all other gamma ray observations of the region. The emission reaches above 100~TeV and its preferred log-parabola shape in the spectrum shows a flux peak in the TeV range. The molecular cloud template fit on the higher energy data reveals that the SNR's energy budget is fully capable of producing a purely hadronic source for UHE gamma rays.

HESS J1809-193 is an unidentified TeV source, first detected by the High Energy Stereoscopic System (H.E.S.S.) Collaboration. The emission originates in a source-rich region that includes several Supernova Remnants (SNR) and Pulsars (PSR) including SNR G11.1+0.1, SNR G11.0-0.0, and the young radio pulsar J1809-1917. Originally classified as a pulsar wind nebula (PWN) candidate, recent studies show the peak of the TeV region overlapping with a system of molecular clouds. This resulted in the revision of the original leptonic scenario to look for alternate hadronic scenarios. Marked as a potential PeVatron candidate, this region has been studied extensively by H.E.S.S. due to its emission extending up-to several tens of TeV. In this work, we use 2398 days of data from the High Altitude Water Cherenkov (HAWC) observatory to carry out a systematic source search for the HESS J1809-193 region. We were able to resolve emission detected as an extended component (modelled as a Symmetric Gaussian with a 1 $\sigma$ radius of 0.21 $^\circ$) with no clear cutoff at high energies and emitting photons up-to 210 TeV. We model the multi-wavelength observations for the region HESS J1809-193 using a time-dependent leptonic model and a lepto-hadronic model. Our model indicates that both scenarios could explain the observed data within the region of HESS J1809-193.

We report an observation of ultra-high energy (UHE) gamma rays from the Galactic Center region, using seven years of data collected by the High-Altitude Water Cherenkov (HAWC) Observatory. The HAWC data are best described as a point-like source (HAWC J1746-2856) with a power-law spectrum ($\mathrm{d}N/\mathrm{d}E=\phi(E/26 \,\text{TeV})^{\gamma}$), where $\gamma=-2.88 \pm 0.15_{\text{stat}} - 0.1_{\text{sys}} $ and $\phi=1.5 \times 10^{-15}$ (TeV cm$^{2}$s)$^{-1}$ $\pm\, 0.3_{\text{stat}}\,^{+0.08_{\text{sys}}}_{-0.13_{\text{sys}}}$ extending from 6 to 114 TeV. We find no evidence of a spectral cutoff up to $100$ TeV using HAWC data. Two known point-like gamma-ray sources are spatially coincident with the HAWC gamma-ray excess: Sgr A$^{*}$ (HESS J1745-290) and the Arc (HESS J1746-285). We subtract the known flux contribution of these point sources from the measured flux of HAWC J1746-2856 to exclude their contamination and show that the excess observed by HAWC remains significant ($>$5$\sigma$) with the spectrum extending to $>$100 TeV. Our result supports that these detected UHE gamma rays can originate via hadronic interaction of PeV cosmic-ray protons with the dense ambient gas and confirms the presence of a proton PeVatron at the Galactic Center.

The first TeV gamma-ray source with no lower energy counterparts, TeV J2032+4130, was discovered by HEGRA. It appears in the third HAWC catalog as 3HWC J2031+415 and it is a bright TeV gamma-ray source whose emission has previously been resolved as 2 sources: HAWC J2031+415 and HAWC J2030+409. While HAWC J2030+409 has since been associated with the \emph{Fermi-LAT} Cygnus Cocoon, no such association for HAWC J2031+415 has yet been found. In this work, we investigate the spectrum and energy-dependent morphology of HAWC J2031+415. We associate HAWC J2031+415 with the pulsar PSR J2032+4127 and perform a combined multi-wavelength analysis using radio, X-ray, and $\gamma$-ray emission. We conclude that HAWC J2031+415 and, by extension, TeV J2032+4130 are most probably a pulsar wind nebula (PWN) powered by PSR J2032+4127.

Text-to-image generation from large generative models like Stable Diffusion, DALLE-2, etc., have become a common base for various tasks due to their superior quality and extensive knowledge bases. As image composition and generation are creative processes the artists need control over various parts of the images being generated. We find that just adding details about parts in the base text prompt either leads to an entirely different image (e.g., missing/incorrect identity) or the extra part details simply being ignored. To mitigate these issues, we introduce PartCraft, which enables image generation based on fine-grained part-level details specified for objects in the base text prompt. This allows more control for artists and enables novel object compositions by combining distinctive object parts. PartCraft first localizes object parts by denoising the object region from a specific diffusion process. This enables each part token to be localized to the right object region. After obtaining part masks, we run a localized diffusion process in each of the part regions based on fine-grained part descriptions and combine them to produce the final image. All the stages of PartCraft are based on repurposing a pre-trained diffusion model, which enables it to generalize across various domains without training. We demonstrate the effectiveness of part-level control provided by PartCraft qualitatively through visual examples and quantitatively in comparison to the contemporary baselines., Comment: Project Page Will Be Here: https://rangwani-harsh.github.io/PartCraft

The need for abundant labelled data in supervised Adversarial Training (AT) has prompted the use of Self-Supervised Learning (SSL) techniques with AT. However, the direct application of existing SSL methods to adversarial training has been sub-optimal due to the increased training complexity of combining SSL with AT. A recent approach, DeACL, mitigates this by utilizing supervision from a standard SSL teacher in a distillation setting, to mimic supervised AT. However, we find that there is still a large performance gap when compared to supervised adversarial training, specifically on larger models. In this work, investigate the key reason for this gap and propose Projected Feature Adversarial Training (ProFeAT) to bridge the same. We show that the sub-optimal distillation performance is a result of mismatch in training objectives of the teacher and student, and propose to use a projection head at the student, that allows it to leverage weak supervision from the teacher while also being able to learn adversarially robust representations that are distinct from the teacher. We further propose appropriate attack and defense losses at the feature and projector, alongside a combination of weak and strong augmentations for the teacher and student respectively, to improve the training data diversity without increasing the training complexity. Through extensive experiments on several benchmark datasets and models, we demonstrate significant improvements in both clean and robust accuracy when compared to existing SSL-AT methods, setting a new state-of-the-art. We further report on-par/ improved performance when compared to TRADES, a popular supervised-AT method.

The High-Altitude Water Cherenkov (HAWC) Gamma-Ray Observatory located on the side of the Sierra Negra volcano in Mexico, has been fully operational since 2015. The HAWC collaboration has recently significantly improved their extensive-air-shower reconstruction algorithms, which has notably advanced the observatory performance. The energy resolution for primary gamma rays with energies below 1~TeV was improved by including a noise-suppression algorithm. Corrections have also been made to systematic errors in direction fitting related to the detector and shower plane inclinations, $\mathcal{O}(0.1^{\circ})$ biases in highly inclined showers, as well as enhancements to the core reconstruction. The angular resolution for gamma rays approaching the HAWC array from large zenith angles ($> 37^{\circ}$) has improved by a factor of four at the highest energies ($> 70$~TeV) as compared to previous reconstructions. The inclusion of a lateral distribution function fit to the extensive air shower footprint on the array to separate gamma-ray primaries from cosmic-ray ones, based on the resulting $\chi^{2}$ values, improved the background rejection performance at all inclinations. At large zenith angles, the improvement in significance is a factor of four compared to previous HAWC publications. These enhancements have been verified by observing the Crab Nebula, which is an overhead source for the HAWC Observatory. We show that the sensitivity to Crab-like point sources ($E^{-2.63}$) with locations overhead to 30$^{\circ}$ zenith is comparable or less than 10\% of the Crab Nebula's flux between 2 and 50~TeV. Thanks to these improvements, HAWC can now detect more sources, including the Galactic Center.

Galactic PeVatrons are sources that can accelerate cosmic rays to PeV energies. The high-energy cosmic rays are expected to interact with the surrounding ambient material or radiation, resulting in the production of gamma rays and neutrinos. To optimize for the detection of such associated production of gamma rays and neutrinos for a given source morphology and spectrum, a multi-messenger analysis that combines gamma rays and neutrinos is required. In this study, we use the Multi-Mission Maximum Likelihood framework (3ML) with IceCube Maximum Likelihood Analysis software (i3mla) and HAWC Accelerated Likelihood (HAL) to search for a correlation between 22 known gamma-ray sources from the third HAWC gamma-ray catalog and 14 years of IceCube track-like data. No significant neutrino emission from the direction of the HAWC sources was found. We report the best-fit gamma-ray model and 90% CL neutrino flux limit from the 22 sources. From the neutrino flux limit, we conclude that the gamma-ray emission from five of the sources can not be produced purely from hadronic interactions. We report the limit for the fraction of gamma rays produced by hadronic interactions for these five sources.

Vision Transformer (ViT) has emerged as a prominent architecture for various computer vision tasks. In ViT, we divide the input image into patch tokens and process them through a stack of self attention blocks. However, unlike Convolutional Neural Networks (CNN), ViTs simple architecture has no informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a large amount of data for pre-training. Various data efficient approaches (DeiT) have been proposed to train ViT on balanced datasets effectively. However, limited literature discusses the use of ViT for datasets with long-tailed imbalances. In this work, we introduce DeiT-LT to tackle the problem of training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an efficient and effective way of distillation from CNN via distillation DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes. This leads to the learning of local CNN-like features in early ViT blocks, improving generalization for tail classes. Further, to mitigate overfitting, we propose distilling from a flat CNN teacher, which leads to learning low-rank generalizable features for DIST tokens across all ViT blocks. With the proposed DeiT-LT scheme, the distillation DIST token becomes an expert on the tail classes, and the classifier CLS token becomes an expert on the head classes. The experts help to effectively learn features corresponding to both the majority and minority classes using a distinct set of tokens within the same ViT architecture. We show the effectiveness of DeiT-LT for training ViT from scratch on datasets ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018., Comment: CVPR 2024. Project Page: https://rangwani-harsh.github.io/DeiT-LT

Diffusion Models (DMs) have emerged as powerful generative models with unprecedented image generation capability. These models are widely used for data augmentation and creative applications. However, DMs reflect the biases present in the training datasets. This is especially concerning in the context of faces, where the DM prefers one demographic subgroup vs others (eg. female vs male). In this work, we present a method for debiasing DMs without relying on additional data or model retraining. Specifically, we propose Distribution Guidance, which enforces the generated images to follow the prescribed attribute distribution. To realize this, we build on the key insight that the latent features of denoising UNet hold rich demographic semantics, and the same can be leveraged to guide debiased generation. We train Attribute Distribution Predictor (ADP) - a small mlp that maps the latent features to the distribution of attributes. ADP is trained with pseudo labels generated from existing attribute classifiers. The proposed Distribution Guidance with ADP enables us to do fair generation. Our method reduces bias across single/multiple attributes and outperforms the baseline by a significant margin for unconditional and text-conditional diffusion models. Further, we present a downstream task of training a fair attribute classifier by rebalancing the training set with our generated data., Comment: CVPR 2024. Project Page : https://ab-34.github.io/balancing_act/

The Galactic gamma-ray diffuse emission (GDE) is emitted by cosmic rays (CRs), ultra-relativistic protons and electrons, interacting with gas and electromagnetic radiation fields in the interstellar medium. Here we present the analysis of TeV diffuse emission from a region of the Galactic Plane over the range in longitude of $l\in[43^\circ,73^\circ]$, using data collected with the High Altitude Water Cherenkov (HAWC) detector. Spectral, longitudinal and latitudinal distributions of the TeV diffuse emission are shown. The radiation spectrum is compatible with the spectrum of the emission arising from a CR population with an "index" similar to that of the observed CRs. When comparing with the \texttt{DRAGON} \textit{base model}, the HAWC GDE flux is higher by about a factor of two. Unresolved sources such as pulsar wind nebulae and TeV halos could explain the excess emission. Finally, deviations of the Galactic CR flux from the locally measured CR flux may additionally explain the difference between the predicted and measured diffuse fluxes.

Vision-Language Models (VLMs) such as CLIP are trained on large amounts of image-text pairs, resulting in remarkable generalization across several data distributions. However, in several cases, their expensive training and data collection/curation costs do not justify the end application. This motivates a vendor-client paradigm, where a vendor trains a large-scale VLM and grants only input-output access to clients on a pay-per-query basis in a black-box setting. The client aims to minimize inference cost by distilling the VLM to a student model using the limited available task-specific data, and further deploying this student model in the downstream application. While naive distillation largely improves the In-Domain (ID) accuracy of the student, it fails to transfer the superior out-of-distribution (OOD) generalization of the VLM teacher using the limited available labeled images. To mitigate this, we propose Vision-Language to Vision - Align, Distill, Predict (VL2V-ADiP), which first aligns the vision and language modalities of the teacher model with the vision modality of a pre-trained student model, and further distills the aligned VLM representations to the student. This maximally retains the pre-trained features of the student, while also incorporating the rich representations of the VLM image encoder and the superior generalization of the text embeddings. The proposed approach achieves state-of-the-art results on the standard Domain Generalization benchmarks in a black-box teacher setting as well as a white-box setting where the weights of the VLM are accessible., Comment: Project page: http://val.cds.iisc.ac.in/VL2V-ADiP/

Neural radiance field (NeRF) based methods enable high-quality novel-view synthesis for multi-view images. This work presents a method for synthesizing colorized novel views from input grey-scale multi-view images. When we apply image or video-based colorization methods on the generated grey-scale novel views, we observe artifacts due to inconsistency across views. Training a radiance field network on the colorized grey-scale image sequence also does not solve the 3D consistency issue. We propose a distillation based method to transfer color knowledge from the colorization networks trained on natural images to the radiance field network. Specifically, our method uses the radiance field network as a 3D representation and transfers knowledge from existing 2D colorization methods. The experimental results demonstrate that the proposed method produces superior colorized novel views for indoor and outdoor scenes while maintaining cross-view consistency than baselines. Further, we show the efficacy of our method on applications like colorization of radiance field network trained from 1.) Infra-Red (IR) multi-view images and 2.) Old grey-scale multi-view image sequences., Comment: AI3DCC @ ICCV 2023

Recently, the region surrounding eHWC J1842-035 has been studied extensively by gamma-ray observatories due to its extended emission reaching up to a few hundred TeV and potential as a hadronic accelerator. In this work, we use 1,910 days of cumulative data from the High Altitude Water Cherenkov (HAWC) observatory to carry out a dedicated systematic source search of the eHWC J1842-035 region. During the search we have found three sources in the region, namely, HAWC J1844-034, HAWC J1843-032, and HAWC J1846-025. We have identified HAWC J1844-034 as the extended source that emits photons with energies up to 175 TeV. We compute the spectrum for HAWC J1844-034 and by comparing with the observational results from other experiments, we have identified HESS J1843-033, LHAASO J1843-0338, and TASG J1844-038 as very-high-energy gamma-ray sources with a matching origin. Also, we present and use the multi-wavelength data to fit the hadronic and leptonic particle spectra. We have identified four pulsar candidates in the nearby region from which PSR J1844-0346 is found to be the most likely candidate due to its proximity to HAWC J1844-034 and the computed energy budget. We have also found SNR G28.6-0.1 as a potential counterpart source of HAWC J1844-034 for which both leptonic and hadronic scenarios are feasible., Comment: 13 pages, 9 figures, published in ApJ

The decay or annihilation of dark matter particles may produce a steady flux of very-high-energy gamma rays detectable above the diffuse background. Nearby clusters of galaxies provide excellent targets to search for the signatures of particle dark matter interactions. In particular, the Virgo cluster spans several degrees across the sky and can be efficiently probed with a wide field-of-view instrument. The High Altitude Water Cherenkov (HAWC) observatory, due to its wide field of view and sensitivity to gamma rays at an energy scale of 300 GeV--100 TeV is well-suited for this search. Using 2141 days of data, we search for gamma-ray emission from the Virgo cluster, assuming well-motivated dark matter sub-structure models. Our results provide some of the strongest constraints on the decay lifetime of dark matter for masses above 10 TeV., Comment: 7 pages, 3 figures. Accepted for publication in Physical Review D

Conventional Domain Adaptation (DA) methods aim to learn domain-invariant feature representations to improve the target adaptation performance. However, we motivate that domain-specificity is equally important since in-domain trained models hold crucial domain-specific properties that are beneficial for adaptation. Hence, we propose to build a framework that supports disentanglement and learning of domain-specific factors and task-specific factors in a unified model. Motivated by the success of vision transformers in several multi-modal vision problems, we find that queries could be leveraged to extract the domain-specific factors. Hence, we propose a novel Domain-specificity-inducing Transformer (DSiT) framework for disentangling and learning both domain-specific and task-specific factors. To achieve disentanglement, we propose to construct novel Domain-Representative Inputs (DRI) with domain-specific information to train a domain classifier with a novel domain token. We are the first to utilize vision transformers for domain adaptation in a privacy-oriented source-free setting, and our approach achieves state-of-the-art performance on single-source, multi-source, and multi-target benchmarks, Comment: ICCV 2023. Project page: http://val.cds.iisc.ac.in/DSiT-SFDA

Neural Radiance Field (NeRF) approaches learn the underlying 3D representation of a scene and generate photo-realistic novel views with high fidelity. However, most proposed settings concentrate on modelling a single object or a single level of a scene. However, in the real world, we may capture a scene at multiple levels, resulting in a layered capture. For example, tourists usually capture a monument's exterior structure before capturing the inner structure. Modelling such scenes in 3D with seamless switching between levels can drastically improve immersive experiences. However, most existing techniques struggle in modelling such scenes. We propose Strata-NeRF, a single neural radiance field that implicitly captures a scene with multiple levels. Strata-NeRF achieves this by conditioning the NeRFs on Vector Quantized (VQ) latent representations which allow sudden changes in scene structure. We evaluate the effectiveness of our approach in multi-layered synthetic dataset comprising diverse scenes and then further validate its generalization on the real-world RealEstate10K dataset. We find that Strata-NeRF effectively captures stratified scenes, minimizes artifacts, and synthesizes high-fidelity views compared to existing approaches., Comment: ICCV 2023, Project Page: https://ankitatiisc.github.io/Strata-NeRF/