Search

A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language models (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, the threat models have changed. Previously, having gray-box access, where input embeddings or output logits/probabilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, no information about the model is available apart from the generated output. This means that current black-box attacks can only utilize the final prediction to detect if an attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to using output probabilities, while having only black-box access in a classification setting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current LLMs. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models (LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique to existing hard-label black-box attack methods that introduce word-level substitutions., Comment: Published in ICLR 2025. The code is publicly available at https://github.com/Aniloid2/Confidence_Elicitation_Attacks

The performance of a machine learning (ML) model depends heavily on the relevance of its training data to the domain of the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often not known to us (e.g., conversations between an LLM and a user are end-to-end encrypted). So, it is not obvious what data would be relevant for training/fine-tuning the ML model to maximize its task performance. Instead, one can only deploy the ML model in the unseen evaluation task to gather multiple rounds of coarse feedback on how well the model has performed. This paper presents a novel global-to-local algorithm called DUET that can exploit the feedback loop by interleaving a data selection method with Bayesian optimization. As a result, DUET can efficiently refine the training data mixture from a pool of data domains to maximize the model's performance on the unseen evaluation task and its convergence to the optimal data mixture can be theoretically guaranteed by analyzing its cumulative regret. Empirical evaluation on image and LLM evaluation tasks shows that DUET finds better training data mixtures than conventional baselines.

A lattice-based signature, called G+G convoluted Gaussian signature was proposed in ASIACRYPT 2023 and was proved secure in the quantum random oracle model. In this paper, we propose a ratio attack on the G+G convoluted Gaussian signature to recover the secret key. The attack exploits the fact, proved in this paper, that the secret key can be obtained from the expected value of the ratio of signatures which follows a truncated Cauchy distribution. Moreover, we also compute the number of signatures required to successfully recover the secret key. Furthermore, we simulate the ratio attack in Sagemath with a few different parameters as a proof-of-concept of the ratio attack.

We show that the average trajectories of relativistic quantum particles in Schwarzschild spacetime, obtained via quantum mechanical weak measurements of momentum and energy, are equivalent to the predicted flow lines of probability current in curved spacetime quantum theory. We subsequently demonstrate that these trajectories correspond exactly to classical null geodesics in a hybrid Schwarzschild-Alcubierre spacetime. This threefold equivalence demonstrates how quantum theory in curved spacetime can be formulated via operationally-defined measurements, and that such a theory may be interpreted deterministically, in the spirit of hidden-variable models such as Bohmian mechanics, through the novel connection to an underlying "guiding metric.", Comment: 5+9 pages, 1 figure

While cancer has traditionally been considered a genetic disease, mounting evidence indicates an important role for non-genetic (epigenetic) mechanisms. Common anti-cancer drugs have recently been observed to induce the adoption of reversible drug-tolerant cell states, thereby accelerating the evolution of drug resistance. Determining how to optimally balance the competing goals of killing the tumor bulk and delaying resistance evolution in this scenario is a nontrivial question of high clinical importance. In this work, we use a combined mathematical and computational approach to study optimal dosing of anti-cancer drug treatment under drug-induced cell plasticity. Our results show that the optimal treatment steers the tumor into a fixed equilibrium composition while balancing the trade-off between cell kill and tolerance induction in a precisely quantifiable way. Under linear induction of tolerance, a low-dose constant strategy is optimal in equilibrium, while under uniform induction of tolerance, alternating between a large dose and no dose is best. The directionality of drug induction, whether the drug elevates transitions from sensitivity to tolerance or inhibits transitions back, significantly affects optimal dosing. To demonstrate the applicability of our approach, we use it to identify an optimal low-dose strategy for colorectal cancer using publicly available in vitro data., Comment: 46 pages, 10 figures

Much effort has been made to explain and improve the success of transfer-based attacks (TBA) on black-box computer vision models. This work provides the first attempt at a priori prediction of attack success by identifying the presence of vulnerable features within target models. Recent work by Chen and Liu (2024) proposed the manifold attack model, a unifying framework proposing that successful TBA exist in a common manifold space. Our work experimentally tests the common manifold space hypothesis by a new methodology: first, projecting feature vectors from surrogate and target feature extractors trained on ImageNet onto the same low-dimensional manifold; second, quantifying any observed structure similarities on the manifold; and finally, by relating these observed similarities to the success of the TBA. We find that shared feature representation moderately correlates with increased success of TBA (\r{ho}= 0.56). This method may be used to predict whether an attack will transfer without information of the model weights, training, architecture or details of the attack. The results confirm the presence of shared feature representations between two feature extractors of different sizes and complexities, and demonstrate the utility of datasets from different target domains as test signals for interpreting black-box feature representations.

Deep learning approaches have been widely adopted for precipitation nowcasting in recent years. Previous studies mainly focus on proposing new model architectures to improve pixel-wise metrics. However, they frequently result in blurry predictions which provide limited utility to forecasting operations. In this work, we propose a new Fourier Amplitude and Correlation Loss (FACL) which consists of two novel loss terms: Fourier Amplitude Loss (FAL) and Fourier Correlation Loss (FCL). FAL regularizes the Fourier amplitude of the model prediction and FCL complements the missing phase information. The two loss terms work together to replace the traditional $L_2$ losses such as MSE and weighted MSE for the spatiotemporal prediction problem on signal-based data. Our method is generic, parameter-free and efficient. Extensive experiments using one synthetic dataset and three radar echo datasets demonstrate that our method improves perceptual metrics and meteorology skill scores, with a small trade-off to pixel-wise accuracy and structural similarity. Moreover, to improve the error margin in meteorological skill scores such as Critical Success Index (CSI) and Fractions Skill Score (FSS), we propose and adopt the Regional Histogram Divergence (RHD), a distance metric that considers the patch-wise similarity between signal-based imagery patterns with tolerance to local transforms. Code is available at https://github.com/argenycw/FACL, Comment: Accepted by NeurIPS 2024. Camera-ready submission

Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack budget, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods., Comment: 19 pages, 4 figures, 8 tables

Data valuation is a class of techniques for quantitatively assessing the value of data for applications like pricing in data marketplaces. Existing data valuation methods define a value for a discrete dataset. However, in many use cases, users are interested in not only the value of the dataset, but that of the distribution from which the dataset was sampled. For example, consider a buyer trying to evaluate whether to purchase data from different vendors. The buyer may observe (and compare) only a small preview sample from each vendor, to decide which vendor's data distribution is most useful to the buyer and purchase. The core question is how should we compare the values of data distributions from their samples? Under a Huber characterization of the data heterogeneity across vendors, we propose a maximum mean discrepancy (MMD)-based valuation method which enables theoretically principled and actionable policies for comparing data distributions from samples. We empirically demonstrate that our method is sample-efficient and effective in identifying valuable data distributions against several existing baselines, on multiple real-world datasets (e.g., network intrusion detection, credit card fraud detection) and downstream applications (classification, regression)., Comment: Accepted to NeurIPS 2024 as a poster. Main paper with appendix (38 pages in total). Code will be released soon at https://github.com/XinyiYS/Data_Distribution_Valuation

Singlish, or formally Colloquial Singapore English, is an English-based creole language originating from the SouthEast Asian country Singapore. The language contains influences from Sinitic languages such as Chinese dialects, Malay, Tamil and so forth. A fundamental task to understanding Singlish is to first understand the pragmatic functions of its discourse particles, upon which Singlish relies heavily to convey meaning. This work offers a preliminary effort to disentangle the Singlish discourse particles (lah, meh and hor) with task-driven representation learning. After disentanglement, we cluster these discourse particles to differentiate their pragmatic functions, and perform Singlish-to-English machine translation. Our work provides a computational method to understanding Singlish discourse particles, and opens avenues towards a deeper comprehension of the language and its usage.

Resistance to therapy remains a significant challenge in cancer treatment, often due to the presence of a stem-like cell population that drives tumor recurrence post-treatment. Moreover, many anticancer therapies induce plasticity, converting initially drug-sensitive cells to a more resistant state, e.g. through epigenetic processes and de-differentiation programs. Understanding the balance between therapeutic anti-tumor effects and induced resistance is critical for identifying treatment strategies. In this study, we introduce a robust statistical framework, based on multi-type branching process models of the evolutionary dynamics of tumor cell populations, to detect and quantify therapy-induced resistance phenomena from high throughput drug screening data. Through comprehensive in silico experiments, we show the efficacy of our framework in estimating parameters governing population dynamics and drug responses in a heterogeneous tumor population where cell state transitions are influenced by the drug. Finally, using recent in vitro data from multiple sources, we demonstrate that our framework is effective for analyzing real-world data and generating meaningful predictions., Comment: We have updated our manuscript to reflect the latest version submitted to the journal

Optical microscopy has long been the standard method for diagnosis in cytopathology. Whole slide scanners can image and digitize large sample areas automatically, but are slow, expensive and therefore not widely available. Clinical diagnosis of cytology specimens is especially challenging since these samples are both spread over large areas and thick, which requires 3D capture. Here, we introduce a new parallelized microscope for scanning thick specimens across extremely wide fields-of-view (54x72 mm^2) at 1.2 and 0.6 {\mu}m resolutions, accompanied by machine learning software to rapidly assess these 16 gigapixel scans. This Multi-Camera Array Scanner (MCAS) comprises 48 micro-cameras closely arranged to simultaneously image different areas. By capturing 624 megapixels per snapshot, the MCAS is significantly faster than most conventional whole slide scanners. We used this system to digitize entire cytology samples (scanning three entire slides in 3D in just several minutes) and demonstrate two machine learning techniques to assist pathologists: first, an adenocarcinoma detection model in lung specimens (0.73 recall); second, a slide-level classification model of lung smears (0.969 AUC).

Magnetic superconductors manifest a fascinating interplay between their magnetic and superconducting properties. This becomes evident, for example, in the significant enhancement of the upper critical field observed in uranium-based superconductors, or the destruction of superconductivity well below the superconducting transition temperature $T_c$ in cobalt-doped NbSe$_2$. In this work, we argue that the Kondo interaction plays a pivotal role in governing these behaviors. By employing a periodic Anderson model, we study the Kondo effect in superconductors with either singlet or triplet pairing. In the regime of small impurity energies and high doping concentrations, we find the emergence of a Kondo resistive region below $T_c$. While a magnetic field suppresses singlet superconductivity, it stabilizes triplet pairing through the screening of magnetic impurities, inducing reentrant superconductivity at high fields. Moreover, introducing an antisymmetric spin-orbital coupling suppresses triplet superconductivity. This framework provides a unified picture to understand the observation of Kondo effect in NbSe$_2$ as well as the phase diagrams in Kondo superconductors such as UTe$_2$, and URhGe., Comment: 9 pages, 3 figures

Language based editing of 3D human avatars to precisely match user requirements is challenging due to the inherent ambiguity and limited expressiveness of natural language. To overcome this, we propose the Avatar Concept Slider (ACS), a 3D avatar editing method that allows precise manipulation of semantic concepts in human avatars towards a specified intermediate point between two extremes of concepts, akin to moving a knob along a slider track. To achieve this, our ACS has three designs. 1) A Concept Sliding Loss based on Linear Discriminant Analysis to pinpoint the concept-specific axis for precise editing. 2) An Attribute Preserving Loss based on Principal Component Analysis for improved preservation of avatar identity during editing. 3) A 3D Gaussian Splatting primitive selection mechanism based on concept-sensitivity, which updates only the primitives that are the most sensitive to our target concept, to improve efficiency. Results demonstrate that our ACS enables fine-grained 3D avatar editing with efficient feedback, without harming the avatar quality or compromising the avatar's identifying attributes.

Accurate waveform modelling is crucial for parameter estimation in gravitational wave astronomy, impacting our understanding of source properties and the testing of general relativity. The precession of orbital and spin angular momenta in binary black hole (BBH) systems with misaligned spins presents a complex challenge for gravitational waveform modelling. Current precessing BBH waveform models employ a co-precessing frame, which precesses along with the binary. In this paper, we investigate a source of bias stemming from the mismodelling of ringdown frequency in the co-precessing frame. We find that this mismodelling of the co-precessing frame ringdown frequency introduces systematic biases in parameter estimation, for high mass systems in particular, and in the Inspiral-Merger-Ringdown (IMR)-consistency test of general relativity. Employing the waveform model IMRPhenomXPHM, we conduct an IMR-consistency test using a Fisher matrix analysis across parameter space, as well as full injected signal parameter estimation studies. Our results show that this mismodelling particularly affects BBH systems with high mass ratios, high spin magnitudes, and highly inclined spins. These findings suggest inconsistencies for all waveform models which do not address this issue., Comment: 16 pages, 5 figures

Machine unlearning (MU) seeks to remove knowledge of specific data samples from trained models without the necessity for complete retraining, a task made challenging by the dual objectives of effective erasure of data and maintaining the overall performance of the model. Despite recent advances in this field, balancing between the dual objectives of unlearning remains challenging. From a fresh perspective of generalization, we introduce a novel Learning-to-Unlearn (LTU) framework, which adopts a meta-learning approach to optimize the unlearning process to improve forgetting and remembering in a unified manner. LTU includes a meta-optimization scheme that facilitates models to effectively preserve generalizable knowledge with only a small subset of the remaining set, while thoroughly forgetting the specific data samples. We also introduce a Gradient Harmonization strategy to align the optimization trajectories for remembering and forgetting via mitigating gradient conflicts, thus ensuring efficient and effective model updates. Our approach demonstrates improved efficiency and efficacy for MU, offering a promising solution to the challenges of data rights and model reusability., Comment: Accepted by ECCV 2024

Protecting intellectual property (IP) of text such as articles and code is increasingly important, especially as sophisticated attacks become possible, such as paraphrasing by large language models (LLMs) or even unauthorized training of LLMs on copyrighted text to infringe such IP. However, existing text watermarking methods are not robust enough against such attacks nor scalable to millions of users for practical implementation. In this paper, we propose Waterfall, the first training-free framework for robust and scalable text watermarking applicable across multiple text types (e.g., articles, code) and languages supportable by LLMs, for general text and LLM data provenance. Waterfall comprises several key innovations, such as being the first to use LLM as paraphrasers for watermarking along with a novel combination of techniques that are surprisingly effective in achieving robust verifiability and scalability. We empirically demonstrate that Waterfall achieves significantly better scalability, robust verifiability, and computational efficiency compared to SOTA article-text watermarking methods, and showed how it could be directly applied to the watermarking of code. We also demonstrated that Waterfall can be used for LLM data provenance, where the watermarks of LLM training data can be detected in LLM output, allowing for detection of unauthorized use of data for LLM training and potentially enabling model-centric watermarking of open-sourced LLMs which has been a limitation of existing LLM watermarking works. Our code is available at https://github.com/aoi3142/Waterfall., Comment: Accepted to EMNLP 2024 Main Conference

We propose an experimental setup to probe the interplay between the quantum superposition principle and the gravitational time dilation arising from the mass-energy equivalence. It capitalizes on state-of-the-art atom interferometers that can keep atoms trapped in a superposition of heights in Earth's gravitational field for exceedingly long times reaching minute-scale. Our proposal consists of adding two additional laser pulses to the existing setups that would set up a clock trapped at a superposition of heights reading a quantum superposition of relativistic proper times. We develop a method to include relativistic corrections to Bloch oscillations which describe the trapped part of the interferometer. As a result, we derive all the trajectories arising in this setup, the phases acquired by different trajectories in the interferometer, and demonstrate that the effect of superposition of proper times would manifest itself in the interference pattern in two distinct ways: as visibility modulations, and as a shift of the resonant frequency of the atom. We argue that the latter might be observable with current technology., Comment: 5+7 pages, 3 figures

As large language models (LLMs) become increasingly prevalent in a wide variety of applications, concerns about the safety of their outputs have become more significant. Most efforts at safety-tuning or moderation today take on a predominantly Western-centric view of safety, especially for toxic, hateful, or violent speech. In this paper, we describe LionGuard, a Singapore-contextualized moderation classifier that can serve as guardrails against unsafe LLM outputs. When assessed on Singlish data, LionGuard outperforms existing widely-used moderation APIs, which are not finetuned for the Singapore context, by 14% (binary) and up to 51% (multi-label). Our work highlights the benefits of localization for moderation classifiers and presents a practical and scalable approach for low-resource languages., Comment: Preprint

We propose a framework to perform Bayesian inference using conditional score-based diffusion models to solve a class of inverse problems in mechanics involving the inference of a specimen's spatially varying material properties from noisy measurements of its mechanical response to loading. Conditional score-based diffusion models are generative models that learn to approximate the score function of a conditional distribution using samples from the joint distribution. More specifically, the score functions corresponding to multiple realizations of the measurement are approximated using a single neural network, the so-called score network, which is subsequently used to sample the posterior distribution using an appropriate Markov chain Monte Carlo scheme based on Langevin dynamics. Training the score network only requires simulating the forward model. Hence, the proposed approach can accommodate black-box forward models and complex measurement noise. Moreover, once the score network has been trained, it can be re-used to solve the inverse problem for different realizations of the measurements. We demonstrate the efficacy of the proposed approach on a suite of high-dimensional inverse problems in mechanics that involve inferring heterogeneous material properties from noisy measurements. Some examples we consider involve synthetic data, while others include data collected from actual elastography experiments. Further, our applications demonstrate that the proposed approach can handle different measurement modalities, complex patterns in the inferred quantities, non-Gaussian and non-additive noise models, and nonlinear black-box forward models. The results show that the proposed framework can solve large-scale physics-based inverse problems efficiently.

New JWST/NIRCam wide-field slitless spectroscopy provides redshifts for four z>8 galaxies located behind the lensing cluster MACS J0416.1-2403. Two of them, "Y1" and "JD", have previously reported spectroscopic redshifts based on ALMA measurements of [OIII] 88 $\mu$m and/or [CII] 157.7 $\mu$m lines. Y1 is a merging system of three components, and the existing redshift z=8.31 is confirmed. However, JD is at z=8.34 instead of the previously claimed z=9.28. JD's close companion, "JD-N", which was a previously discovered z>8 candidate, is now identified at the same redshift as JD. JD and JD-N form an interacting pair. A new candidate at z>8, "f090d_018", is also confirmed and is at z=8.49. These four objects are likely part of an overdensity that signposts a large structure extending ~165 kpc in projected distance and ~48.7 Mpc in radial distance. They are magnified by less than one magnitude and have intrinsic $M_{UV}$ ranging from -19.57 to -20.83 mag. Their spectral energy distributions show that the galaxies are all very young with ages ~ 4-18 Myr and stellar masses about $10^{7-8}$ ${\rm M_\odot}$. These infant galaxies have very different star formation rates ranging from a few to over a hundred $\rm{M_\odot}$ yr$^{-1}$, but only two of them (JD and f090d_018) have blue rest-frame UV slopes $\beta<-2.0$ indicative of a high Lyman-continuum photon escape fraction that could contribute significantly to the cosmic hydrogen-reionizing background. Interestingly, these two galaxies are the least massive and least active ones among the four. The other two systems have much flatter UV slopes largely because of their high dust extinction ($A_{\rm V}$=0.9-1.0 mag). Their much lower indicated escape fractions show that even very young, actively star-forming galaxies can have negligible contribution to reionization when they quickly form dust throughout their bodies., Comment: 18 pages, 6 figures, after addressing the referee report

The increasing complexity of foundational models underscores the necessity for explainability, particularly for fine-tuning, the most widely used training method for adapting models to downstream tasks. Instance attribution, one type of explanation, attributes the model prediction to each training example by an instance score. However, the robustness of instance scores, specifically towards dataset resampling, has been overlooked. To bridge this gap, we propose a notion of robustness on the sign of the instance score. We theoretically and empirically demonstrate that the popular leave-one-out-based methods lack robustness, while the Shapley value behaves significantly better, but at a higher computational cost. Accordingly, we introduce an efficient fine-tuning-free approximation of the Shapley value (FreeShap) for instance attribution based on the neural tangent kernel. We empirically demonstrate that FreeShap outperforms other methods for instance attribution and other data-centric applications such as data removal, data selection, and wrong label detection, and further generalize our scale to large language models (LLMs). Our code is available at https://github.com/JTWang2000/FreeShap., Comment: Accepted to ICML 2024

Source-free domain adaptation (SFDA) aims to adapt a model pre-trained on a labeled source domain to an unlabeled target domain without access to source data, preserving the source domain's privacy. While SFDA is prevalent in computer vision, it remains largely unexplored in time series analysis. Existing SFDA methods, designed for visual data, struggle to capture the inherent temporal dynamics of time series, hindering adaptation performance. This paper proposes MAsk And imPUte (MAPU), a novel and effective approach for time series SFDA. MAPU addresses the critical challenge of temporal consistency by introducing a novel temporal imputation task. This task involves randomly masking time series signals and leveraging a dedicated temporal imputer to recover the original signal within the learned embedding space, bypassing the complexities of noisy raw data. Notably, MAPU is the first method to explicitly address temporal consistency in the context of time series SFDA. Additionally, it offers seamless integration with existing SFDA methods, providing greater flexibility. We further introduce E-MAPU, which incorporates evidential uncertainty estimation to address the overconfidence issue inherent in softmax predictions. To achieve that, we leverage evidential deep learning to obtain a better-calibrated pre-trained model and adapt the target encoder to map out-of-support target samples to a new feature representation closer to the source domain's support. This fosters better alignment, ultimately enhancing adaptation performance. Extensive experiments on five real-world time series datasets demonstrate that both MAPU and E-MAPU achieve significant performance gains compared to existing methods. These results highlight the effectiveness of our proposed approaches for tackling various time series domain adaptation problems.

Predicting cancer dynamics under treatment is challenging due to high inter-patient heterogeneity, lack of predictive biomarkers, and sparse and noisy longitudinal data. Mathematical models can summarize cancer dynamics by a few interpretable parameters per patient. Machine learning methods can then be trained to predict the model parameters from baseline covariates, but do not account for uncertainty in the parameter estimates. Instead, hierarchical Bayesian modeling can model the relationship between baseline covariates to longitudinal measurements via mechanistic parameters while accounting for uncertainty in every part of the model. The mapping from baseline covariates to model parameters can be modeled in several ways. A linear mapping simplifies inference but fails to capture nonlinear covariate effects and scale poorly for interaction modeling when the number of covariates is large. In contrast, Bayesian neural networks can potentially discover interactions between covariates automatically, but at a substantial cost in computational complexity. In this work, we develop a hierarchical Bayesian model of subpopulation dynamics that uses baseline covariate information to predict cancer dynamics under treatment, inspired by cancer dynamics in multiple myeloma (MM), where serum M protein is a well-known proxy of tumor burden. As a working example, we apply the model to a simulated dataset and compare its ability to predict M protein trajectories to a model with linear covariate effects. Our results show that the Bayesian neural network covariate effect model predicts cancer dynamics more accurately than a linear covariate effect model when covariate interactions are present. The framework can also be applied to other types of cancer or other time series prediction problems that can be described with a parametric model., Comment: 22 pages, 10 figures

Creating and customizing a 3D clothed avatar from textual descriptions is a critical and challenging task. Traditional methods often treat the human body and clothing as inseparable, limiting users' ability to freely mix and match garments. In response to this limitation, we present LAyered Gaussian Avatar (LAGA), a carefully designed framework enabling the creation of high-fidelity decomposable avatars with diverse garments. By decoupling garments from avatar, our framework empowers users to conviniently edit avatars at the garment level. Our approach begins by modeling the avatar using a set of Gaussian points organized in a layered structure, where each layer corresponds to a specific garment or the human body itself. To generate high-quality garments for each layer, we introduce a coarse-to-fine strategy for diverse garment generation and a novel dual-SDS loss function to maintain coherence between the generated garments and avatar components, including the human body and other garments. Moreover, we introduce three regularization losses to guide the movement of Gaussians for garment transfer, allowing garments to be freely transferred to various avatars. Extensive experimentation demonstrates that our approach surpasses existing methods in the generation of 3D clothed humans.

Source-free domain adaptation (SFDA) aims to adapt a source model trained on a fully-labeled source domain to a related but unlabeled target domain. While the source model is a key avenue for acquiring target pseudolabels, the generated pseudolabels may exhibit source bias. In the conventional SFDA pipeline, a large data (e.g. ImageNet) pre-trained feature extractor is used to initialize the source model at the start of source training, and subsequently discarded. Despite having diverse features important for generalization, the pre-trained feature extractor can overfit to the source data distribution during source training and forget relevant target domain knowledge. Rather than discarding this valuable knowledge, we introduce an integrated framework to incorporate pre-trained networks into the target adaptation process. The proposed framework is flexible and allows us to plug modern pre-trained networks into the adaptation process to leverage their stronger representation learning capabilities. For adaptation, we propose the Co-learn algorithm to improve target pseudolabel quality collaboratively through the source model and a pre-trained feature extractor. Building on the recent success of the vision-language model CLIP in zero-shot image recognition, we present an extension Co-learn++ to further incorporate CLIP's zero-shot classification decisions. We evaluate on 4 benchmark datasets and include more challenging scenarios such as open-set, partial-set and open-partial SFDA. Experimental results demonstrate that our proposed strategy improves adaptation performance and can be successfully integrated with existing SFDA methods. Project code is available at https://github.com/zwenyu/colearn-plus., Comment: Extension of ICCV paper arXiv:2212.07585; Published at IJCV

For twisted bilayer graphene close to magic angle, we show that the effects of lattice relaxation and the Hartree interaction both become simultaneously important. Including both effects in a continuum theory reveals a Lifshitz transition to a Fermi surface topology that supports both a ``heavy fermion" pocket and an ultraflat band ($\approx 8~{\rm meV}$) that is pinned to the Fermi energy for a large range of fillings. We provide analytical and numerical results to understand the narrow ``magic angle range" that supports this pinned ultraflat band and make predictions for its experimental observation. We believe that the bands presented here are accurate at high temperature and provide a good starting point to understand the myriad of complex behaviour observed in this system.

In this paper, we address the challenge of reconstructing general articulated 3D objects from a single video. Existing works employing dynamic neural radiance fields have advanced the modeling of articulated objects like humans and animals from videos, but face challenges with piece-wise rigid general articulated objects due to limitations in their deformation models. To tackle this, we propose Quasi-Rigid Blend Skinning, a novel deformation model that enhances the rigidity of each part while maintaining flexible deformation of the joints. Our primary insight combines three distinct approaches: 1) an enhanced bone rigging system for improved component modeling, 2) the use of quasi-sparse skinning weights to boost part rigidity and reconstruction fidelity, and 3) the application of geodesic point assignment for precise motion and seamless deformation. Our method outperforms previous works in producing higher-fidelity 3D reconstructions of general articulated objects, as demonstrated on both real and synthetic datasets. Project page: https://chaoyuesong.github.io/REACTO.

We present a new parametric lens model for the G165.7+67.0 galaxy cluster, which was discovered with $Planck$ through its bright submillimeter flux, originating from a pair of extraordinary dusty star-forming galaxies (DSFGs) at $z\approx 2.2$. Using JWST and interferometric mm/radio observations, we characterize the intrinsic physical properties of the DSFGs, which are separated by only $\sim 1^{\prime\prime}$ (8 kpc) and a velocity difference $\Delta V \lesssim 600~{\rm km}~{\rm s}^{-1}$ in the source plane, and thus likely undergoing a major merger. Boasting intrinsic star formation rates ${\rm SFR}_{\rm IR} = 320 \pm 70$ and $400 \pm 80~ M_\odot~{\rm yr}^{-1}$, stellar masses ${\rm log}[M_\star/M_\odot] = 10.2 \pm 0.1$ and $10.3 \pm 0.1$, and dust attenuations $A_V = 1.5 \pm 0.3$ and $1.2 \pm 0.3$, they are remarkably similar objects. We perform spatially-resolved pixel-by-pixel SED fitting using rest-frame near-UV to near-IR imaging from JWST/NIRCam for both galaxies, resolving some stellar structures down to 100 pc scales. Based on their resolved specific SFRs and $UVJ$ colors, both DSFGs are experiencing significant galaxy-scale star formation events. If they are indeed interacting gravitationally, this strong starburst could be the hallmark of gas that has been disrupted by an initial close passage. In contrast, the host galaxy of the recently discovered triply-imaged SN H0pe has a much lower SFR than the DSFGs, and we present evidence for the onset of inside-out quenching and large column densities of dust even in regions of low specific SFR. Based on the intrinsic SFRs of the DSFGs inferred from UV through FIR SED modeling, this pair of objects alone is predicted to yield an observable $1.1 \pm 0.2~{\rm CCSNe~yr}^{-1}$, making this cluster field ripe for continued monitoring., Comment: 47 pages, 21 figures, 5 tables. Submitted to ApJ, comments welcome!

A bright ($m_{\rm F150W,AB}$=24 mag), $z=1.95$ supernova (SN) candidate was discovered in JWST/NIRCam imaging acquired on 2023 November 17. The SN is quintuply-imaged as a result of strong gravitational lensing by a foreground galaxy cluster, detected in three locations, and remarkably is the second lensed SN found in the same host galaxy. The previous lensed SN was called "Requiem", and therefore the new SN is named "Encore". This makes the MACS J0138.0$-$2155 cluster the first known system to produce more than one multiply-imaged SN. Moreover, both SN Requiem and SN Encore are Type Ia SNe (SNe Ia), making this the most distant case of a galaxy hosting two SNe Ia. Using parametric host fitting, we determine the probability of detecting two SNe Ia in this host galaxy over a $\sim10$ year window to be $\approx3\%$. These observations have the potential to yield a Hubble Constant ($H_0$) measurement with $\sim10\%$ precision, only the third lensed SN capable of such a result, using the three visible images of the SN. Both SN Requiem and SN Encore have a fourth image that is expected to appear within a few years of $\sim2030$, providing an unprecedented baseline for time-delay cosmography., Comment: Accepted, ApJL

Action detection aims to localize the starting and ending points of action instances in untrimmed videos, and predict the classes of those instances. In this paper, we make the observation that the outputs of the action detection task can be formulated as images. Thus, from a novel perspective, we tackle action detection via a three-image generation process to generate starting point, ending point and action-class predictions as images via our proposed Action Detection Image Diffusion (ADI-Diff) framework. Furthermore, since our images differ from natural images and exhibit special properties, we further explore a Discrete Action-Detection Diffusion Process and a Row-Column Transformer design to better handle their processing. Our ADI-Diff framework achieves state-of-the-art results on two widely-used datasets., Comment: Accepted to CVPR 2024

Sign Language Translation (SLT) is a challenging task that aims to translate sign videos into spoken language. Inspired by the strong translation capabilities of large language models (LLMs) that are trained on extensive multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT. In this paper, we regularize the sign videos to embody linguistic characteristics of spoken language, and propose a novel SignLLM framework to transform sign videos into a language-like representation for improved readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The Vector-Quantized Visual Sign module converts sign videos into a sequence of discrete character-level sign tokens, and (2) the Codebook Reconstruction and Alignment module converts these character-level tokens into word-level sign representations using an optimal transport formulation. A sign-text alignment loss further bridges the gap between sign and text tokens, enhancing semantic compatibility. We achieve state-of-the-art gloss-free results on two widely-used SLT benchmarks., Comment: Accepted to CVPR 2024

Language models (LMs) are indispensable tools for natural language processing tasks, but their vulnerability to adversarial attacks remains a concern. While current research has explored adversarial training techniques, their improvements to defend against word-level attacks have been limited. In this work, we propose a novel approach called Semantic Robust Defence (SemRoDe), a Macro Adversarial Training strategy to enhance the robustness of LMs. Drawing inspiration from recent studies in the image domain, we investigate and later confirm that in a discrete data setting such as language, adversarial samples generated via word substitutions do indeed belong to an adversarial domain exhibiting a high Wasserstein distance from the base domain. Our method learns a robust representation that bridges these two domains. We hypothesize that if samples were not projected into an adversarial domain, but instead to a domain with minimal shift, it would improve attack robustness. We align the domains by incorporating a new distance-based objective. With this, our model is able to learn more generalized representations by aligning the model's high-level output features and therefore better handling unseen adversarial samples. This method can be generalized across word embeddings, even when they share minimal overlap at both vocabulary and word-substitution levels. To evaluate the effectiveness of our approach, we conduct experiments on BERT and RoBERTa models on three datasets. The results demonstrate promising state-of-the-art robustness., Comment: Published in NAACL 2024 (Main Track)

Supernova (SN) H0pe is a gravitationally lensed, triply-imaged, Type Ia SN (SN Ia) discovered in James Webb Space Telescope imaging of the PLCK G165.7+67.0 cluster of galaxies. Well-observed multiply-imaged SNe provide a rare opportunity to constrain the Hubble constant ($H_0$), by measuring the relative time delay between the images and modeling the foreground mass distribution. SN H0pe is located at $z=1.783$, and is the first SN Ia with sufficient light curve sampling and long enough time delays for an $H_0$ inference. Here we present photometric time-delay measurements and SN properties of SN H0pe. Using JWST/NIRCam photometry we measure time delays of $\Delta t_{ab}=-116.6^{+10.8}_{-9.3}$ and $\Delta t_{cb}=-48.6^{+3.6}_{-4.0}$ observer-frame days relative to the last image to arrive (image 2b; all uncertainties are $1\sigma$), which corresponds to a $\sim5.6\%$ uncertainty contribution for $H_0$ assuming $70 \rm{km s^{-1} Mpc^{-1}}$. We also constrain the absolute magnification of each image to $\mu_{a}=4.3^{+1.6}_{-1.8}$, $\mu_{b}=7.6^{+3.6}_{-2.6}$, $\mu_{c}=6.4^{+1.6}_{-1.5}$ by comparing the observed peak near-IR magnitude of SN H0pe to the non-lensed population of SNe Ia., Comment: Accepted in ApJ

The first James Webb Space Telescope ({\it JWST}) Near InfraRed Camera (NIRCam) imaging in the field of the galaxy cluster PLCK G165.7+67.0 ($z=0.35$) uncovered a Type Ia supernova (SN Ia) at $z=1.78$, called ``SN H0pe." Three different images of this one SN were detected as a result of strong gravitational lensing, each one traversing a different path in spacetime, thereby inducing a relative delay in the arrival of each image. Follow-up {\it JWST} observations of all three SN images enabled photometric and rare spectroscopic measurements of the two relative time delays. Following strict blinding protocols which oversaw a live unblinding and regulated post-unblinding changes, these two measured time delays were compared to the predictions of seven independently constructed cluster lens models to measure a value for the Hubble constant, $H_0=71.8^{+9.2}_{-8.1}$ km s$^{-1}$ Mpc$^{-1}$. The range of admissible $H_0$ values predicted across the lens models limits further precision, reflecting the well-known degeneracies between lens model constraints and time delays. It has long been theorized that a way forward is to leverage a standard candle, but this has not been realized until now. For the first time, the lens models are evaluated by their agreement with the SN absolute magnifications, breaking degeneracies and producing our best estimate, $H_0=75.7^{+8.1}_{-5.5}$ km s$^{-1}$ Mpc$^{-1}$. This is the first precision measurement of $H_0$ from a multiply-imaged SN Ia and only the second from any multiply-imaged SN., Comment: Accepted to ApJ. 27 pages, 7 Figures

Domain adaptation is a critical task in machine learning that aims to improve model performance on a target domain by leveraging knowledge from a related source domain. In this work, we introduce Universal Semi-Supervised Domain Adaptation (UniSSDA), a practical yet challenging setting where the target domain is partially labeled, and the source and target label space may not strictly match. UniSSDA is at the intersection of Universal Domain Adaptation (UniDA) and Semi-Supervised Domain Adaptation (SSDA): the UniDA setting does not allow for fine-grained categorization of target private classes not represented in the source domain, while SSDA focuses on the restricted closed-set setting where source and target label spaces match exactly. Existing UniDA and SSDA methods are susceptible to common-class bias in UniSSDA settings, where models overfit to data distributions of classes common to both domains at the expense of private classes. We propose a new prior-guided pseudo-label refinement strategy to reduce the reinforcement of common-class bias due to pseudo-labeling, a common label propagation strategy in domain adaptation. We demonstrate the effectiveness of the proposed strategy on benchmark datasets Office-Home, DomainNet, and VisDA. The proposed strategy attains the best performance across UniSSDA adaptation settings and establishes a new baseline for UniSSDA., Comment: Accepted to CVPR 2024

Recent works on text-to-3d generation show that using only 2D diffusion supervision for 3D generation tends to produce results with inconsistent appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals with extra legs). Existing methods mainly address this issue by retraining diffusion models with images rendered from 3D data to ensure multi-view consistency while struggling to balance 2D generation quality with 3D consistency. In this paper, we present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. Moreover, to ensure accurate appearances of different views, we further modulate the output of the 2D diffusion model to the correct patterns of the template views without altering the generated object's style. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model. Extensive experiments show our method can largely improve the multi-view consistency while retaining fidelity and diversity. Our project page is available at: https://stellarcheng.github.io/Sculpt3D/., Comment: Accepted by CVPR 2024. Project Page: https://stellarcheng.github.io/Sculpt3D/