Search

We develop theoretical foundations for widely used falsification tests for instrumental variable (IV) designs. We characterize these tests as conditional independence tests between negative control variables - proxies for potential threats - and either the IV or the outcome. We find that conventional applications of these falsification tests would flag problems in exogenous IV designs, and propose simple solutions to avoid this. We also propose new falsification tests that incorporate new types of negative control variables or alternative statistical tests. Finally, we illustrate that under stronger assumptions, negative control variables can also be used for bias correction.

A growing number of products use layer 2 solutions to expand the capabilities of primary blockchains like Ethereum, where computation is off-loaded from the root chain, and the results are published to it in bulk. Those include optimistic and zero-knowledge rollups, information oracles, and app-specific chains. This work presents an analysis of layer 2 blockchain strategies determining the optimal times for publishing transactions on the root chain. There is a trade-off between waiting for a better layer 1 gas price and the urgency to finalize layer 2 transactions. We present a model for the problem that captures this trade-off, generalizing previous works, and we analyze the properties of optimal publishing strategies. We show that such optimal strategies hold a computable simple form for a large class of cost functions., Comment: To be presented at Financial Cryptography and Data Security 2024 (FC 2024)

In-context learning allows adapting a model to new tasks given a task description at test time. In this paper, we present IMProv - a generative model that is able to in-context learn visual tasks from multimodal prompts. Given a textual description of a visual task (e.g. "Left: input image, Right: foreground segmentation"), a few input-output visual examples, or both, the model in-context learns to solve it for a new test input. We train a masked generative transformer on a new dataset of figures from computer vision papers and their associated captions, together with a captioned large-scale image-text dataset. During inference time, we prompt the model with text and/or image task example(s) and have the model inpaint the corresponding output. We show that training our model with text conditioning and scaling the dataset size improves in-context learning for computer vision tasks by over +10\% AP for Foreground Segmentation, over +5\% gains in AP for Single Object Detection, and almost 20\% lower LPIPS in Colorization. Our empirical results suggest that vision and language prompts are complementary and it is advantageous to use both to achieve better in-context learning performance. Project page is available at https://jerryxu.net/IMProv ., Comment: Project page: https://jerryxu.net/IMProv

We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources such as semantic segmentations and depth reconstructions without needing any meta-knowledge beyond the pixels. Once this wide variety of visual data (comprising 420 billion tokens) is represented as sequences, the model can be trained to minimize a cross-entropy loss for next token prediction. By training across various scales of model architecture and data diversity, we provide empirical evidence that our models scale effectively. Many different vision tasks can be solved by designing suitable visual prompts at test time., Comment: Website: https://yutongbai.com/lvm.html

2D materials are emerging as promising nanomaterials for applications in energy storage and catalysis. In the wet chemical synthesis of MXenes, these 2D transition metal carbides and nitrides are terminated with a variety of functional groups, and cations such as Li+ are often used to intercalate into the structure to obtain exfoliated nanosheets. Given the various elements involved in their synthesis, it is crucial to determine the detailed chemical composition of the final product, in order to better assess and understand the relationships between composition and properties of these materials. To facilitate atom probe tomography analysis of these materials, a revised specimen preparation method is presented in this study. A colloidal Ti3C2Tz MXene solution was processed into an additive-free free-standing film and specimens were prepared using a dual beam scanning electron microscope / focused ion beam. To mechanically stabilize the fragile specimens, they were coated using an in situ sputtering technique. As various 2D material inks can be processed into such free-standing films, the presented approach is pivotal for enabling atom probe analysis of other 2D materials.

We present a new method for text-driven motion transfer - synthesizing a video that complies with an input text prompt describing the target objects and scene while maintaining an input video's motion and scene layout. Prior methods are confined to transferring motion across two subjects within the same or closely related object categories and are applicable for limited domains (e.g., humans). In this work, we consider a significantly more challenging setting in which the target and source objects differ drastically in shape and fine-grained motion characteristics (e.g., translating a jumping dog into a dolphin). To this end, we leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors. The pillar of our method is a new space-time feature loss derived directly from the model. This loss guides the generation process to preserve the overall motion of the input video while complying with the target object in terms of shape and fine-grained motion traits., Comment: Project page: https://diffusion-motion-transfer.github.io/

We study the time-reflection and time-refraction of waves caused by a spatial interface with a medium undergoing a sudden temporal change in permittivity. We show that monochromatic waves are transformed into a pulse by the permittivity change, and that time-reflection is enhanced at the vicinity of the critical angle for total internal reflection. In this regime, we find that the evanescent field is transformed into a propagating pulse by the sudden change in permittivity. These effects display enhancement of the time-reflection and high sensitivity near the critical angle, paving the way to experiments on time-reflection and photonic time-crystals at optical frequencies

We introduce the class of \textit{Generalized Poincar\'{e}-Duality groups}: i.e, pro-$p$ groups of infinite rank which satisfy a Poincar\'{e}-duality. We prove some basic properties of Generalized Poincar\'{e}-Duality groups, and show that under some conditions, a group that fits into an exact sequence of GPD groups must be a Demushkin group.

We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image. To integrate semantic information into our framework, our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model. Specifically, we derive novel disentangled representations of structure and appearance extracted from deep ViT features. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Based on our objective function, we propose two frameworks of semantic appearance transfer -- "Splice", which works by training a generator on a single and arbitrary pair of structure-appearance images, and "SpliceNet", a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain. Our frameworks do not involve adversarial training, nor do they require any additional input information such as semantic segmentation or correspondences. We demonstrate high-resolution results on a variety of in-the-wild image pairs, under significant variations in the number of objects, pose, and appearance. Code and supplementary material are available in our project page: splice-vit.github.io., Comment: Accepted to ACM Transactions on Graphics. arXiv admin note: substantial text overlap with arXiv:2201.00424

Various NLP tasks require a complex hierarchical structure over nodes, where each node is a cluster of items. Examples include generating entailment graphs, hierarchical cross-document coreference resolution, annotating event and subevent relations, etc. To enable efficient annotation of such hierarchical structures, we release CHAMP, an open source tool allowing to incrementally construct both clusters and hierarchy simultaneously over any type of texts. This incremental approach significantly reduces annotation time compared to the common pairwise annotation approach and also guarantees maintaining transitivity at the cluster and hierarchy levels. Furthermore, CHAMP includes a consolidation mode, where an adjudicator can easily compare multiple cluster hierarchy annotations and resolve disagreements., Comment: EMNLP 2023

We study bulk particle transport in a Fermi-Hubbard model on an infinite-dimensional Bethe lattice, driven by a constant electric field. Previous numerical studies showed that one dimensional analogs of this system exhibit a breakdown of diffusion due to Stark many-body localization (Stark-MBL) at least up to time which scales exponentially with the system size. Here, we consider systems initially in a spin density wave state using a combination of numerically exact and approximate techniques. We show that for sufficiently weak electric fields, the wave's momentum component decays exponentially with time in a way consistent with normal diffusion. By studying different wavelengths, we extract the dynamical exponent and the generalized diffusion coefficient at each field strength. Interestingly, we find a non-monotonic dependence of the dynamical exponent on the electric field. As the field increases towards a critical value proportional to the Hubbard interaction strength, transport slows down, becoming sub-diffusive. At large interaction strengths, however, transport speeds up again with increasing field, exhibiting super-diffusive characteristics when the electric field is comparable to the interaction strength. Eventually, at the large field limit, localization occurs and the current through the system is suppressed.

In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. A recent line of work suggests that ICL performs gradient descent (GD)-based optimization implicitly. While appealing, much of the research focuses on simplified settings, where the parameters of a shallow model are optimized. In this work, we revisit evidence for ICL-GD correspondence on realistic NLP tasks and models. We find gaps in evaluation, both in terms of problematic metrics and insufficient baselines. We show that surprisingly, even untrained models achieve comparable ICL-GD similarity scores despite not exhibiting ICL. Next, we explore a major discrepancy in the flow of information throughout the model between ICL and GD, which we term Layer Causality. We propose a simple GD-based optimization procedure that respects layer causality, and show it improves similarity scores significantly., Comment: Accepted to NAACL 2024 main conference

The Thurston metric on Teichmuller space, first introduced by W. P. Thurston is an asymmetric metric on Teichmuller space defined by $d_{Th}(X,Y) = \frac12 log\sup_{\alpha} \frac{l_{\alpha}(Y)}{l_{\alpha}(X)}$. This metric is geodesic, but geodesics are far from unique. In this thesis, we show that in the once-punctured torus, and in the four-times punctured sphere, geodesics stay a uniformly-bounded distance from each other. In other words, we show that the \textbf{width} of the geodesic envelope, E(X,Y) between any pair of points $X,Y \in T(S)$ (where $S = S_{1,1}$ or $S = S_{0,4}$) is bounded uniformly. To do this, we first identify extremal geodesics in Env(X,Y), and show that these correspond to stretch vectors, proving a conjecture of Huang, Ohshika and Papadopoulos. We then compute Fenchel-Nielsen twisting along these paths, and use these computations, along with estimates on earthquake path lengths, to prove the main theorem., Comment: 26 pages, 7.5 figures. Comments and criticism welcome. This is a no-frills version of my PhD thesis :)

Deepfake attacks, malicious manipulation of media containing people, are a serious concern for society. Conventional deepfake detection methods train supervised classifiers to distinguish real media from previously encountered deepfakes. Such techniques can only detect deepfakes similar to those previously seen, but not zero-day (previously unseen) attack types. As current deepfake generation techniques are changing at a breathtaking pace, new attack types are proposed frequently, making this a major issue. Our main observations are that: i) in many effective deepfake attacks, the fake media must be accompanied by false facts i.e. claims about the identity, speech, motion, or appearance of the person. For instance, when impersonating Obama, the attacker explicitly or implicitly claims that the fake media show Obama; ii) current generative techniques cannot perfectly synthesize the false facts claimed by the attacker. We therefore introduce the concept of "fact checking", adapted from fake news detection, for detecting zero-day deepfake attacks. Fact checking verifies that the claimed facts (e.g. identity is Obama), agree with the observed media (e.g. is the face really Obama's?), and thus can differentiate between real and fake media. Consequently, we introduce FACTOR, a practical recipe for deepfake fact checking and demonstrate its power in critical attack settings: face swapping and audio-visual synthesis. Although it is training-free, relies exclusively on off-the-shelf features, is very easy to implement, and does not see any deepfakes, it achieves better than state-of-the-art accuracy., Comment: Our code is available at https://github.com/talreiss/FACTOR

Despite tremendous advances in AI, it remains a significant challenge to develop interactive task guidance systems that can offer situated, personalized guidance and assist humans in various tasks. These systems need to have a sophisticated understanding of the user as well as the environment, and make timely accurate decisions on when and what to say. To address this issue, we created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We further proposed two tasks: User and Environment Understanding, and Instructor Decision Making. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance. Our quantitative, qualitative, and human evaluation results show that these models can demonstrate fair performances in some cases with no task-specific training, but a fast and reliable adaptation remains a significant challenge. Our benchmark and baselines will provide a stepping stone for future work on situated task guidance., Comment: Accepted to EMNLP 2023 Findings

The algebra of supernatural matrices is a key example in the theory of locally finite central simple algebras, which studied in a previous paper of the authors (\cite{Local}). It is also a stand-alone object admits a rich study and various connections to other fields. The goal of this paper is to expose some new information about supernatural matrices, mainly in terms of the "inner" ways to identify such algebras, and their appearance as minimal solutions to equations of the form $M_n(A)\cong A$. Viewing a natural representation of this algebra, we show that supernatural matrices generalize both McCrimmon's deep matrices algebra and $m$-petal Leavitt path algebra. We also study their simple representations.

We say that a group $G$ is of \textit{profinite type} if it can be realized as a Galois group of some field extension. Using Krull's theory, this is equivalent to the ability of $G$ to be equipped with a profinite topology. We also say that a group of profinite type is \textit{profinitely rigid} if it admits a unique profinite topology. In this paper we study when abelian groups and some group extensions are of profinite type or profinitely rigid. We also discuss the connection between the properties of profinite type and profinite rigidity to the injectivity and surjectivity of the cohomology comparison maps, which were studied by Sury and other authors.

Grokking is the intriguing phenomenon where a model learns to generalize long after it has fit the training data. We show both analytically and numerically that grokking can surprisingly occur in linear networks performing linear tasks in a simple teacher-student setup with Gaussian inputs. In this setting, the full training dynamics is derived in terms of the training and generalization data covariance matrix. We present exact predictions on how the grokking time depends on input and output dimensionality, train sample size, regularization, and network initialization. We demonstrate that the sharp increase in generalization accuracy may not imply a transition from "memorization" to "understanding", but can simply be an artifact of the accuracy measure. We provide empirical verification for our calculations, along with preliminary results indicating that some predictions also hold for deeper networks, with non-linear activations., Comment: 17 pages, 6 figures

The COVID-19 pandemic has heightened the urgency to understand and prevent pathogen transmission, specifically regarding infectious airborne particles. Extensive studies validate the understanding of larger (droplets) and smaller (aerosols) particles in disease transmission. Similarly, N95 respirators, and other forms of respiratory protection, have proven efficacy in reducing the risk of infection across various environments. Even though multiple studies confirm their protective effect when adopted in healthcare and public settings for infection prevention, studies on their adoption over the last several decades in both clinical trials and observational studies have not provided as clear an understanding. Here we show that the standard analytical equations used in the analysis of these studies do not accurately represent the random variables impacting study results. By correcting these equations, it is demonstrated that conclusions drawn from these studies are heavily biased and uncertain, providing little useful information. Despite these limitations, we show that when outcome measures are properly analyzed, existing results consistently point to the benefit of N95 respirators over medical masks, and masking over its absence. Correcting errors in widely reported meta-analyses also yields statistically significant estimates. These findings have important implications for study design and using existing evidence for infection control policy guidelines., Comment: 23 pages, 8 figures

Evasion Attacks (EA) are used to test the robustness of trained neural networks by distorting input data to misguide the model into incorrect classifications. Creating these attacks is a challenging task, especially with the ever-increasing complexity of models and datasets. In this work, we introduce a self-supervised, computationally economical method for generating adversarial examples, designed for the unseen black-box setting. Adapting techniques from representation learning, our method generates on-manifold EAs that are encouraged to resemble the data distribution. These attacks are comparable in effectiveness compared to the state-of-the-art when attacking the model trained on, but are significantly more effective when attacking unseen models, as the attacks are more related to the data rather than the model itself. Our experiments consistently demonstrate the method is effective across various models, unseen data categories, and even defended models, suggesting a significant role for on-manifold EAs when targeting unseen models., Comment: ICCV 2023, AROW Workshop