Search

Image Quality of MRI brain scans is strongly influenced by within scanner head movements and the resulting image artifacts alter derived measures like brain volume and cortical thickness. Automated image quality assessment is key to controlling for confounding effects of poor image quality. In this study, we systematically test for the influence of image quality on univariate statistics and machine learning classification. We analyzed group effects of sex/gender on local brain volume and made predictions of sex/gender using logistic regression, while correcting for brain size. From three large publicly available datasets, two age and sex-balanced samples were derived to test the generalizability of the effect for pooled sample sizes of n=760 and n=1094. Results of the Bonferroni corrected t-tests over 3747 gray matter features showed a strong influence of low-quality data on the ability to find significant sex/gender differences for the smaller sample. Increasing sample size and more so image quality showed a stark increase in detecting significant effects in univariate group comparisons. For the classification of sex/gender using logistic regression, both increasing sample size and image quality had a marginal effect on the Area under the Receiver Operating Characteristic Curve for most datasets and subsamples. Our results suggest a more stringent quality control for univariate approaches than for multivariate classification with a leaning towards higher quality for classical group statistics and bigger sample sizes for machine learning applications in neuroimaging.

Machine learning (ML) models benefit from large datasets. Collecting data in biomedical domains is costly and challenging, hence, combining datasets has become a common practice. However, datasets obtained under different conditions could present undesired site-specific variability. Data harmonization methods aim to remove site-specific variance while retaining biologically relevant information. This study evaluates the effectiveness of popularly used ComBat-based methods for harmonizing data in scenarios where the class balance is not equal across sites. We find that these methods struggle with data leakage issues. To overcome this problem, we propose a novel approach PrettYharmonize, designed to harmonize data by pretending the target labels. We validate our approach using controlled datasets designed to benchmark the utility of harmonization. Finally, using real-world MRI and clinical data, we compare leakage-prone methods with PrettYharmonize and show that it achieves comparable performance while avoiding data leakage, particularly in site-target-dependence scenarios.

We employ new tools from mechanistic interpretability in order to ask whether the internal structure of large language models (LLMs) shows correspondence to the linguistic structures which underlie the languages on which they are trained. In particular, we ask (1) when two languages employ the same morphosyntactic processes, do LLMs handle them using shared internal circuitry? and (2) when two languages require different morphosyntactic processes, do LLMs handle them using different internal circuitry? Using English and Chinese multilingual and monolingual models, we analyze the internal circuitry involved in two tasks. We find evidence that models employ the same circuit to handle the same syntactic process independently of the language in which it occurs, and that this is the case even for monolingual models trained completely independently. Moreover, we show that multilingual models employ language-specific components (attention heads and feed-forward networks) when needed to handle linguistic processes (e.g., morphological marking) that only exist in some languages. Together, our results provide new insights into how LLMs trade off between exploiting common structures and preserving linguistic differences when tasked with modeling multiple languages simultaneously.

Prognostic and diagnostic AI-based medical devices hold immense promise for advancing healthcare, yet their rapid development has outpaced the establishment of appropriate validation methods. Existing approaches often fall short in addressing the complexity of practically deploying these devices and ensuring their effective, continued operation in real-world settings. Building on recent discussions around the validation of AI models in medicine and drawing from validation practices in other fields, a framework to address this gap is presented. It offers a structured, robust approach to validation that helps ensure device reliability across differing clinical environments. The primary challenges to device performance upon deployment are discussed while highlighting the impact of changes related to individual healthcare institutions and operational processes. The presented framework emphasizes the importance of repeating validation and fine-tuning during deployment, aiming to mitigate these issues while being adaptable to challenges unforeseen during device development. The framework is also positioned within the current US and EU regulatory landscapes, underscoring its practical viability and relevance considering regulatory requirements. Additionally, a practical example demonstrating potential benefits of the framework is presented. Lastly, guidance on assessing model performance is offered and the importance of involving clinical stakeholders in the validation and fine-tuning process is discussed., Comment: 13 pages, 1 figure

AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model training is driven by the emergence of generative AI and foundational models, where on occasion thousands of GPUs must cooperate on a single training job for the model to be trained in a reasonable time. Delivering efficient and high-performing AI training requires an end-to-end solution that combines hardware, software and holistic telemetry to cater for multiple types of AI workloads. In this report, we describe IBM's hybrid cloud infrastructure that powers our generative AI model development. This infrastructure includes (1) Vela: an AI-optimized supercomputing capability directly integrated into the IBM Cloud, delivering scalable, dynamic, multi-tenant and geographically distributed infrastructure for large-scale model training and other AI workflow steps and (2) Blue Vela: a large-scale, purpose-built, on-premises hosting environment that is optimized to support our largest and most ambitious AI model training tasks. Vela provides IBM with the dual benefit of high performance for internal use along with the flexibility to adapt to an evolving commercial landscape. Blue Vela provides us with the benefits of rapid development of our largest and most ambitious models, as well as future-proofing against the evolving model landscape in the industry. Taken together, they provide IBM with the ability to rapidly innovate in the development of both AI models and commercial offerings., Comment: Corresponding Authors: Talia Gershon, Seetharami Seelam,Brian Belgodere, Milton Bonilla

Vision-Language Models (VLMs) have gained community-spanning prominence due to their ability to integrate visual and textual inputs to perform complex tasks. Despite their success, the internal decision-making processes of these models remain opaque, posing challenges in high-stakes applications. To address this, we introduce NOTICE, the first Noise-free Text-Image Corruption and Evaluation pipeline for mechanistic interpretability in VLMs. NOTICE incorporates a Semantic Minimal Pairs (SMP) framework for image corruption and Symmetric Token Replacement (STR) for text. This approach enables semantically meaningful causal mediation analysis for both modalities, providing a robust method for analyzing multimodal integration within models like BLIP. Our experiments on the SVO-Probes, MIT-States, and Facial Expression Recognition datasets reveal crucial insights into VLM decision-making, identifying the significant role of middle-layer cross-attention heads. Further, we uncover a set of ``universal cross-attention heads'' that consistently contribute across tasks and modalities, each performing distinct functions such as implicit image segmentation, object inhibition, and outlier inhibition. This work paves the way for more transparent and interpretable multimodal systems.

Zero-shot text learning enables text classifiers to handle unseen classes efficiently, alleviating the need for task-specific training data. A simple approach often relies on comparing embeddings of query (text) to those of potential classes. However, the embeddings of a simple query sometimes lack rich contextual information, which hinders the classification performance. Traditionally, this has been addressed by improving the embedding model with expensive training. We introduce QZero, a novel training-free knowledge augmentation approach that reformulates queries by retrieving supporting categories from Wikipedia to improve zero-shot text classification performance. Our experiments across six diverse datasets demonstrate that QZero enhances performance for state-of-the-art static and contextual embedding models without the need for retraining. Notably, in News and medical topic classification tasks, QZero improves the performance of even the largest OpenAI embedding model by at least 5% and 3%, respectively. Acting as a knowledge amplifier, QZero enables small word embedding models to achieve performance levels comparable to those of larger contextual models, offering the potential for significant computational savings. Additionally, QZero offers meaningful insights that illuminate query context and verify topic relevance, aiding in understanding model predictions. Overall, QZero improves embedding-based zero-shot classifiers while maintaining their simplicity. This makes it particularly valuable for resource-constrained environments and domains with constantly evolving information., Comment: Proceedings of the 2024 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR '24), July 13, 2024, Washington DC, DC, USA

Although it is known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and routed by the model. We analyze a mechanism used in two LMs to selectively inhibit items in a context in one task, and find that it underlies a commonly used abstraction across many context-retrieval behaviors. Specifically, we find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers, forming low-rank communication channels (Elhage et al., 2021) between layers. A particular 3D subspace in model activations in GPT-2 can be traversed to positionally index items in lists, and we show that this mechanism can explain an otherwise arbitrary-seeming sensitivity of the model to the order of items in the prompt. That is, the model has trouble copying the correct information from context when many items ``crowd" this limited space. By decomposing attention heads with the Singular Value Decomposition (SVD), we find that previously described interactions between heads separated by one or more layers can be predicted via analysis of their weight matrices alone. We show that it is possible to manipulate the internal model representations as well as edit model weights based on the mechanism we discover in order to significantly improve performance on our synthetic Laundry List task, which requires recall from a list, often improving task accuracy by over 20%. Our analysis reveals a surprisingly intricate interpretable structure learned from language model pretraining, and helps us understand why sophisticated LMs sometimes fail in simple domains, facilitating future analysis of more complex behaviors., Comment: Neurips 2024

Neural models have demonstrated remarkable performance across diverse ranking tasks. However, the processes and internal mechanisms along which they determine relevance are still largely unknown. Existing approaches for analyzing neural ranker behavior with respect to IR properties rely either on assessing overall model behavior or employing probing methods that may offer an incomplete understanding of causal mechanisms. To provide a more granular understanding of internal model decision-making processes, we propose the use of causal interventions to reverse engineer neural rankers, and demonstrate how mechanistic interpretability methods can be used to isolate components satisfying term-frequency axioms within a ranking model. We identify a group of attention heads that detect duplicate tokens in earlier layers of the model, then communicate with downstream heads to compute overall document relevance. More generally, we propose that this style of mechanistic analysis opens up avenues for reverse engineering the processes neural retrieval models use to compute relevance. This work aims to initiate granular interpretability efforts that will not only benefit retrieval model development and training, but ultimately ensure safer deployment of these models., Comment: 10 pages, 10 figures, accepted at SIGIR 2024 as perspective paper

In this paper we consider a multi band extension to the periodic Anderson model. We use a single site DMFT(NRG) in order to study the impact of the conduction band mediated effective hopping of the correlated electrons between the correlated orbitals onto the heavy Fermi liquid formation. Whereas the hybridization of a single impurity model with two distinct conduction bands always adds up constructively, $T_{K}\propto \exp(-\mathrm{const}\, U/(\Gamma_1+\Gamma_2))$, we show that this does not have to be the case in lattice models, where, in remarkable contrast, also an low-energy Fermi liquid scale $T_0\propto \exp(-\mathrm{const}\, U/(\Gamma_1-\Gamma_2))$ can emerge due to quantum interference effects in multi band models, where $U$ denotes the local Coulomb matrix element of the correlated orbitals and $\Gamma_i$ the local hybridization strength of band $i$. At high symmetry points, heavy Fermi liquid formation is suppressed which is associated with a breakdown of the Kondo effect. This results in an asymptotically scale-invariant (i.e., power-law) spectrum of the correlated orbitals $\propto|\omega|^{1/3}$, indicating non-Fermi liquid properties of the quantum critical point, and a small Fermi surface including only the light quasi-particles. This orbital selective Mott phase demonstrates the possibility of metallic local criticality within the general framework of ordinary single site DMFT.

The growth of $\alpha$-Ga$_2$O$_3$ and $\alpha$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ on $m$-plane $\alpha$-Al$_2$O$_3$(10$\bar{1}$0) by molecular beam epitaxy (MBE) and metal-oxide-catalyzed epitaxy (MOCATAXY) is investigated. By systematically exploring the parameter space accessed by MBE and MOCATAXY, phase-pure $\alpha$-Ga$_2$O$_3$(10$\bar{1}$0) and $\alpha$-(In$_x$Ga$_{1-x}$)$_2$O$_3$(10$\bar{1}$0) thin films are realized. The presence of In on the $\alpha$-Ga$_2$O$_3$ growth surface remarkably expands its growth window far into the metal-rich flux regime and to higher growth temperatures. With increasing O-to-Ga flux ratio ($R_{\text{O}}$), In incorporates into $\alpha$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ up to $x \leq 0.08$. Upon a critical thickness, $\beta$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ nucleates and subsequently heteroepitaxially grows on top of $\alpha$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ facets. Metal-rich MOCATAXY growth conditions, where $\alpha$-Ga$_2$O$_3$ would not conventionally stabilize, lead to single-crystalline $\alpha$-Ga$_2$O$_3$ with negligible In incorporation and improved surface morphology. Higher $T_{\text{G}}$ further results in single-crystalline $\alpha$-Ga$_2$O$_3$ with well-defined terraces and step edges at their surfaces. For $R_{\text{O}} \leq 0.53$, In acts as a surfactant on the $\alpha$-Ga$_2$O$_3$ growth surface by favoring step edges, while for $R_{\text{O}} \geq 0.8$, In incorporates and leads to a-plane $\alpha$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ faceting and the subsequent ($\bar{2}$01) $\beta$-(In$_x$Ga$_{1-x}$)$_2$O$_3$ growth on top. Thin film analysis by STEM reveals highly crystalline $\alpha$-Ga$_2$O$_3$ layers and interfaces. We provide a phase diagram to guide the MBE and MOCATAXY growth of single-crystalline $\alpha$-Ga$_2$O$_3$ on $\alpha$-Al$_2$O$_3$(10$\bar{1}$0).

Finite element models of electrical machines allow insights in electrothermal stresses which endanger the insulation system of the machine. This paper presents a thermal finite element model of a 3.7 kW squirrel-cage induction machine. The model resolves the conductors and the surrounding insulation materials in the stator slots. A set of transient thermal scenarios is defined and measured in the machine laboratory. These data are used to assess the finite element model.

We propose a new scheme to control the shape of the Autler-Townes (AT) doublet in the photoelectron spectrum from atomic resonance-enhanced multiphoton ionization (REMPI). The scheme is based on the interference of two AT doublets created by ionization of the strongly driven atom from the ground and the resonantly excited state using tailored bichromatic femtosecond (fs) laser pulses. In this scheme, the quantum phase of the photoelectrons is crucial for the manipulation of the AT doublet. The laser polarization state and the relative optical phase between the two colors are used to manipulate the interference pattern. We develop an analytical model to describe the bichromatic REMPI process and provide a physical picture of the control mechanism. To validate the model, the results are compared to an ab initio calculation based on the solution of the 2D time-dependent Schr\"odinger equation for the non-perturbative interaction of an atom with intense polarization-shaped bichromatic fs-laser pulses. Our results indicate that the control mechanism is robust with respect to the laser intensity facilitating its experimental observation., Comment: 13 pages, 5 figures

Controlled abstractive summarization focuses on producing condensed versions of a source article to cover specific aspects by shifting the distribution of generated text towards a desired style, e.g., a set of topics. Subsequently, the resulting summaries may be tailored to user-defined requirements. This paper presents a new Transformer-based architecture capable of producing topic-focused summaries. The architecture modifies the cross-attention mechanism of the Transformer to bring topic-focus control to the generation process while not adding any further parameters to the model. We show that our model sets a new state of the art on the NEWTS dataset in terms of topic-focused abstractive summarization as well as a topic-prevalence score. Moreover, we show via extensive experiments that our proposed topical cross-attention mechanism can be plugged into various Transformer models, such as BART and T5, improving their performance on the CNN/Dailymail and XSum benchmark datasets for abstractive summarization. This is achieved via fine-tuning, without requiring training from scratch. Finally, we show through human evaluation that our model generates more faithful summaries outperforming the state-of-the-art Frost model.

Machine learning (ML) provides powerful tools for predictive modeling. ML's popularity stems from the promise of sample-level prediction with applications across a variety of fields from physics and marketing to healthcare. However, if not properly implemented and evaluated, ML pipelines may contain leakage typically resulting in overoptimistic performance estimates and failure to generalize to new data. This can have severe negative financial and societal implications. Our aim is to expand understanding associated with causes leading to leakage when designing, implementing, and evaluating ML pipelines. Illustrated by concrete examples, we provide a comprehensive overview and discussion of various types of leakage that may arise in ML pipelines., Comment: second draft

Representations from large language models (LLMs) are known to be dominated by a small subset of dimensions with exceedingly high variance. Previous works have argued that although ablating these outlier dimensions in LLM representations hurts downstream performance, outlier dimensions are detrimental to the representational quality of embeddings. In this study, we investigate how fine-tuning impacts outlier dimensions and show that 1) outlier dimensions that occur in pre-training persist in fine-tuned models and 2) a single outlier dimension can complete downstream tasks with a minimal error rate. Our results suggest that outlier dimensions can encode crucial task-specific knowledge and that the value of a representation in a single outlier dimension drives downstream model decisions., Comment: Camera-ready version for EMNLP 2023

Recent work in mechanistic interpretability has shown that behaviors in language models can be successfully reverse-engineered through circuit analysis. A common criticism, however, is that each circuit is task-specific, and thus such analysis cannot contribute to understanding the models at a higher level. In this work, we present evidence that insights (both low-level findings about specific heads and higher-level findings about general algorithms) can indeed generalize across tasks. Specifically, we study the circuit discovered in Wang et al. (2022) for the Indirect Object Identification (IOI) task and 1.) show that it reproduces on a larger GPT2 model, and 2.) that it is mostly reused to solve a seemingly different task: Colored Objects (Ippolito & Callison-Burch, 2023). We provide evidence that the process underlying both tasks is functionally very similar, and contains about a 78% overlap in in-circuit attention heads. We further present a proof-of-concept intervention experiment, in which we adjust four attention heads in middle layers in order to 'repair' the Colored Objects circuit and make it behave like the IOI circuit. In doing so, we boost accuracy from 49.6% to 93.7% on the Colored Objects task and explain most sources of error. The intervention affects downstream attention heads in specific ways predicted by their interactions in the IOI circuit, indicating that this subcircuit behavior is invariant to the different task inputs. Overall, our results provide evidence that it may yet be possible to explain large language models' behavior in terms of a relatively small number of interpretable task-general algorithmic building blocks and computational components., Comment: Accepted at ICLR 2024

We consider dynamic auctions for finding Walrasian equilibria in markets with indivisible items and strong gross substitutes valuation functions. Each price adjustment step in these auction algorithms requires finding an inclusion-wise minimal maximal overdemanded set or an inclusion-wise minimal maximal underdemanded set at the current prices. Both can be formulated as a submodular function minimization problem. We observe that minimizing this submodular function corresponds to a polymatroid sum problem, and using this viewpoint, we give a fast and simple push-relabel algorithm for finding the required sets. This improves on the previously best running time of Murota, Shioura and Yang (ISAAC 2013). Our algorithm is an adaptation of the push-relabel framework by Frank and Mikl\'os (JJIAM 2012) to the particular setting. We obtain a further improvement for the special case of unit-supplies. We further show the following monotonicty properties of Walrasian prices: both the minimal and maximal Walrasian prices can only increase if supply of goods decreases, or if the demand of buyers increases. This is derived from a fine-grained analysis of market prices. We call "packing prices" a price vector such that there is a feasible allocation where each buyer obtains a utility maximizing set. Conversely, by "covering prices" we mean a price vector such that there exists a collection of utility maximizing sets of the buyers that include all available goods. We show that for strong gross substitutes valuations, the component-wise minimal packing prices coincide with the minimal Walrasian prices and the component-wise maximal covering prices coincide with the maximal Walrasian prices. These properties in turn lead to the price monotonicity results., Comment: Added new results for maximal Walrasian prices and other types of ascending auctions. Improved introduction

An increasing number of CS researchers are employed in academic non-CS departments where publication output is measured in terms of journal impact factors. To foster recognition of publications in peer-reviewed CS conference proceedings, we analyzed more than 40,000 CS publications and computed journal impact factors for 88 top-ranking conferences across a representative range of fields, finding that some conferences have impact factors corresponding to those of high-ranking journals., Comment: Submitted to Communications of the ACM

Societal biases that are contained in retrieved documents have received increased interest. Such biases, which are often prevalent in the training data and learned by the model, can cause societal harms, by misrepresenting certain groups, and by enforcing stereotypes. Mitigating such biases demands algorithms that balance the trade-off between maximized utility for the user with fairness objectives, which incentivize unbiased rankings. Prior work on bias mitigation often assumes that ranking scores, which correspond to the utility that a document holds for a user, can be accurately determined. In reality, there is always a degree of uncertainty in the estimate of expected document utility. This uncertainty can be approximated by viewing ranking models through a Bayesian perspective, where the standard deterministic score becomes a distribution. In this work, we investigate whether uncertainty estimates can be used to decrease the amount of bias in the ranked results, while minimizing loss in measured utility. We introduce a simple method that uses the uncertainty of the ranking scores for an uncertainty-aware, post hoc approach to bias mitigation. We compare our proposed method with existing baselines for bias mitigation with respect to the utility-fairness trade-off, the controllability of methods, and computational costs. We show that an uncertainty-based approach can provide an intuitive and flexible trade-off that outperforms all baselines without additional training requirements, allowing for the post hoc use of this approach on top of arbitrary retrieval models.

In recent research, in the domain of speech processing, large End-to-End (E2E) systems for Automatic Speech Recognition (ASR) have reported state-of-the-art performance on various benchmarks. These systems intrinsically learn how to handle and remove noise conditions from speech. Previous research has shown, that it is possible to extract the denoising capabilities of these models into a preprocessor network, which can be used as a frontend for downstream ASR models. However, the proposed methods were limited to specific fully convolutional architectures. In this work, we propose a novel method to extract the denoising capabilities, that can be applied to any encoder-decoder architecture. We propose the Cleancoder preprocessor architecture that extracts hidden activations from the Conformer ASR model and feeds them to a decoder to predict denoised spectrograms. We train our pre-processor on the Noisy Speech Database (NSD) to reconstruct denoised spectrograms from noisy inputs. Then, we evaluate our model as a frontend to a pretrained Conformer ASR model as well as a frontend to train smaller Conformer ASR models from scratch. We show that the Cleancoder is able to filter noise from speech and that it improves the total Word Error Rate (WER) of the downstream model in noisy conditions for both applications., Comment: Submitted and accepted for ICANN 2023 (32nd International Conference on Artificial Neural Networks)

High-precision remote clock synchronization is crucial for many classical and quantum network applications. Evaluating options for space-Earth links, we find that traditional solutions may not produce the desired synchronization for low Earth orbits and unnecessarily complicate quantum-networking architectures. Demonstrating an alternative, we use commercial off-the-shelf quantum-photon sources and detection equipment to synchronize two remote clocks across our freespace testbed utilizing a method called two-way quantum time transfer (QTT). We reach picosecond-scale timing precision under very lossy and noisy channel conditions representative of daytime space-Earth links and software-emulated satellite motion. This work demonstrates how QTT is potentially relevant for daytime space-Earth quantum networking and/or providing high-precision timing in GPS-denied environments., Comment: arXiv admin note: text overlap with arXiv:2211.00737

A Stackelberg Vertex Cover game is played on an undirected graph $\mathcal{G}$ where some of the vertices are under the control of a \emph{leader}. The remaining vertices are assigned a fixed weight. The game is played in two stages. First, the leader chooses prices for the vertices under her control. Afterward, the second player, called \emph{follower}, selects a min weight vertex cover in the resulting weighted graph. That is, the follower selects a subset of vertices $C^*$ such that every edge has at least one endpoint in $C^*$ of minimum weight w.r.t.\ to the fixed weights, and the prices set by the leader. Stackelberg Vertex Cover (StackVC) describes the leader's optimization problem to select prices in the first stage of the game so as to maximize her revenue, which is the cumulative price of all her (priceable) vertices that are contained in the follower's solution. Previous research showed that StackVC is \textsf{NP}-hard on bipartite graphs, but solvable in polynomial time in the special case of bipartite graphs, where all priceable vertices belong to the same side of the bipartition. In this paper, we investigate StackVC on paths and present a dynamic program with linear time and space complexity., Comment: 22 pages, 2 figures, 4 algorithms, extended abstract published at SAGT2023

Multimodal learning models have become increasingly important as they surpass single-modality approaches on diverse tasks ranging from question-answering to autonomous driving. Despite the importance of multimodal learning, existing efforts focus on NLP applications, where the number of modalities is typically less than four (audio, video, text, images). However, data inputs in other domains, such as the medical field, may include X-rays, PET scans, MRIs, genetic screening, clinical notes, and more, creating a need for both efficient and accurate information fusion. Many state-of-the-art models rely on pairwise cross-modal attention, which does not scale well for applications with more than three modalities. For $n$ modalities, computing attention will result in $n \choose 2$ operations, potentially requiring considerable amounts of computational resources. To address this, we propose a new domain-neutral attention mechanism, One-Versus-Others (OvO) attention, that scales linearly with the number of modalities and requires only $n$ attention operations, thus offering a significant reduction in computational complexity compared to existing cross-modal attention algorithms. Using three diverse real-world datasets as well as an additional simulation experiment, we show that our method improves performance compared to popular fusion techniques while decreasing computation costs.

Explainable Information Retrieval (XIR) is a growing research area focused on enhancing transparency and trustworthiness of the complex decision-making processes taking place in modern information retrieval systems. While there has been progress in developing XIR systems, empirical evaluation tools to assess the degree of explainability attained by such systems are lacking. To close this gap and gain insights into the true merit of XIR systems, we extend existing insights from a factor analysis of search explainability to introduce SSE (Search System Explainability), an evaluation metric for XIR search systems. Through a crowdsourced user study, we demonstrate SSE's ability to distinguish between explainable and non-explainable systems, showing that systems with higher scores indeed indicate greater interpretability. Additionally, we observe comparable perceived temporal demand and performance levels between non-native and native English speakers. We hope that aside from these concrete contributions to XIR, this line of work will serve as a blueprint for similar explainability evaluation efforts in other domains of machine learning and natural language processing., Comment: arXiv admin note: substantial text overlap with arXiv:2210.09430

Given the success of Large Language Models (LLMs), there has been considerable interest in studying the properties of model activations. The literature overwhelmingly agrees that LLM representations are dominated by a few "outlier dimensions" with exceedingly high variance and magnitude. Several studies in Natural Language Processing (NLP) have sought to mitigate the impact of such outlier dimensions and force LLMs to be isotropic (i.e., have uniform variance across all dimensions in embedding space). Isotropy is thought to be a desirable property for LLMs that improves model performance and more closely aligns textual representations with human intuition. However, many of the claims regarding isotropy in NLP have been based on the average cosine similarity of embeddings, which has recently been shown to be a flawed measure of isotropy. In this paper, we propose I-STAR: IsoScore*-based STable Anisotropic Regularization, a novel regularization method that can be used to increase or decrease levels of isotropy in embedding space during training. I-STAR uses IsoScore*, the first accurate measure of isotropy that is both differentiable and stable on mini-batch computations. In contrast to several previous works, we find that decreasing isotropy in contextualized embeddings improves performance on the majority of tasks and models considered in this paper., Comment: Camera-ready version for ICLR 2024

Sparse annotation poses persistent challenges to training dense retrieval models; for example, it distorts the training signal when unlabeled relevant documents are used spuriously as negatives in contrastive learning. To alleviate this problem, we introduce evidence-based label smoothing, a novel, computationally efficient method that prevents penalizing the model for assigning high relevance to false negatives. To compute the target relevance distribution over candidate documents within the ranking context of a given query, we assign a non-zero relevance probability to those candidates most similar to the ground truth based on the degree of their similarity to the ground-truth document(s). To estimate relevance we leverage an improved similarity metric based on reciprocal nearest neighbors, which can also be used independently to rerank candidates in post-processing. Through extensive experiments on two large-scale ad hoc text retrieval datasets, we demonstrate that reciprocal nearest neighbors can improve the ranking effectiveness of dense retrieval models, both when used for label smoothing, as well as for reranking. This indicates that by considering relationships between documents and queries beyond simple geometric distance we can effectively enhance the ranking context., Comment: EMNLP 2023

A primary criticism towards language models (LMs) is their inscrutability. This paper presents evidence that, despite their size and complexity, LMs sometimes exploit a simple vector arithmetic style mechanism to solve some relational tasks using regularities encoded in the hidden space of the model (e.g., Poland:Warsaw::China:Beijing). We investigate a range of language model sizes (from 124M parameters to 176B parameters) in an in-context learning setting, and find that for a variety of tasks (involving capital cities, uppercasing, and past-tensing) a key part of the mechanism reduces to a simple additive update typically applied by the feedforward (FFN) networks. We further show that this mechanism is specific to tasks that require retrieval from pretraining memory, rather than retrieval from local context. Our results contribute to a growing body of work on the interpretability of LMs, and offer reason to be optimistic that, despite the massive and non-linear nature of the models, the strategies they ultimately use to solve tasks can sometimes reduce to familiar and even intuitive algorithms., Comment: NAACL

Hospital discharge documentation is among the most essential, yet time-consuming documents written by medical practitioners. The objective of this study was to automatically generate hospital discharge summaries using neural network summarization models. We studied various data preparation and neural network training techniques that generate discharge summaries. Using nursing notes and discharge summaries from the MIMIC-III dataset, we studied the viability of the automatic generation of various sections of a discharge summary using four state-of-the-art neural network summarization models (BART, T5, Longformer and FLAN-T5). Our experiments indicated that training environments including nursing notes as the source, and discrete sections of the discharge summary as the target output (e.g. "History of Present Illness") improve language model efficiency and text quality. According to our findings, the fine-tuned BART model improved its ROUGE F1 score by 43.6% against its standard off-the-shelf version. We also found that fine-tuning the baseline BART model with other setups caused different degrees of improvement (up to 80% relative improvement). We also observed that a fine-tuned T5 generally achieves higher ROUGE F1 scores than other fine-tuned models and a fine-tuned FLAN-T5 achieves the highest ROUGE score overall, i.e., 45.6. For majority of the fine-tuned language models, summarizing discharge summary report sections separately outperformed the summarization the entire report quantitatively. On the other hand, fine-tuning language models that were previously instruction fine-tuned showed better performance in summarizing entire reports. This study concludes that a focused dataset designed for the automatic generation of discharge summaries by a language model can produce coherent Discharge Summary sections.

We consider a market where a set of objects is sold to a set of buyers, each equipped with a valuation function for the objects. The goal of the auctioneer is to determine reasonable prices together with a stable allocation. One definition of "reasonable" and "stable" is a Walrasian equilibrium, which is a tuple consisting of a price vector together with an allocation satisfying the following desirable properties: (i) the allocation is market-clearing in the sense that as much as possible is sold, and (ii) the allocation is stable in the sense that every buyer ends up with an optimal set with respect to the given prices. Moreover, "buyer-optimal" means that the prices are smallest possible among all Walrasian prices. In this paper, we present a combinatorial network flow algorithm to compute buyer-optimal Walrasian prices in a multi-unit matching market with additive valuation functions and buyer demands. The algorithm can be seen as a generalization of the classical housing market auction and mimics the very natural procedure of an ascending auction. We use our structural insights to prove monotonicity of the buyer-optimal Walrasian prices with respect to changes in supply or demand.

This paper provides an introduction to and discussion of SIGNAL, an industry-scale process data querying language and engine for large-scale cloud-based systems that is developed by SAP Signavio. SIGNAL is optimized for fast read access to process data in event log format and utilizes an in-memory columnar store to this end. To facilitate usability, SIGNAL uses an SQL-like syntax with additional domain-specific querying features and in particular row-pattern matching-based temporal operators. Also, the paper highlights research challenges related to process querying that are informed by the implementation and application of SIGNAL.

Cross-lingual summarization (CLS) has attracted increasing interest in recent years due to the availability of large-scale web-mined datasets and the advancements of multilingual language models. However, given the rareness of naturally occurring CLS resources, the majority of datasets are forced to rely on translation which can contain overly literal artifacts. This restricts our ability to observe naturally occurring CLS pairs that capture organic diction, including instances of code-switching. This alteration between languages in mid-message is a common phenomenon in multilingual settings yet has been largely overlooked in cross-lingual contexts due to data scarcity. To address this gap, we introduce CroCoSum, a dataset of cross-lingual code-switched summarization of technology news. It consists of over 24,000 English source articles and 18,000 human-written Chinese news summaries, with more than 92% of the summaries containing code-switched phrases. For reference, we evaluate the performance of existing approaches including pipeline, end-to-end, and zero-shot methods. We show that leveraging existing CLS resources as a pretraining step does not improve performance on CroCoSum, indicating the limited generalizability of current datasets. Finally, we discuss the challenges of evaluating cross-lingual summarizers on code-switched generation through qualitative error analyses., Comment: Accepted at LREC-COLING 2024

Large pre-trained language models contain societal biases and carry along these biases to downstream tasks. Current in-processing bias mitigation approaches (like adversarial training) impose debiasing by updating a model's parameters, effectively transferring the model to a new, irreversible debiased state. In this work, we propose a novel approach to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand, while keeping the core model untouched. Drawing from the concept of AdapterFusion in multi-task learning, we introduce DAM (Debiasing with Adapter Modules) - a debiasing approach to first encapsulate arbitrary bias mitigation functionalities into separate adapters, and then add them to the model on-demand in order to deliver fairness qualities. We conduct a large set of experiments on three classification tasks with gender, race, and age as protected attributes. Our results show that DAM improves or maintains the effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance, while granting parameter-efficiency and easy switching between the original and debiased models., Comment: Post EACL 2023 version

We report the growth of $\alpha-Ga_2O_3$ on $m$-plane $Al_2O_3$ by conventional plasma-assisted molecular-beam epitaxy (MBE) and In-mediated metal-oxide-catalyzed epitaxy (MOCATAXY). We report a growth-rate-diagram for $\alpha-Ga_2O_3$ (10-10), and observe (i) a growth rate increase, (ii) an expanded growth window, and (iii) reduced out-of-lane mosaic spread when MOCATAXY is employed for the growth of $\alpha-Ga_2O_3$. Through the use of In-mediated catalysis, growth rates over $0.2\,\mu\text{m}\,\text{hr}^{-1}$ and rocking curves with full width at half maxima of $\Delta\omega \approx 0.45^{\circ}$ are achieved. Faceting is observed along the $\alpha-Ga_2O_3$ film surface and is explored through scanning transmission electron microscopy.

Epilepsy is one of the most common neurological disorders, typically observed via seizure episodes. Epileptic seizures are commonly monitored through electroencephalogram (EEG) recordings due to their routine and low expense collection. The stochastic nature of EEG makes seizure identification via manual inspections performed by highly-trained experts a tedious endeavor, motivating the use of automated identification. The literature on automated identification focuses mostly on supervised learning methods requiring expert labels of EEG segments that contain seizures, which are difficult to obtain. Motivated by these observations, we pose seizure identification as an unsupervised anomaly detection problem. To this end, we employ the first unsupervised transformer-based model for seizure identification on raw EEG. We train an autoencoder involving a transformer encoder via an unsupervised loss function, incorporating a novel masking strategy uniquely designed for multivariate time-series data such as EEG. Training employs EEG recordings that do not contain any seizures, while seizures are identified with respect to reconstruction errors at inference time. We evaluate our method on three publicly available benchmark EEG datasets for distinguishing seizure vs. non-seizure windows. Our method leads to significantly better seizure identification performance than supervised learning counterparts, by up to 16% recall, 9% accuracy, and 9% Area under the Receiver Operating Characteristics Curve (AUC), establishing particular benefits on highly imbalanced data. Through accurate seizure identification, our method could facilitate widely accessible and early detection of epilepsy development, without needing expensive label collection or manual feature extraction., Comment: IEEE Conference on Machine Learning and Applications (ICMLA) 2022

As information retrieval (IR) systems, such as search engines and conversational agents, become ubiquitous in various domains, the need for transparent and explainable systems grows to ensure accountability, fairness, and unbiased results. Despite recent advances in explainable AI and IR techniques, there is no consensus on the definition of explainability. Existing approaches often treat it as a singular notion, disregarding the multidimensional definition postulated in the literature. In this paper, we use psychometrics and crowdsourcing to identify human-centered factors of explainability in Web search systems and introduce SSE (Search System Explainability), an evaluation metric for explainable IR (XIR) search systems. In a crowdsourced user study, we demonstrate SSE's ability to distinguish between explainable and non-explainable systems, showing that systems with higher scores indeed indicate greater interpretability. We hope that aside from these concrete contributions to XIR, this line of work will serve as a blueprint for similar explainability evaluation efforts in other domains of machine learning and natural language processing., Comment: 11 pages, 4 figures, accepted at SIGIR 2024 as full paper

Machine learning (ML) approaches to data analysis are now widely adopted in many fields including epidemiology and medicine. To apply these approaches, confounds must first be removed as is commonly done by featurewise removal of their variance by linear regression before applying ML. Here, we show this common approach to confound removal biases ML models, leading to misleading results. Specifically, this common deconfounding approach can leak information such that what are null or moderate effects become amplified to near-perfect prediction when nonlinear ML approaches are subsequently applied. We identify and evaluate possible mechanisms for such confound-leakage and provide practical guidance to mitigate its negative impact. We demonstrate the real-world importance of confound-leakage by analyzing a clinical dataset where accuracy is overestimated for predicting attention deficit hyperactivity disorder (ADHD) with depression as a confound. Our results have wide-reaching implications for implementation and deployment of ML workflows and beg caution against na\"ive use of standard confound removal approaches., Comment: Revised Introduction, added CoI, results unchanged

The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber, Comment: Accepted at ICLR 2023

Inferring linear relationships lies at the heart of many empirical investigations. A measure of linear dependence should correctly evaluate the strength of the relationship as well as qualify whether it is meaningful for the population. Pearson's correlation coefficient (PCC), the \textit{de-facto} measure for bivariate relationships, is known to lack in both regards. The estimated strength $r$ maybe wrong due to limited sample size, and nonnormality of data. In the context of statistical significance testing, erroneous interpretation of a $p$-value as posterior probability leads to Type I errors -- a general issue with significance testing that extends to PCC. Such errors are exacerbated when testing multiple hypotheses simultaneously. To tackle these issues, we propose a machine-learning-based predictive data calibration method which essentially conditions the data samples on the expected linear relationship. Calculating PCC using calibrated data yields a calibrated $p$-value that can be interpreted as posterior probability together with a calibrated $r$ estimate, a desired outcome not provided by other methods. Furthermore, the ensuing independent interpretation of each test might eliminate the need for multiple testing correction. We provide empirical evidence favouring the proposed method using several simulations and application to real-world data.

Transformers are powerful text representation learners, useful for all kinds of clinical decision support tasks. Although they outperform baselines on readmission prediction, they are not infallible. Here, we look into one such failure case, and report patterns that lead to inferior predictive performance.

Deep neural networks currently provide the most advanced and accurate machine learning models to distinguish between structural MRI scans of subjects with Alzheimer's disease and healthy controls. Unfortunately, the subtle brain alterations captured by these models are difficult to interpret because of the complexity of these multi-layer and non-linear models. Several heatmap methods have been proposed to address this issue and analyze the imaging patterns extracted from the deep neural networks, but no quantitative comparison between these methods has been carried out so far. In this work, we explore these questions by deriving heatmaps from Convolutional Neural Networks (CNN) trained using T1 MRI scans of the ADNI data set, and by comparing these heatmaps with brain maps corresponding to Support Vector Machines (SVM) coefficients. Three prominent heatmap methods are studied: Layer-wise Relevance Propagation (LRP), Integrated Gradients (IG), and Guided Grad-CAM (GGC). Contrary to prior studies where the quality of heatmaps was visually or qualitatively assessed, we obtained precise quantitative measures by computing overlap with a ground-truth map from a large meta-analysis that combined 77 voxel-based morphometry (VBM) studies independently from ADNI. Our results indicate that all three heatmap methods were able to capture brain regions covering the meta-analysis map and achieved better results than SVM coefficients. Among them, IG produced the heatmaps with the best overlap with the independent meta-analysis.

Lexical semantics and cognitive science point to affordances (i.e. the actions that objects support) as critical for understanding and representing nouns and verbs. However, study of these semantic features has not yet been integrated with the "foundation" models that currently dominate language representation research. We hypothesize that predictive modeling of object state over time will result in representations that encode object affordance information "for free". We train a neural network to predict objects' trajectories in a simulated interaction and show that our network's latent representations differentiate between both observed and unobserved affordances. We find that models trained using 3D simulations from our SPATIAL dataset outperform conventional 2D computer vision models trained on a similar task, and, on initial inspection, that differences between concepts correspond to expected features (e.g., roll entails rotation). Our results suggest a way in which modern deep learning approaches to grounded language learning can be integrated with traditional formal semantic notions of lexical representations., Comment: *SEM 2022

Distributed model predictive control (DMPC) is often used to tackle path planning for unmanned aerial vehicle (UAV) swarms. However, it requires considerable computations on-board the UAV, leading to increased weight and power consumption. In this work, we propose to offload path planning computations to multiple ground-based computation units. As simultaneously communicating and recomputing all trajectories is not feasible for a large swarm with tight timing requirements, we develop a novel event-triggered DMPC that selects a subset of most relevant UAV trajectories to be replanned. The resulting architecture reduces UAV weight and power consumption, while the active redundancy provides robustness against computation unit failures. Moreover, the DMPC guarantees feasible and collision-free trajectories for UAVs with linear dynamics. In simulations, we demonstrate that our method can reliably plan trajectories, while saving 60% of network traffic and required computational power. Hardware-in-the-loop experiments show that it is suitable to control real quadcopter swarms., Comment: Accepted for publication at the IFAC Conference on Networked Systems (NecSys) 2022

Alzheimer's Disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis. We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities - a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance. MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-Scores. Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses. This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support., Comment: 11 pages, 5 figures

Text summarization models are approaching human levels of fidelity. Existing benchmarking corpora provide concordant pairs of full and abridged versions of Web, news or, professional content. To date, all summarization datasets operate under a one-size-fits-all paradigm that may not reflect the full range of organic summarization needs. Several recently proposed models (e.g., plug and play language models) have the capacity to condition the generated summaries on a desired range of themes. These capacities remain largely unused and unevaluated as there is no dedicated dataset that would support the task of topic-focused summarization. This paper introduces the first topical summarization corpus NEWTS, based on the well-known CNN/Dailymail dataset, and annotated via online crowd-sourcing. Each source article is paired with two reference summaries, each focusing on a different theme of the source document. We evaluate a representative range of existing techniques and analyze the effectiveness of different prompting methods.

In recent years, large-scale transformer decoders such as the GPT-x family of models have become increasingly popular. Studies examining the behavior of these models tend to focus only on the output of the language modeling head and avoid analysis of the internal states of the transformer decoder. In this study, we present a collection of methods to analyze the hidden states of GPT-2 and use the model's navigation of garden path sentences as a case study. To enable this, we compile the largest currently available dataset of garden path sentences. We show that Manhattan distances and cosine similarities provide more reliable insights compared to established surprisal methods that analyze next-token probabilities computed by a language modeling head. Using these methods, we find that negating tokens have minimal impacts on the model's representations for unambiguous forms of sentences with ambiguity solely over what the object of a verb is, but have a more substantial impact of representations for unambiguous sentences whose ambiguity would stem from the voice of a verb. Further, we find that analyzing the decoder model's hidden states reveals periods of ambiguity that might conclude in a garden path effect but happen not to, whereas surprisal analyses routinely miss this detail., Comment: 9 pages, 6 figures, Accepted to EMNLP BlackBox NLP 2022

Research on knowledge graph embedding (KGE) has emerged as an active field in which most existing KGE approaches mainly focus on static structural data and ignore the influence of temporal variation involved in time-aware triples. In order to deal with this issue, several temporal knowledge graph embedding (TKGE) approaches have been proposed to integrate temporal and structural information in recent years. However, these methods only employ a uniformly random sampling to construct negative facts. As a consequence, the corrupted samples are often too simplistic for training an effective model. In this paper, we propose a new temporal knowledge graph embedding framework by introducing adversarial learning to further refine the performance of traditional TKGE models. In our framework, a generator is utilized to construct high-quality plausible quadruples and a discriminator learns to obtain the embeddings of entities and relations based on both positive and negative samples. Meanwhile, we also apply a Gumbel-Softmax relaxation and the Wasserstein distance to prevent vanishing gradient problems on discrete data; an inherent flaw in traditional generative adversarial networks. Through comprehensive experimentation on temporal datasets, the results indicate that our proposed framework can attain significant improvements based on benchmark models and also demonstrate the effectiveness and applicability of our framework.

We present a multichannel photoelectron interferometry technique based on trichromatic pulse shaping for unambiguous determination of quantum phases in multiphoton ionization (MPI) of potassium atoms. The colors of the laser field are chosen to produce three energetically separated photoelectron interferograms in the continuum. While the red pulse is two-photon resonant with the $3d$-state resulting in a (2+1) resonance-enhanced MPI (REMPI), a (1+2) REMPI occurs via the non-resonant intermediate $4p$-state with an initial green or blue pulse. We show that ionization via a non-resonant intermediate state lifts the degeneracy of photoelectron interferograms from pathways consisting of permutations of the colors. The analysis of the interferograms reveals a phase shift of $\pm \pi/2$ depending on the sign of the detunings in the (1+2) REMPI pathways. In addition, we demonstrate that the photoionization time delay in the resonant (2+1) REMPI pathway gives rise to a linear spectral phase in the photoelectron spectra. Insights into the underlying MPI processes are gained through an analytic perturbative description and numerical simulations of a trichromatic driven three level system coupled to the continuum., Comment: 17 pages, 5 figures

Introduction: Identifying neurobiological differences between patients suffering from Major Depressive Disorder (MDD) and healthy individuals has been a mainstay of clinical neuroscience for decades. However, recent meta- and mega-analyses have raised concerns regarding the replicability and clinical relevance of brain alterations in depression. Methods: Here, we systematically investigate healthy controls and MDD patients across a comprehensive range of modalities including structural magnetic resonance imaging (MRI), diffusion tensor imaging, functional task-based and resting-state MRI under near-ideal conditions. To this end, we quantify the upper bounds of univariate effect sizes, predictive utility, and distributional dissimilarity in a fully harmonized cohort of N=1,809 participants. We compare the results to an MDD polygenic risk score (PRS) and environmental variables. Results: The upper bound of the effect sizes range from partial eta squared = .004 to .017, distributions overlap between 89% and 95%, with classification accuracies ranging between 54% and 55% across neuroimaging modalities. This pattern remains virtually unchanged when considering only acutely or chronically depressed patients. Differences are comparable to those found for PRS, but substantially smaller than for environmental variables. Discussion: We provide a large-scale, multimodal analysis of univariate biological differences between MDD patients and controls and show that even under near-ideal conditions and for maximum biological differences, deviations are extremely small and similarity dominates. We sketch an agenda for a new focus of future research in biological psychiatry facilitating quantitative, theory-driven research, an emphasis on computational psychiatry and multivariate machine learning approaches, as well as the utilization of ecologically valid phenotyping., Comment: 12 pages, 3 figures

Contrastive learning has been the dominant approach to training dense retrieval models. In this work, we investigate the impact of ranking context - an often overlooked aspect of learning dense retrieval models. In particular, we examine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of random negatives, and a fully list-wise loss. To incorporate these factors into training, we introduce Contextual Document Embedding Reranking (CODER), a highly efficient retrieval framework. When reranking, it incurs only a negligible computational overhead on top of a first-stage method at run time (delay per query in the order of milliseconds), allowing it to be easily combined with any state-of-the-art dual encoder method. After fine-tuning through CODER, which is a lightweight and fast process, models can also be used as stand-alone retrievers. Evaluating CODER in a large set of experiments on the MS~MARCO and TripClick collections, we show that the contextual reranking of precomputed document embeddings leads to a significant improvement in retrieval performance. This improvement becomes even more pronounced when more relevance information per query is available, shown in the TripClick collection, where we establish new state-of-the-art results by a large margin., Comment: EMNLP 2022