Search

The considerable body of data available for evaluating biometric recognition systems in Research and Development (R\&D) environments has contributed to the increasingly common problem of target performance mismatch. Biometric algorithms are frequently tested against data that may not reflect the real world applications they target. From a Testing and Evaluation (T\&E) standpoint, this domain mismatch causes difficulty assessing when improvements in State-of-the-Art (SOTA) research actually translate to improved applied outcomes. This problem can be addressed with thoughtful preparation of data and experimental methods to reflect specific use-cases and scenarios. To that end, this paper evaluates research solutions for identifying individuals at ranges and altitudes, which could support various application areas such as counterterrorism, protection of critical infrastructure facilities, military force protection, and border security. We address challenges including image quality issues and reliance on face recognition as the sole biometric modality. By fusing face and body features, we propose developing robust biometric systems for effective long-range identification from both the ground and steep pitch angles. Preliminary results show promising progress in whole-body recognition. This paper presents these early findings and discusses potential future directions for advancing long-range biometric identification systems based on mission-driven metrics.

This paper examines covariate effects on fused whole body biometrics performance in the IARPA BRIAR dataset, specifically focusing on UAV platforms, elevated positions, and distances up to 1000 meters. The dataset includes outdoor videos compared with indoor images and controlled gait recordings. Normalized raw fusion scores relate directly to predicted false accept rates (FAR), offering an intuitive means for interpreting model results. A linear model is developed to predict biometric algorithm scores, analyzing their performance to identify the most influential covariates on accuracy at altitude and range. Weather factors like temperature, wind speed, solar loading, and turbulence are also investigated in this analysis. The study found that resolution and camera distance best predicted accuracy and findings can guide future research and development efforts in long-range/elevated/UAV biometrics and support the creation of more reliable and robust systems for national security and other critical domains.

It has recently been shown that extended cosmological epochs can exist during which the abundances associated with different energy components remain constant despite cosmological expansion. Indeed, this "stasis" behavior has been found to arise generically in many BSM theories containing large towers of states, and even serves as a cosmological attractor. However, all previous studies of stasis took place within non-thermal environments, or more specifically within environments in which thermal effects played no essential role in realizing or sustaining the stasis. In this paper, we demonstrate that stasis can emerge and serve as an attractor even within thermal environments, with thermal effects playing a critical role in the stasis dynamics. Moreover, within such environments, we find that no towers of states are needed -- a single state experiencing two-body annihilations will suffice. This work thus extends the stasis phenomenon into the thermal domain and demonstrates that thermal effects can also generally give rise to an extended stasis epoch, even when only a single matter species is involved., Comment: 16 pages, LaTeX, 7 figures

Wind power generation plays a crucial role in transitioning away from fossil fuel-dependent energy sources, contributing significantly to the mitigation of climate change. Monitoring and evaluating the aerodynamics of large wind turbine rotors is crucial to enable more wind energy deployment. This is necessary to achieve the European climate goal of a reduction in net greenhouse gas emissions by at least 55% by 2030, compared to 1990 levels. This paper presents a comparison between two measurement systems for evaluating the aerodynamic performance of wind turbine rotor blades on a full-scale wind tunnel test. One system uses an array of ten commercial compact ultra-low power micro-electromechanical systems (MEMS) pressure sensors placed on the blade surface, while the other employs high-accuracy lab-based pressure scanners embedded in the airfoil. The tests are conducted at a Reynolds number of 3.5 x 10^6, which represents typical operating conditions for wind turbines. MEMS sensors are of particular interest, as they can enable real-time monitoring which would be impossible with the ground truth system. This work provides an accurate quantification of the impact of the MEMS system on the blade aerodynamics and its measurement accuracy. Our results indicate that MEMS sensors, with a total sensing power below 1.6 mW, can measure key aerodynamic parameters like Angle of Attack (AoA) and flow separation with a precision of 1{\deg}. Although there are minor differences in measurements due to sensor encapsulation, the MEMS system does not significantly compromise blade aerodynamics, with a maximum shift in the angle of attack for flow separation of only 1{\deg}. These findings indicate that surface and low-power MEMS sensor systems are a promising approach for efficient and sustainable wind turbine monitoring using self-sustaining Internet of Things devices and wireless sensor networks.

Given a family of pretrained models and a hold-out set, how can we construct a valid conformal prediction set while selecting a model that minimizes the width of the set? If we use the same hold-out data set both to select a model (the model that yields the smallest conformal prediction sets) and then to construct a conformal prediction set based on that selected model, we suffer a loss of coverage due to selection bias. Alternatively, we could further splitting the data to perform selection and calibration separately, but this comes at a steep cost if the size of the dataset is limited. In this paper, we address the challenge of constructing a valid prediction set after efficiency-oriented model selection. Our novel methods can be implemented efficiently and admit finite-sample validity guarantees without invoking additional sample-splitting. We show that our methods yield prediction sets with asymptotically optimal size under certain notion of continuity for the model class. The improved efficiency of the prediction sets constructed by our methods are further demonstrated through applications to synthetic datasets in various settings and a real data example.

Laser-driven (LD) ion acceleration has been explored in a newly constructed short focal length beamline at the BELLA petawatt facility (interaction point 2, iP2). For applications utilizing such LD ion beams, a beam transport system is required, which for reasons of compactness be ideally contained within 3 m. The large divergence and energy spread of LD ion beams present a unique challenge to transporting them compared to beams from conventional accelerators. This work gives an overview of proposed compact transport designs that can satisfy different requirements depending on the application for the iP2 proton beamline such as radiation biology, material science, and high energy density science. These designs are optimized for different parameters such as energy spread and peak proton density according to an application's need. The various designs consist solely of permanent magnet elements, which can provide high magnetic field gradients on a small footprint. While the field strengths are fixed, we have shown that the beam size and energy can be tuned effectively by varying the placement of the magnets. The performance of each design was evaluated based on high order particle tracking simulations of typical LD proton beams. A more detailed investigation was carried out for a design to deliver 10 MeV LD accelerated ions for radiation biology applications. With these transport system designs, the iP2 beamline is ready to house various application experiments., Comment: 6 pages, 3 figures, accepted for publication in the proceedings of the 2022 IEEE Advanced Accelerator Concepts Workshop (AAC). J.T. De Chant is also affiliated with Michigan State University

Observing and characterizing young planetary systems can aid in unveiling the evolutionary mechanisms that sculpt the mature exoplanet population. As an all-sky survey, NASA's Transiting Exoplanet Survey Satellite (TESS) has expanded the known young planet population as it has observed young comoving stellar populations. This work presents the discovery of a multiplanet system orbiting the 61 Myr old G4V star TIC 434398831 (M = 0.99 Msun, R = 0.91 Rsun, Teff = 5638 K, Tmag = 11.31) located in the Theia 116 comoving population. We estimate the population's age based on rotation periods measured from the TESS light curves, isochrone fitting, and measurements of lithium equivalent widths in the spectra of Theia 116 members. The TESS FFI light curves reveal a mini-Neptune (Rb = 3.51 Rearth, Pb = 3.69 days) and super-Neptune (Rc = 5.63 Rearth, Pc = 6.21 days) with an orbital period ratio slightly larger than 5:3. Follow-up observations from CHEOPS and ground-based telescopes confirm the transits of TIC 434398831 b and c, and constrain their transit times. We explore the potential mass-loss histories of the two planets in order to probe possible initial conditions of the planets immediately after formation., Comment: 14 pages, 13 figures, 5 tables. Submitted to MNRAS

Precipitation nowcasting, critical for flood emergency and river management, has remained challenging for decades, although recent developments in deep generative modeling (DGM) suggest the possibility of improvements. River management centers, such as the Tennessee Valley Authority, have been using Numerical Weather Prediction (NWP) models for nowcasting but have struggled with missed detections even from best-in-class NWP models. While decades of prior research achieved limited improvements beyond advection and localized evolution, recent attempts have shown progress from physics-free machine learning (ML) methods and even greater improvements from physics-embedded ML approaches. Developers of DGM for nowcasting have compared their approaches with optical flow (a variant of advection) and meteorologists' judgment but not with NWP models. Further, they have not conducted independent co-evaluations with water resources and river managers. Here, we show that the state-of-the-art physics-embedded deep generative model, specifically NowcastNet, outperforms the High-Resolution Rapid Refresh (HRRR) model, the latest generation of NWP, along with advection and persistence, especially for heavy precipitation events. For grid-cell extremes over 16 mm/h, NowcastNet demonstrated a median critical success index (CSI) of 0.30, compared with a median CSI of 0.04 for HRRR. However, despite hydrologically relevant improvements in point-by-point forecasts from NowcastNet, caveats include the overestimation of spatially aggregated precipitation over longer lead times. Our co-evaluation with ML developers, hydrologists, and river managers suggests the possibility of improved flood emergency response and hydropower management.

Statistical learning under distribution shift is challenging when neither prior knowledge nor fully accessible data from the target distribution is available. Distributionally robust learning (DRL) aims to control the worst-case statistical performance within an uncertainty set of candidate distributions, but how to properly specify the set remains challenging. To enable distributional robustness without being overly conservative, in this paper, we propose a shape-constrained approach to DRL, which incorporates prior information about the way in which the unknown target distribution differs from its estimate. More specifically, we assume the unknown density ratio between the target distribution and its estimate is isotonic with respect to some partial order. At the population level, we provide a solution to the shape-constrained optimization problem that does not involve the isotonic constraint. At the sample level, we provide consistency results for an empirical estimator of the target in a range of different settings. Empirical studies on both synthetic and real data examples demonstrate the improved accuracy of the proposed shape-constrained approach.

Linear recurrent models, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple model with fewer theoretical assumptions than prior works, with a practical implementation that remains faithful to its theoretical derivation, achieving comparable scores to the LRU and SSMs on several long sequence modelling datasets., Comment: Next Generation of Sequence Modeling Architectures Workshop at ICML 2024

The youngest ($<$50 Myr) planets are vital to understand planet formation and early evolution. The 17 Myr system HIP 67522 is already known to host a giant ($\simeq$10$R_\oplus$) planet on a tight orbit. In the discovery paper, Rizzuto et al. 2020 reported a tentative single transit detection of an additional planet in the system using TESS. Here, we report the discovery of HIP 67522 c which matches with that single transit event. We confirm the signal with ground-based multi-wavelength photometry from Sinistro and MuSCAT4. At a period of 14.33 days, planet c is close to a 2:1 mean motion resonance with b (6.96 days or 2.06:1). The light curve shows distortions during many of the transits, which are consistent with spot crossing events and/or flares. Fewer stellar activity events are seen in the transits of planet b, suggesting that planet c is crossing a more active latitude. Such distortions, combined with systematics in the TESS light curve extraction, likely explain why planet c was previously missed., Comment: Submitted to AAS Journals

Single-step Coulomb excitation of $^{46,48,49,50}$Ti is presented. A complete set of $E2$ matrix elements for the quintuplet of states in $^{49}$Ti, centered on the $2^+$ core excitation, was measured for the first time. A total of nine $E2$ matrix elements are reported, four of which were previously unknown. $^{49}_{22}$Ti$_{27}$ shows a $20\%$ quenching in electric quadrupole transition strength as compared to its semi-magic $^{50}_{22}$Ti$_{28}$ neighbour. This $20\%$ quenching, while empirically unprecedented, can be explained with a remarkably simple two-state mixing model, which is also consistent with other ground-state properties such as the magnetic dipole moment and electric quadrupole moment. A connection to nucleon transfer data and the quenching of single-particle strength is also demonstrated. The simplicity of the $^{49}$Ti-$^{50}$Ti pair (i.e., approximate single-$j$ $0f_{7/2}$ valence space and isolation of yrast states from non-yrast states) provides a unique opportunity to disentangle otherwise competing effects in the ground-state properties of atomic nuclei, the emergence of collectivity, and the role of proton-neutron interactions., Comment: 8 pages, 5 figures, accepted in Physics Letters B

It has long been known that a vertex-transitive graph $\Gamma$ is isomorphic to a double coset graph $\text{Cos}(G,H,S)$ of a transitive group $G\le\text{Aut}(\Gamma)$, a vertex stabilizer $H\le G$, and some subset $S\subseteq G$. We show that the automorphism group of the Cayley graph $\text{Cay}(G,S)$ with connection set $S$ can be obtained from the automorphism group of $\text{Cos}(G,H,S)$ and vice versa. We also show that the isomorphism problem for double coset graphs is equivalent to the isomorphism problem for Cayley graphs provided one knows all groups $G$ for which a fixed Cayley graph is a Cayley graph of $G$. Our main tool is a "recognition theorem", which recognizes when a Cayley graph of a group $G$ is a wreath product of two graphs based upon its connection set., Comment: This work is supported in part by the Slovenian Research Agency (research program P1-0285 and research projects N1-0140, N1-0160, J1-2451, N1-0208, J1-3001, J1-3003, J1-4008, and J1-50000)

Error correction allows a quantum computer to preserve a state long beyond the decoherence time of its physical qubits by encoding logical qubits in a larger number of physical qubits. The leading proposal for a scheme of quantum error correction is based on the surface code, but several recently proposed quantum low-density parity check (qLDPC) codes allow more logical information to be encoded in significantly fewer physical qubits. Key to any scheme of quantum error correction is the decoder, an algorithm that estimates the error state of the qubits from the results of syndrome measurements performed on them. The surface code has a variety of fast and accurate decoders, but the state-of-the-art decoder for general qLDPC codes, BP-OSD, has a high computational complexity. Here we introduce Ambiguity Clustering (AC), an algorithm which seeks to divide the measurement data into clusters which are decoded independently. We benchmark AC on the recently proposed bivariate bicycle codes and find that, at physically realistic error rates, AC is between one and three orders of magnitude faster than BP-OSD with no reduction in logical fidelity. Our CPU implementation of AC is already fast enough to decode the 144-qubit Gross code in real time for neutral atom and trapped ion systems.

High entropy oxides (HEO)s have garnered much interest due to their available high degree of tunability. Here, we study the local structure of (MgNiCuCoZn)0.167(MnCr)0.083O, a composition based on the parent HEO (MgNiCuCoZn)0.2O.We synthesized a series of thin films via pulsed laser deposition at incremental oxygen partial pressures. X-ray diffraction shows lattice parameter to decrease with increased pO2 pressures until the onset of phase separation. X-ray absorption fine structure shows that specific atomic species in the composition dictate the global structure of the material as Cr, Co, and Mn shift to energetically favorable coordination with increasing pressure. Transmission electron microscopy analysis on a lower-pressure sample exhibits a rock salt structure, but the higher-pressure sample reveals reflections reminiscent of the spinel structure. In all, these findings give a more complete picture on how (MgNiCuCoZn)0.167(MnCr)0.083O forms with varying initial conditions and advances fundamental knowledge of cation behavior in high entropy oxides.

The probabilistic diffusion model has become highly effective across various domains. Typically, sampling from a diffusion model involves using a denoising distribution characterized by a Gaussian with a learned mean and either fixed or learned covariances. In this paper, we leverage the recently proposed full covariance moment matching technique and introduce a novel method for learning covariances. Unlike traditional data-driven covariance approximation approaches, our method involves directly regressing the optimal analytic covariance using a new, unbiased objective named Optimal Covariance Matching (OCM). This approach can significantly reduce the approximation error in covariance prediction. We demonstrate how our method can substantially enhance the sampling efficiency of both Markovian (DDPM) and non-Markovian (DDIM) diffusion model families.

This paper introduces a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon a predefined conformity score function. This process is guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model predictions and without modifying the trained model (e.g., the deep network). Systematic experiments demonstrate that starting from conventional conformal methods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage., Comment: 22 pages, 9 figures

Young (<500 Myr) planets are critical to studying how planets form and evolve. Among these young planetary systems, multi-planet configurations are particularly useful as they provide a means to control for variables within a system. Here, we report the discovery and characterization of a young planetary system, TOI-1224. We show that the planet-host resides within a young population we denote as MELANGE-5 . By employing a range of age-dating methods -- isochrone fitting, lithium abundance analysis, gyrochronology, and Gaia excess variability -- we estimate the age of MELANGE-5 to be 210$\pm$27 Myr. MELANGE-5 is situated in close proximity to previously identified younger (80 -110 Myr) associations, Crius 221 and Theia 424/Volans-Carina, motivating further work to map out the group boundaries. In addition to a planet candidate detected by the TESS pipeline and alerted as a TESS Object of Interest, TOI-1224 b, we identify a second planet, TOI-1224 c, using custom search tools optimized for young stars (Notch and LOCoR). We find the planets are 2.10$\pm$0.09$R_\oplus$ and 2.88$\pm$0.10$R_\oplus$ and orbit their host star every 4.18 and 17.95 days, respectively. With their bright ($K$=9.1 mag), small ($R_{*}$=0.44R$_{\odot}$), and cool ($T_{eff}$ =3326K) host star, these planets represent excellent candidates for atmospheric characterization with JWST., Comment: Accepted for publication in The Astronomical Journal; 33 pages, 17 figures, 9 tables

Algorithmic stability is a central notion in learning theory that quantifies the sensitivity of an algorithm to small changes in the training data. If a learning algorithm satisfies certain stability properties, this leads to many important downstream implications, such as generalization, robustness, and reliable predictive inference. Verifying that stability holds for a particular algorithm is therefore an important and practical question. However, recent results establish that testing the stability of a black-box algorithm is impossible, given limited data from an unknown distribution, in settings where the data lies in an uncountably infinite space (such as real-valued data). In this work, we extend this question to examine a far broader range of settings, where the data may lie in any space -- for example, categorical data. We develop a unified framework for quantifying the hardness of testing algorithmic stability, which establishes that across all settings, if the available data is limited then exhaustive search is essentially the only universally valid mechanism for certifying algorithmic stability. Since in practice, any test of stability would naturally be subject to computational constraints, exhaustive search is impossible and so this implies fundamental limits on our ability to test the stability property for a black-box algorithm.

We propose a new framework for algorithmic stability in the context of multiclass classification. In practice, classification algorithms often operate by first assigning a continuous score (for instance, an estimated probability) to each possible label, then taking the maximizer -- i.e., selecting the class that has the highest score. A drawback of this type of approach is that it is inherently unstable, meaning that it is very sensitive to slight perturbations of the training data, since taking the maximizer is discontinuous. Motivated by this challenge, we propose a pipeline for constructing stable classifiers from data, using bagging (i.e., resampling and averaging) to produce stable continuous scores, and then using a stable relaxation of argmax, which we call the "inflated argmax," to convert these scores to a set of candidate labels. The resulting stability guarantee places no distributional assumptions on the data, does not depend on the number of classes or dimensionality of the covariates, and holds for any base classifier. Using a common benchmark data set, we demonstrate that the inflated argmax provides necessary protection against unstable classifiers, without loss of accuracy.

Model averaging techniques based on resampling methods (such as bootstrapping or subsampling) have been utilized across many areas of statistics, often with the explicit goal of promoting stability in the resulting output. We provide a general, finite-sample theoretical result guaranteeing the stability of bagging when applied to algorithms that return outputs in a general space, so that the output is not necessarily a real-valued -- for example, an algorithm that estimates a vector of weights or a density function. We empirically assess the stability of bagging on synthetic and real-world data for a range of problem settings, including causal inference, nonparametric regression, and Bayesian model selection.

Financial firms often rely on factor models to explain correlations among asset returns. These models are important for managing risk, for example by modeling the probability that many assets will simultaneously lose value. Yet after major events, e.g., COVID-19, analysts may reassess whether existing models continue to fit well: specifically, after accounting for the factor exposures, are the residuals of the asset returns independent? With this motivation, we introduce the mosaic permutation test, a nonparametric goodness-of-fit test for preexisting factor models. Our method allows analysts to use nearly any machine learning technique to detect model violations while provably controlling the false positive rate, i.e., the probability of rejecting a well-fitting model. Notably, this result does not rely on asymptotic approximations and makes no parametric assumptions. This property helps prevent analysts from unnecessarily rebuilding accurate models, which can waste resources and increase risk. We illustrate our methodology by applying it to the Blackrock Fundamental Equity Risk (BFRE) model. Using the mosaic permutation test, we find that the BFRE model generally explains the most significant correlations among assets. However, we find evidence of unexplained correlations among certain real estate stocks, and we show that adding new factors improves model fit. We implement our methods in the python package mosaicperm., Comment: 38 pages, 13 figures

Fractional quantum Hall effect (FQHE) is a prime example of topological quantum many-body phenomena, arising from the interplay between strong electron correlation, topological order, and time reversal symmetry breaking. Recently, a lattice analog of FQHE at zero magnetic field has been observed, confirming the existence of a zero-field fractional Chern insulator (FCI). Despite this, the bulk-edge correspondence -- a hallmark of FCI featuring an insulating bulk with conductive edges -- has not been directly observed. In fact, this correspondence has not been visualized in any system for fractional states due to experimental challenges. Here we report the imaging of FCI edge states in twisted MoTe2 by employing a newly developed modality of microwave-impedance microscopy. By tuning the carrier density, we observe the system evolving between metallic and FCI states, the latter of which exhibits insulating bulk and conductive edges as expected from bulk-boundary correspondence. We also observe the evolution of edge states across the topological phase transition from an incompressible Chern insulator state to a metal and finally to a putative charge ordered insulating state as a function of interlayer electric field. The local measurement further reveals tantalizing prospects of neighboring domains with different fractional orders. These findings pave the way for research into topologically protected 1D interfaces between various anyonic states at zero magnetic field, such as topological entanglement entropy, Halperin-Laughlin interfaces, and the creation of non-abelian anyons.

The aim of this paper is to establish Hoeffding and Bernstein type concentration inequalities for weighted sums of exchangeable random variables. A special case is the i.i.d. setting, where random variables are sampled independently from some distribution (and are therefore exchangeable). In contrast to the existing literature on this problem, our results provide a natural unified view of both the i.i.d. and the exchangeable setting.

Within the first few hundreds of millions of years, many physical processes sculpt the eventual properties of young planets. NASA's TESS mission has surveyed young stellar associations across the entire sky for transiting planets providing glimpses into the various stages of planetary evolution. Using our own detection pipeline, we search a magnitude-limited sample of 7219 young stars ($\leq$200 Myr) observed in the first four years of TESS for small (2-8 R$_\oplus$), short period (1.6-20 days) transiting planets. The completeness of our survey is characterized by a series of injection and recovery simulations. Our analysis of TESS 2-minute cadence and Full Frame Image (FFI) light curves recover all known TOIs, as well as four new planet candidates not previously identified as TOIs. We derive an occurrence rate of $35^{+13}_{-10}$% for mini-Neptunes and $27^{+10}_{-8}$% for super-Neptunes from the 2-minute cadence data, and $22^{+8.6}_{-6.8}$% for mini-Neptunes and $13^{+3.9}_{-4.9}$% for super-Neptunes from FFI data. To independently validate our results, we compare our survey yield with the predicted planet yield assuming Kepler planet statistics. We consistently find a mild increase in the occurrence of super-Neptunes and a significant increase in the occurrence of Neptune-sized planets with orbital periods of 6.2-12 days when compared to their mature counterparts. The young planet distribution from our study is most consistent with evolution models describing the early contraction of hydrogen-dominated atmospheres undergoing atmospheric escape and inconsistent with heavier atmosphere models offering only mild radial contraction early on., Comment: Accepted to AJ. Updated version after proofing. 24 pages, 13 figures, 3 tables

The time complexity of the standard attention mechanism in a transformer scales quadratically with the length of the sequence. We introduce a method to reduce this to linear scaling with time, based on defining attention via latent vectors. The method is readily usable as a drop-in replacement for the standard attention mechanism. Our "Latte Transformer" model can be implemented for both bidirectional and unidirectional tasks, with the causal version allowing a recurrent implementation which is memory and time-efficient during inference of language generation tasks. Whilst next token prediction scales linearly with the sequence length for a standard transformer, a Latte Transformer requires constant time to compute the next token. The empirical performance of our method is comparable to standard attention, yet allows scaling to context windows much larger than practical in standard attention.

Retrieval Augmented Generation (RAG) has emerged as an effective solution for mitigating hallucinations in Large Language Models (LLMs). The retrieval stage in RAG typically involves a pre-trained embedding model, which converts queries and passages into vectors to capture their semantics. However, a standard pre-trained embedding model may exhibit sub-optimal performance when applied to specific domain knowledge, necessitating fine-tuning. This paper addresses scenarios where the embeddings are only available from a black-box model. We introduce Model augmented fine-tuning (Mafin) -- a novel approach for fine-tuning a black-box embedding model by augmenting it with a trainable embedding model. Our results demonstrate that Mafin significantly enhances the performance of the black-box embeddings by only requiring the training of a small augmented model. We validate the effectiveness of our method on both labeled and unlabeled datasets, illustrating its broad applicability and efficiency.

As large language models (LLMs) become more capable, fine-tuning techniques for aligning with human intent are increasingly important. A key consideration for aligning these models is how to most effectively use human resources, or model resources in the case where LLMs themselves are used as oracles. Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most prominent example of such a technique, but is complex and often unstable. Direct Preference Optimization (DPO) has recently been proposed as a simpler and more stable alternative. In this work, we develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model and a measure of certainty of the implicit preference model optimized by DPO. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data., Comment: 13 pages, 5 figures, 6 tables

Algorithm evaluation and comparison are fundamental questions in machine learning and statistics -- how well does an algorithm perform at a given modeling task, and which algorithm performs best? Many methods have been developed to assess algorithm performance, often based around cross-validation type strategies, retraining the algorithm of interest on different subsets of the data and assessing its performance on the held-out data points. Despite the broad use of such procedures, the theoretical properties of these methods are not yet fully understood. In this work, we explore some fundamental limits for answering these questions with limited amounts of data. In particular, we make a distinction between two questions: how good is an algorithm $A$ at the problem of learning from a training set of size $n$, versus, how good is a particular fitted model produced by running $A$ on a particular training data set of size $n$? Our main results prove that, for any test that treats the algorithm $A$ as a ``black box'' (i.e., we can only study the behavior of $A$ empirically), there is a fundamental limit on our ability to carry out inference on the performance of $A$, unless the number of available data points $N$ is many times larger than the sample size $n$ of interest. (On the other hand, evaluating the performance of a particular fitted model is easy as long as a holdout data set is available -- that is, as long as $N-n$ is not too small.) We also ask whether an assumption of algorithmic stability might be sufficient to circumvent this hardness result. Surprisingly, we find that this is not the case: the same hardness result still holds for the problem of evaluating the performance of $A$, aside from a high-stability regime where fitted models are essentially nonrandom. Finally, we also establish similar hardness results for the problem of comparing multiple algorithms.

The inadequate mixing of conventional Markov Chain Monte Carlo (MCMC) methods for multi-modal distributions presents a significant challenge in practical applications such as Bayesian inference and molecular dynamics. Addressing this, we propose Diffusive Gibbs Sampling (DiGS), an innovative family of sampling methods designed for effective sampling from distributions characterized by distant and disconnected modes. DiGS integrates recent developments in diffusion models, leveraging Gaussian convolution to create an auxiliary noisy distribution that bridges isolated modes in the original space and applying Gibbs sampling to alternately draw samples from both spaces. A novel Metropolis-within-Gibbs scheme is proposed to enhance mixing in the denoising sampling step. DiGS exhibits a better mixing property for sampling multi-modal distributions than state-of-the-art methods such as parallel tempering, attaining substantially improved performance across various tasks, including mixtures of Gaussians, Bayesian neural networks and molecular dynamics., Comment: Accepted for publication at ICML 2024. Code available: https://github.com/Wenlin-Chen/DiGS

We introduce a method for online conformal prediction with decaying step sizes. Like previous methods, ours possesses a retrospective guarantee of coverage for arbitrary sequences. However, unlike previous methods, we can simultaneously estimate a population quantile when it exists. Our theory and experiments indicate substantially improved practical properties: in particular, when the distribution is stable, the coverage is close to the desired level for every time point, not just on average over the observed sequence.

Conformal prediction (CP) is a method for constructing a prediction interval around the output of a fitted model, whose validity does not rely on the model being correct--the CP interval offers a coverage guarantee that is distribution-free, but relies on the training data being drawn from the same distribution as the test data. A recent variant, weighted conformal prediction (WCP), reweights the method to allow for covariate shift between the training and test distributions. However, WCP requires knowledge of the nature of the covariate shift-specifically,the likelihood ratio between the test and training covariate distributions. In practice, since this likelihood ratio is estimated rather than known exactly, the coverage guarantee may degrade due to the estimation error. In this paper, we consider a special scenario where observations belong to a finite number of groups, and these groups determine the covariate shift between the training and test distributions-for instance, this may arise if the training set is collected via stratified sampling. Our results demonstrate that in this special case, the predictive coverage guarantees of WCP can be drastically improved beyond the bounds given by existing estimation error bounds.

Low-temperature scanning probe microscopes (SPMs) are critical for the study of quantum materials and quantum information science. Due to the rising costs of helium, cryogen-free cryostats have become increasingly desirable. However, they typically suffer from comparatively worse vibrations than cryogen-based systems, necessitating the understanding and mitigation of vibrations for SPM applications. Here we demonstrate the construction of two cryogen-free dilution refrigerator SPMs with minimal modifications to the factory default and we systematically characterize their vibrational performance. We measure the absolute vibrations at the microscope stage with geophones, and use both microwave impedance microscopy and a scanning single electron transistor to independently measure tip-sample vibrations. Additionally, we implement customized filtering and thermal anchoring schemes, and characterize the cooling power at the scanning stage and the tip electron temperature. This work serves as a reference to researchers interested in cryogen-free SPMs, as such characterization is not standardized in the literature or available from manufacturers.