Search

In machine learning research, it is common to evaluate algorithms via their performance on standard benchmark datasets. While a growing body of work establishes guidelines for -- and levies criticisms at -- data and benchmarking practices in machine learning, comparatively less attention has been paid to the data repositories where these datasets are stored, documented, and shared. In this paper, we analyze the landscape of these $\textit{benchmark data repositories}$ and the role they can play in improving benchmarking. This role includes addressing issues with both datasets themselves (e.g., representational harms, construct validity) and the manner in which evaluation is carried out using such datasets (e.g., overemphasis on a few datasets and metrics, lack of reproducibility). To this end, we identify and discuss a set of considerations surrounding the design and use of benchmark data repositories, with a focus on improving benchmarking practices in machine learning., Comment: Accepted to NeurIPS Datasets and Benchmarks 2024

Stein variational gradient descent (SVGD) [Liu and Wang, 2016] performs approximate Bayesian inference by representing the posterior with a set of particles. However, SVGD suffers from variance collapse, i.e. poor predictions due to underestimating uncertainty [Ba et al., 2021], even for moderately-dimensional models such as small Bayesian neural networks (BNNs). To address this issue, we generalize SVGD by letting each particle parameterize a component distribution in a mixture model. Our method, Stein Mixture Inference (SMI), optimizes a lower bound to the evidence (ELBO) and introduces user-specified guides parameterized by particles. SMI extends the Nonlinear SVGD framework [Wang and Liu, 2019] to the case of variational Bayes. SMI effectively avoids variance collapse, judging by a previously described test developed for this purpose, and performs well on standard data sets. In addition, SMI requires considerably fewer particles than SVGD to accurately estimate uncertainty for small BNNs. The synergistic combination of NSVGD, ELBO optimization and user-specified guides establishes a promising approach towards variational Bayesian inference in the case of tall and wide data.

Continuous-time event sequences, in which events occur at irregular intervals, are ubiquitous across a wide range of industrial and scientific domains. The contemporary modeling paradigm is to treat such data as realizations of a temporal point process, and in machine learning it is common to model temporal point processes in an autoregressive fashion using a neural network. While autoregressive models are successful in predicting the time of a single subsequent event, their performance can be unsatisfactory in forecasting longer horizons due to cascading errors. We propose EventFlow, a non-autoregressive generative model for temporal point processes. Our model builds on the flow matching framework in order to directly learn joint distributions over event times, side-stepping the autoregressive process. EventFlow is likelihood-free, easy to implement and sample from, and either matches or surpasses the performance of state-of-the-art models in both unconditional and conditional generation tasks on a set of standard benchmarks

A generative diffusion model is used to produce probabilistic ensembles of precipitation intensity maps at the 1-hour 5-km resolution. The generation is conditioned on infrared and microwave radiometric measurements from the GOES and DMSP satellites and is trained with merged ground radar and gauge data over southeastern United States. The generated precipitation maps reproduce the spatial autocovariance and other multiscale statistical properties of the gauge-radar reference fields on average. Conditioning the generation on the satellite measurements allows us to constrain the magnitude and location of each generated precipitation feature. The mean of the 128- member ensemble shows high spatial coherence with the reference fields with 0.82 linear correlation between the two. On average, the coherence between any two ensemble members is approximately the same as the coherence between any ensemble member and the ground reference, attesting that the ensemble dispersion is a proper measure of the estimation uncertainty. From the generated ensembles we can easily derive the probability of the precipitation intensity exceeding any given intensity threshold, at the 5-km resolution of the generation or at any desired aggregated resolution.

Ionising radiation interactions in matter can trigger a cascade of processes that underpin long-lived damage in the medium. To date, however, a lack of suitable methodologies has precluded our ability to understand the role that material nanostructure plays in this cascade. Here, we use transient photoabsorption to track the lifetime of free electrons (t_c) in bulk and nanostructured SiO2 (aerogel) irradiated by picosecond-scale (10^-12 s) bursts of X-rays and protons from a laser-driven accelerator. Optical streaking reveals a sharp increase in t_c from < 1 ps to > 50 ps over a narrow average density (p_av) range spanning the expected phonon-fracton crossover in aerogels. Numerical modelling suggests that this discontinuity can be understood by a quenching of rapid, phonon-assisted recovery in irradiated nanostructured SiO_2. This is shown to lead to an extended period of enhanced energy density in the excited electron population. Overall, these results open a direct route to tracking how low-level processes in complex systems can underpin macroscopically observed phenomena and, importantly, the conditions that permit them to emerge.

We say that a family $\mathcal{W}$ of strings over $\Sigma^+$ forms a Unique Maximal Factorization Family (UMFF) if and only if every $w \in \mathcal{W}$ has a unique maximal factorization. Further, an UMFF $\mathcal{W}$ is called a circ-UMFF whenever it contains exactly one rotation of every primitive string $x \in \Sigma^+$. $V$-order is a non-lexicographical total ordering on strings that determines a circ-UMFF. In this paper we propose a generalization of circ-UMFF called the substring circ-UMFF and extend combinatorial research on $V$-order by investigating connections to Lyndon words. Then we extend these concepts to any total order. Applications of this research arise in efficient text indexing, compression, and search problems., Comment: 30 pages

There has been some concern about the impact of predatory publishers on scientific research for some time. Recently, publishers that might previously have been considered `predatory' have established their bona fides, at least to the extent that they are included in citation impact scores such as the field-weighted citation impact (FWCI). These are sometimes called `grey' publishers (MDPI, Frontiers, Hindawi). In this paper, we show that the citation landscape for these grey publications is significantly different from the mainstream landscape and that affording publications in these venues the same status as publications in mainstream journals may significantly distort metrics such as the FWCI., Comment: 12 pages, 8 figures

Object detection as part of computer vision can be crucial for traffic management, emergency response, autonomous vehicles, and smart cities. Despite significant advances in object detection, detecting small objects in images captured by distant cameras remains challenging due to their size, distance from the camera, varied shapes, and cluttered backgrounds. To address these challenges, we propose Small Object Detection YOLOv8 (SOD-YOLOv8), a novel model specifically designed for scenarios involving numerous small objects. Inspired by Efficient Generalized Feature Pyramid Networks (GFPN), we enhance multi-path fusion within YOLOv8 to integrate features across different levels, preserving details from shallower layers and improving small object detection accuracy. Also, A fourth detection layer is added to leverage high-resolution spatial information effectively. The Efficient Multi-Scale Attention Module (EMA) in the C2f-EMA module enhances feature extraction by redistributing weights and prioritizing relevant features. We introduce Powerful-IoU (PIoU) as a replacement for CIoU, focusing on moderate-quality anchor boxes and adding a penalty based on differences between predicted and ground truth bounding box corners. This approach simplifies calculations, speeds up convergence, and enhances detection accuracy. SOD-YOLOv8 significantly improves small object detection, surpassing widely used models in various metrics, without substantially increasing computational cost or latency compared to YOLOv8s. Specifically, it increases recall from 40.1\% to 43.9\%, precision from 51.2\% to 53.9\%, $\text{mAP}_{0.5}$ from 40.6\% to 45.1\%, and $\text{mAP}_{0.5:0.95}$ from 24\% to 26.6\%. In dynamic real-world traffic scenes, SOD-YOLOv8 demonstrated notable improvements in diverse conditions, proving its reliability and effectiveness in detecting small objects even in challenging environments., Comment: 15 pages, 14 figures

Representation learning has emerged as a powerful paradigm for extracting valuable latent features from complex, high-dimensional data. In financial domains, learning informative representations for assets can be used for tasks like sector classification, and risk management. However, the complex and stochastic nature of financial markets poses unique challenges. We propose a novel contrastive learning framework to generate asset embeddings from financial time series data. Our approach leverages the similarity of asset returns over many subwindows to generate informative positive and negative samples, using a statistical sampling strategy based on hypothesis testing to address the noisy nature of financial data. We explore various contrastive loss functions that capture the relationships between assets in different ways to learn a discriminative representation space. Experiments on real-world datasets demonstrate the effectiveness of the learned asset embeddings on benchmark industry classification and portfolio optimization tasks. In each case our novel approaches significantly outperform existing baselines highlighting the potential for contrastive learning to capture meaningful and actionable relationships in financial data., Comment: 9 pages, 4 figures, 4 tables

_Uncertainty expressions_ such as "probably" or "highly unlikely" are pervasive in human language. While prior work has established that there is population-level agreement in terms of how humans quantitatively interpret these expressions, there has been little inquiry into the abilities of language models in the same context. In this paper, we investigate how language models map linguistic expressions of uncertainty to numerical responses. Our approach assesses whether language models can employ theory of mind in this setting: understanding the uncertainty of another agent about a particular statement, independently of the model's own certainty about that statement. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. However, we observe systematically different behavior depending on whether a statement is actually true or false. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge (as compared to humans). These findings raise important questions and have broad implications for human-AI and AI-AI communication., Comment: Accepted at EMNLP 2024 (Main)

Conformal prediction provides machine learning models with prediction sets that offer theoretical guarantees, but the underlying assumption of exchangeability limits its applicability to time series data. Furthermore, existing approaches struggle to handle multi-step ahead prediction tasks, where uncertainty estimates across multiple future time points are crucial. We propose JANET (Joint Adaptive predictioN-region Estimation for Time-series), a novel framework for constructing conformal prediction regions that are valid for both univariate and multivariate time series. JANET generalises the inductive conformal framework and efficiently produces joint prediction regions with controlled K-familywise error rates, enabling flexible adaptation to specific application needs. Our empirical evaluation demonstrates JANET's superior performance in multi-step prediction tasks across diverse time series datasets, highlighting its potential for reliable and interpretable uncertainty quantification in sequential data., Comment: Alternate Title: Conformalised Joint Prediction Region for Time Series

Large language models (LLMs) have shown their potential in long-context understanding and mathematical reasoning. In this paper, we study the problem of using LLMs to detect tabular anomalies and show that pre-trained LLMs are zero-shot batch-level anomaly detectors. That is, without extra distribution-specific model fitting, they can discover hidden outliers in a batch of data, demonstrating their ability to identify low-density data regions. For LLMs that are not well aligned with anomaly detection and frequently output factual errors, we apply simple yet effective data-generating processes to simulate synthetic batch-level anomaly detection datasets and propose an end-to-end fine-tuning strategy to bring out the potential of LLMs in detecting real anomalies. Experiments on a large anomaly detection benchmark (ODDS) showcase i) GPT-4 has on-par performance with the state-of-the-art transductive learning-based anomaly detection methods and ii) the efficacy of our synthetic dataset and fine-tuning strategy in aligning LLMs to this task., Comment: accepted at the Anomaly Detection with Foundation Models workshop

DAO Governance is currently broken. We survey the state of the art and find worrying conclusions. Vote buying, vote selling and coercion are easy. The wealthy rule, decentralisation is a myth. Hostile take-overs are incentivised. Ballot secrecy is non-existent or short lived, despite being a human right. Verifiablity is achieved at the expense of privacy. These privacy concerns are highlighted with case study analyses of Vocdoni's governance protocol. This work presents two contributions: firstly a review of current DAO governance protocols, and secondly, an illustration of their vulnerabilities, showcasing the privacy and security threats these entail.

A standard question in real algebraic geometry is to compute the number of connected components of a real algebraic variety in affine space. By adapting an approach for determining connectivity in complements of real hypersurfaces by Hong, Rohal, Safey El Din, and Schost, algorithms are presented for computing the number of connected components, the Euler characteristic, and deciding the connectivity between two points for a smooth manifold arising as the complement of a real hypersurface of a real algebraic variety. When taking such real hypersurface to be the set of singular points, this yields an approach for determining smooth connectivity in a real algebraic variety. The method is based upon gradient ascent/descent paths on the real algebraic variety and several examples are included to demonstrate the approach., Comment: 19 pages, 7 figures

We recently described the results of an initial search through TESS Sector 61 for free-floating planets. In this short note, we provide important context for our results and clarify the language used in our initial manuscript to ensure that our intended message is appropriately conveyed., Comment: 2 pages; note regarding arXiv:2404.11666

Protein Language Models (PLMs) have emerged as performant and scalable tools for predicting the functional impact and clinical significance of protein-coding variants, but they still lag experimental accuracy. Here, we present a novel fine-tuning approach to improve the performance of PLMs with experimental maps of variant effects from Deep Mutational Scanning (DMS) assays using a Normalised Log-odds Ratio (NLR) head. We find consistent improvements in a held-out protein test set, and on independent DMS and clinical variant annotation benchmarks from ProteinGym and ClinVar. These findings demonstrate that DMS is a promising source of sequence diversity and supervised training data for improving the performance of PLMs for variant effect prediction., Comment: Machine Learning for Genomics Explorations workshop at ICLR 2024

Large-scale geolocation telematics data acquired from connected vehicles has the potential to significantly enhance mobility infrastructures and operational systems within smart cities. To effectively utilize this data, it is essential to accurately match the geolocation data to the road segments. However, this matching is often not trivial due to the low sampling rate and errors exacerbated by multipath effects in urban environments. Traditionally, statistical modeling techniques such as Hidden-Markov models incorporating domain knowledge into the matching process have been extensively used for map-matching tasks. However, rule-based map-matching tasks are noise-sensitive and inefficient in processing large-scale trajectory data. Deep learning techniques directly learn the relationship between observed data and road networks from the data, often without the need for hand-crafted rules or domain knowledge. This renders them an efficient approach for map-matching large-scale datasets and more robust to the noise. This paper introduces a deep-learning model, specifically the transformer-based encoder-decoder model, to perform as a surrogate for offline map-matching algorithms. The encoder-decoder architecture initially encodes the series of noisy GPS points into a representation that automatically captures autoregressive behavior and spatial correlations between GPS points. Subsequently, the decoder associates data points with the road network features and thus transforms these representations into a sequence of road segments. The model is trained and evaluated using GPS traces collected in Manhattan, New York. Achieving an accuracy of 75%, transformer-based encoder-decoder models extensively employed in natural language processing presented a promising performance for translating noisy GPS data to the navigated routes in urban road networks., Comment: 15 pages, 10 figures

Though free-floating planets (FFPs) that have been ejected from their natal star systems may outpopulate their bound counterparts in the terrestrial-mass range, they remain one of the least explored exoplanet demographics. Due to their negligible electromagnetic emission at all wavelengths, the only observational technique able to detect these worlds is gravitational microlensing. Microlensing by terrestrial-mass FFPs induces rare, short-duration magnifications of background stars, requiring high-cadence, wide-field surveys to detect these events. The Transiting Exoplanet Survey Satellite (TESS), though designed to detect close-bound exoplanets via the transit technique, boasts a cadence as short as 200 seconds and has monitored hundreds of millions of stars, making it well-suited to search for short-duration microlensing events as well. We have used existing data products from the TESS Quick-Look Pipeline (QLP) to perform a preliminary search for FFP microlensing candidates in 1.3 million light curves from TESS Sector 61. We find one compelling candidate associated with TIC-107150013, a source star at $d_s = 3.194$ kpc. The event has a duration $t_E = 0.074^{+0.002}_{-0.002}$ days and shows prominent finite-source features ($\rho = 4.55^{+0.08}_{-0.07}$), making it consistent with an FFP in the terrestrial-mass range. This exciting result indicates that our ongoing search through all TESS sectors has the opportunity to shed new light on this enigmatic population of worlds., Comment: 10 pages, 7 figures, submitted to MNRAS

We study the geometry of conditional optimal transport (COT) and prove a dynamical formulation which generalizes the Benamou-Brenier Theorem. Equipped with these tools, we propose a simulation-free flow-based method for conditional generative modeling. Our method couples an arbitrary source distribution to a specified target distribution through a triangular COT plan, and a conditional generative model is obtained by approximating the geodesic path of measures induced by this COT plan. Our theory and methods are applicable in infinite-dimensional settings, making them well suited for a wide class of Bayesian inverse problems. Empirically, we demonstrate that our method is competitive on several challenging conditional generation tasks, including an infinite-dimensional inverse problem.

The tidal disruption of old, compact stellar structures provides strong constraints on macroscopic dark matter candidates such as primordial black holes. In view of recent, new observational data on the Eridanus II dwarf galaxy and on its central stellar cluster, we employ, for the first time, $N$-body simulations to assess the impact of compact massive dark matter candidates on the gravitational stability of the cluster. We find evidence that such candidates must be lighter than about one solar mass if they constitute the totality of the dark matter. We additionally derive robust constraints on the fraction of the dark matter in macroscopic objects as a function of mass, by suitably modeling the remainder of the dark matter as standard fluid-like cold dark matter., Comment: 17 pages, 6 figures

We illustrate how purpose-specific, graphical modeling enables application experts with different levels of expertise to collaboratively design and then produce complex applications using their individual, purpose-specific modeling language. Our illustration includes seven graphical Integrated Modeling Environments (IMEs) that support full code generation, as well as four browser-based applications that were modeled and then fully automatically generated and produced using DIME, our most complex graphical IME. While the seven IMEs were chosen to illustrate the types of languages we support with our Language-Driven Engineering (LDE) approach, the four DIME products were chosen to give an impression of the power of our LDE-generated IMEs. In fact, Equinocs, Springer Nature's future editorial system for proceedings, is also being fully automatically generated and then deployed at their Dordrecht site using a deployment pipeline generated with Rig, one of the IMEs presented. Our technology is open source and the products presented are currently in use., Comment: 43 pages, 30 figures

For large language models (LLMs) to be trusted by humans they need to be well-calibrated in the sense that they can accurately assess and communicate how likely it is that their predictions are correct. Recent work has focused on the quality of internal LLM confidence assessments, but the question remains of how well LLMs can communicate this internal model confidence to human users. This paper explores the disparity between external human confidence in an LLM's responses and the internal confidence of the model. Through experiments involving multiple-choice questions, we systematically examine human users' ability to discern the reliability of LLM outputs. Our study focuses on two key areas: (1) assessing users' perception of true LLM confidence and (2) investigating the impact of tailored explanations on this perception. The research highlights that default explanations from LLMs often lead to user overestimation of both the model's confidence and its' accuracy. By modifying the explanations to more accurately reflect the LLM's internal confidence, we observe a significant shift in user perception, aligning it more closely with the model's actual confidence levels. This adjustment in explanatory approach demonstrates potential for enhancing user trust and accuracy in assessing LLM outputs. The findings underscore the importance of transparent communication of confidence levels in LLMs, particularly in high-stakes applications where understanding the reliability of AI-generated information is essential., Comment: 27 pages, 10 figures

Neural marked temporal point processes have been a valuable addition to the existing toolbox of statistical parametric models for continuous-time event data. These models are useful for sequences where each event is associated with a single item (a single type of event or a "mark") -- but such models are not suited for the practical situation where each event is associated with a set of items. In this work, we develop a general framework for modeling set-valued data in continuous-time, compatible with any intensity-based recurrent neural point process model. In addition, we develop inference methods that can use such models to answer probabilistic queries such as "the probability of item $A$ being observed before item $B$," conditioned on sequence history. Computing exact answers for such queries is generally intractable for neural models due to both the continuous-time nature of the problem setting and the combinatorially-large space of potential outcomes for each event. To address this, we develop a class of importance sampling methods for querying with set-based sequences and demonstrate orders-of-magnitude improvements in efficiency over direct sampling via systematic experiments with four real-world datasets. We also illustrate how to use this framework to perform model selection using likelihoods that do not involve one-step-ahead prediction., Comment: Oral presentation at AISTATS 2024

The Roman Space Telescope's Galactic Bulge Time Domain Survey will constitute the most sensitive microlensing survey of the Galactic Bulge to date, opening up new opportunities to search for dark matter (DM). Many extensions of the Standard Model predict the formation of extended DM substructures, such as DM subhalos, boson/axion stars, and halo-dressed primordial black holes. We demonstrate that for such targets, Roman will be sensitive to a broad parameter space up to four orders of magnitude below existing constraints. Our analysis can be readily applied to other extended DM configurations as well., Comment: 9 pages, 2 figures; v2 updated to match accepted ApJL version

As per Adusumilli (2015),'70% of corporate business systems today are legacy applications. Recent statistics prove that over 60% of IT budget is spent on maintaining these Legacy systems, showing the rigidity and the fragile nature of these systems.' Usually, testing is included during the software development cycle, using testing techniques such as unit testing, integration testing, and system testing before releasing the product. After the software product is released to production, no additional testing is done; the testing process is back to the table only when modifications are made. Techniques such as regression testing are included to ensure the changes do not affect existing functionality, but testing nonfunctional features that are rarely included in such regression tests' scope. Schrader (2021) affirms that 'legacy systems are often maintained only to ensure function,' and IT organizations may fail to consider the cybersecurity perspective to remain secure. Legacy systems are a high-risk component for the organization that must be carefully considered when structuring a cyber security strategy. This paper aims to help the reader understand some measures that can be taken to secure legacy systems, explaining what penetration testing is and how this testing technique can help secure legacy systems. Keywords: Testing, legacy, security, risks, prevention, mitigation, pentesting.

Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.

We formalise definitions of ballot secrecy and ballot independence by Smyth, JCS'21 as indistinguishability games in the computational model of security. These definitions improve upon Smyth, draft '21 to consider a wider class of voting systems. Both Smyth, JCS'21 and Smyth, draft '21 improve on earlier works by considering a more realistic adversary model wherein they have access to the ballot collection. We prove that ballot secrecy implies ballot independence. We say ballot independence holds if a system has non-malleable ballots. We construct games for ballot secrecy and non-malleability and show that voting schemes with malleable ballots do not preserve ballot secrecy. We demonstrate that Helios does not satisfy our definition of ballot secrecy. Furthermore, the Python framework we constructed for our case study shows that if an attack exists against non-malleability, this attack can be used to break ballot secrecy.

Gravitational microlensing is one of the strongest observational techniques to observe non-luminous astrophysical bodies. Existing microlensing observations provide tantalizing evidence of a population of low-mass objects whose origin is unknown. These events may be caused by terrestrial-mass free-floating planets or by exotic objects such as primordial black holes. However, the nature of these objects cannot be resolved on an event-by-event basis, as the induced light curve is degenerate for lensing bodies of identical mass. One must instead statistically compare \textit{distributions} of lensing events to determine the nature of the lensing population. While existing surveys lack the statistics required to identify multiple subpopulations of lenses, this will change with the launch of the Nancy Grace Roman Space Telescope. Roman's Galactic Bulge Time Domain Survey is expected to observe hundreds of low-mass microlensing events, enabling a robust statistical characterization of this population. In this paper, we show that by exploiting features in the distribution of lensing event durations, Roman will be sensitive to a subpopulation of primordial black holes hidden amongst a background of free-floating planets. Roman's reach will extend to primordial black hole dark matter fractions as low as $f_\text{PBH} = 10^{-4}$ at peak sensitivity, and will be able to conclusively determine the origin of existing ultrashort-timescale microlensing events. A positive detection would provide evidence that a significant fraction of the cosmological dark matter consists of macroscopic, non-luminous objects., Comment: 11 pages, 6 figures