Search

This paper presents a novel approach utilizing a scalable neural decoder application-specific integrated circuit (ASIC) based on metal oxide memristors in a 180nm CMOS technology. The ASIC architecture employs in-memory computing with memristor crossbars for efficient vector-matrix multiplications (VMM). The ASIC decoder architecture includes an input layer implemented with a VMM and an analog sigmoid activation function, a recurrent layer with analog memory, and an output layer with a VMM and a threshold activation function. Cryogenic characterization of the ASIC is conducted, demonstrating its performance at both room temperature and cryogenic temperatures down to 1.2K. Results indicate stable activation function shapes and pulse responses at cryogenic temperatures. Moreover, power consumption measurements reveal consistent behavior at room and cryogenic temperatures. Overall, this study lays the foundation for developing efficient and scalable neural decoders for quantum error correction in cryogenic environments.

The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs) for web interaction tasks. Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. BrowserGym aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. Combined with AgentLab, a complementary framework that aids in agent creation, testing, and analysis, BrowserGym offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. This standardized approach seeks to reduce the time and complexity of developing web agents, supporting more reliable comparisons and facilitating in-depth analysis of agent behaviors, and could result in more adaptable, capable agents, ultimately accelerating innovation in LLM-driven automation. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across all benchmarks currently available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.

Causal discovery aims to automatically uncover causal relationships from data, a capability with significant potential across many scientific disciplines. However, its real-world applications remain limited. Current methods often rely on unrealistic assumptions and are evaluated only on simple synthetic toy datasets, often with inadequate evaluation metrics. In this paper, we substantiate these claims by performing a systematic review of the recent causal discovery literature. We present applications in biology, neuroscience, and Earth sciences - fields where causal discovery holds promise for addressing key challenges. We highlight available simulated and real-world datasets from these domains and discuss common assumption violations that have spurred the development of new methods. Our goal is to encourage the community to adopt better evaluation practices by utilizing realistic datasets and more adequate metrics., Comment: 39 pages, 8 figures

Forecasting is a critical task in decision making across various domains. While numerical data provides a foundation, it often lacks crucial context necessary for accurate predictions. Human forecasters frequently rely on additional information, such as background knowledge or constraints, which can be efficiently communicated through natural language. However, the ability of existing forecasting models to effectively integrate this textual information remains an open question. To address this, we introduce "Context is Key" (CiK), a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, requiring models to integrate both modalities. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters, and propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings. By presenting this benchmark, we aim to advance multimodal forecasting, promoting models that are both accurate and accessible to decision-makers with varied technical expertise. The benchmark can be visualized at https://servicenow.github.io/context-is-key-forecasting/v0/ ., Comment: Preprint; under review. First two authors contributed equally

Reconstruction functions are pivotal in sample compression theory, a framework for deriving tight generalization bounds. From a small sample of the training set (the compression set) and an optional stream of information (the message), they recover a predictor previously learned from the whole training set. While usually fixed, we propose to learn reconstruction functions. To facilitate the optimization and increase the expressiveness of the message, we derive a new sample compression generalization bound for real-valued messages. From this theoretical analysis, we then present a new hypernetwork architecture that outputs predictors with tight generalization guarantees when trained using an original meta-learning framework. The results of promising preliminary experiments are then reported., Comment: Accepted at the NeurIPS 2024 workshop on Compression in Machine Learning

Scientific research often seeks to understand the causal structure underlying high-level variables in a system. For example, climate scientists study how phenomena, such as El Ni\~no, affect other climate processes at remote locations across the globe. However, scientists typically collect low-level measurements, such as geographically distributed temperature readings. From these, one needs to learn both a mapping to causally-relevant latent variables, such as a high-level representation of the El Ni\~no phenomenon and other processes, as well as the causal model over them. The challenge is that this task, called causal representation learning, is highly underdetermined from observational data alone, requiring other constraints during learning to resolve the indeterminacies. In this work, we consider a temporal model with a sparsity assumption, namely single-parent decoding: each observed low-level variable is only affected by a single latent variable. Such an assumption is reasonable in many scientific applications that require finding groups of low-level variables, such as extracting regions from geographically gridded measurement data in climate research or capturing brain regions from neural activity data. We demonstrate the identifiability of the resulting model and propose a differentiable method, Causal Discovery with Single-parent Decoding (CDSD), that simultaneously learns the underlying latents and a causal graph over them. We assess the validity of our theoretical results using simulated data and showcase the practical validity of our method in an application to real-world data from the climate science field., Comment: 33 pages, 17 figures

Spin-based semiconductor qubits hold promise for scalable quantum computing, yet they require reliable autonomous calibration procedures. This study presents an experimental demonstration of online single-dot charge autotuning using a convolutional neural network integrated into a closed-loop calibration system. The autotuning algorithm explores the gates' voltage space to localize charge transition lines, thereby isolating the one-electron regime without human intervention. In 20 experimental runs on a device cooled to 25mK, the method achieved a success rate of 95% in locating the target electron regime, highlighting the robustness of this method against noise and distribution shifts from the offline training set. Each tuning run lasted an average of 2 hours and 9 minutes, primarily due to the limited speed of the current measurement. This work validates the feasibility of machine learning-driven real-time charge autotuning for quantum dot devices, advancing the development toward the control of large qubit arrays., Comment: 6 pages (main) + 5 pages (supplementary)

We propose a mathematical walk around the gcd of the values $A(n)$ and $B(n)$ of two polynomials evaluated at an integer $n$. This is an opportunity to use a very powerful tool: the resultant., Comment: The final version will appear in the American Mathematical Monthly

Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We introduce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics.

The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress toward capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena/tree/workarena-plus-plus.

Heterogeneous systems with analog CMOS circuits integrated with nanoscale memristive devices enable efficient deployment of neural networks on neuromorphic hardware. CMOS Neuron with low footprint can emulate slow temporal dynamics by operating with extremely low current levels. Nevertheless, the current read from the memristive synapses can be higher by several orders of magnitude, and performing impedance matching between neurons and synapses is mandatory. In this paper, we implement an analog leaky integrate and fire (LIF) neuron with a voltage regulator and current attenuator for interfacing CMOS neurons with memristive synapses. In addition, the neuron design proposes a dual leakage that could enable the implementation of local learning rules such as voltage-dependent synaptic plasticity. We also propose a connection scheme to implement adaptive LIF neurons based on two-neuron interaction. The proposed circuits can be used to interface with a variety of synaptic devices and process signals of diverse temporal dynamics., Comment: Accepted to International Conference on Neuromorphic Systems (ICONS 2024)

This study presents a machine-learning-based procedure to automate the charge tuning of semiconductor spin qubits with minimal human intervention, addressing one of the significant challenges in scaling up quantum dot technologies. This method exploits artificial neural networks to identify noisy transition lines in stability diagrams, guiding a robust exploration strategy leveraging neural networks' uncertainty estimations. Tested across three distinct offline experimental datasets representing different single quantum dot technologies, the approach achieves over 99% tuning success rate in optimal cases, where more than 10% of the success is directly attributable to uncertainty exploitation. The challenging constraints of small training sets containing high diagram-to-diagram variability allowed us to evaluate the capabilities and limits of the proposed procedure., Comment: 12 pages (main) + 14 pages (supplementary)

The tetragonal heavy-fermion superconductor CeRh$_2$As$_2$ ($T_{\rm c}=0.3$ K) exhibits an exceptionally high critical field of 14 T for $\textbf{B} \parallel \textbf{c}$. It undergoes a field-driven first-order phase transition between superconducting (SC) states, potentially transitioning from spin-singlet to spin-triplet superconductivity. To elucidate the underlying pairing mechanism, we probe spin fluctuations in CeRh$_2$As$_2$ using neutron scattering. We find dynamic $(\pi,\pi)$ antiferromagnetic spin correlations with an anisotropic quasi-two-dimensional correlation volume. Our data place an upper limit of 0.31 $\mu_{\rm B}$ on the staggered magnetization of corresponding N\'{e}el orders at $T=0.08$ K. Density functional theory (DFT) calculations, treating Ce $4f$ electrons as core states, show that the AFM wave vector connects significant areas of the Fermi surface. Our findings show the dominant excitations in CeRh$_2$As$_2$ for $\hbar\omega< 1.2$~meV are magnetic and indicate superconductivity in CeRh$_2$As$_2$ is mediated by AFM spin fluctuations associated with a proximate quantum critical point., Comment: 7+6 pages, 4+4 figures

Cryogenic memristor-based DC sources offer a promising avenue for in situ biasing of quantum dot arrays. In this study, we present experimental results and discuss the scaling potential for such DC sources. We first demonstrate the operation of a commercial discrete operational amplifier down to 1.2K which is used on the DC source prototype. Then, the tunability of the memristor-based DC source is validated by performing several 250mV-DC sweeps with a resolution of 10mV at room temperature and at 1.2K. Additionally, the DC source prototype exhibits a limited output drift of $\approx1\mathrm{\mu Vs^{-1}}$ at 1.2K. This showcases the potential of memristor-based DC sources for quantum dot biasing. Limitations in power consumption and voltage resolution using discrete components highlight the need for a fully integrated and scalable complementary metal-oxide-semiconductor-based (CMOS-based) approach. To address this, we propose to monolithically co-integrate emerging non-volatile memories (eNVMs) and 65nm CMOS circuitry. Simulations reveal a reduction in power consumption, down to $\mathrm{10\mu W}$ per DC source and in footprint. This allows for the integration of up to one million eNVM-based DC sources at the 4.2K stage of a dilution fridge, paving the way for near term large-scale quantum computing applications.

Numerous decision-making tasks require estimating causal effects under interventions on different parts of a system. As practitioners consider using large language models (LLMs) to automate decisions, studying their causal reasoning capabilities becomes crucial. A recent line of work evaluates LLMs ability to retrieve commonsense causal facts, but these evaluations do not sufficiently assess how LLMs reason about interventions. Motivated by the role that interventions play in causal inference, in this paper, we conduct empirical analyses to evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention. We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types, and enable a study of intervention-based reasoning. These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts. We evaluate six LLMs on the benchmarks, finding that GPT models show promising accuracy at predicting the intervention effects., Comment: 17 pages

We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field., Comment: 21 pages, 11 figures, preprint

We present broadband dual frequency comb laser absorption measurements of 2% H$_2$O (natural isotopic abundance of 99.7% H$_2^{16}$O) in air from 6600-7650 cm$^{-1}$ (1307-1515 nm) with a spectral point spacing of 0.0068 cm$^{-1}$. Twenty-nine datasets were collected at temperatures between 300 and 1300 K ($\pm$0.82% average uncertainty) and pressures ranging from 20 to 600 Torr ($\pm$0.25%) with an average residual absorbance noise of 8.0E-4 across the spectrum for all measurements. We fit measurements using a quadratic speed-dependent Voigt profile to determine 7088 absorption parameters for 3366 individual transitions found in HITRAN2020. These measurements build on the line strength, line center, self-broadening, and self-shift parameters determined in the Part I companion of this work. Here we measure air-broadened width (with temperature- and speed-dependence) and air pressure shift (with temperature dependence) parameters. Various trends are explored for extrapolation to weak transitions that were not covered in this work. Improvements made in this work are predominantly due to the inclusion of air pressure shift temperature dependence values. In aggregate, these updates improved RMS absorbance error by a factor of 4.2 on average, and the remaining residual is predominantly spectral noise. This updated database improves high temperature spectroscopic knowledge across the 6600 7650 cm$^{-1}$ region of H$_2$O absorption., Comment: Database files available upon request. Will be included with published manuscript following review process

The extraction of a small number of relevant insights from vast amounts of data is a crucial component of data-driven decision-making. However, accomplishing this task requires considerable technical skills, domain expertise, and human labor. This study explores the potential of using Large Language Models (LLMs) to automate the discovery of insights in data, leveraging recent advances in reasoning and code generation techniques. We propose a new evaluation methodology based on a "capture the flag" principle, measuring the ability of such models to recognize meaningful and pertinent information (flags) in a dataset. We further propose two proof-of-concept agents, with different inner workings, and compare their ability to capture such flags in a real-world sales dataset. While the work reported here is preliminary, our results are sufficiently interesting to mandate future exploration by the community., Comment: 14 pages, 1 figure, Foundation Models for Decision Making Workshop at NeurIPS 2023

Over the past years, foundation models have caused a paradigm shift in machine learning due to their unprecedented capabilities for zero-shot and few-shot generalization. However, despite the success of foundation models in modalities such as natural language processing and computer vision, the development of foundation models for time series forecasting has lagged behind. We present Lag-Llama, a general-purpose foundation model for univariate probabilistic time series forecasting based on a decoder-only transformer architecture that uses lags as covariates. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities compared to a wide range of forecasting models on downstream datasets across domains. Moreover, when fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance, outperforming prior deep learning approaches, emerging as the best general-purpose model on average. Lag-Llama serves as a strong contender to the current state-of-art in time series forecasting and paves the way for future advancements in foundation models tailored to time series data., Comment: First two authors contributed equally. All data, models and code used are open-source. GitHub: https://github.com/time-series-foundation-models/lag-llama

The presence of competing interactions due to geometry leads to frustration in quantum spin models. As a consequence, the ground state of such systems often displays a large degeneracy that can be lifted due to thermal or quantum effects. One such example is the antiferromagnetic Ising model on the Kagome lattice. It was shown that while the same model on the triangular lattice is ordered at zero temperature for small transverse field due to an order by disorder mechanism, the Kagome lattice resists any such effects and exhibits only short range spin correlations and a trivial paramagnetic phase. We embed this model on the latest architecture of D-Wave's quantum annealer, the Advantage2 prototype, which uses the highly connected Zephyr graph. Using advanced embedding and calibration techniques, we are able to embed a Kagome lattice with mixed open and periodic boundary conditions of 231 sites on the full graph of the currently available prototype. Through forward annealing experiments, we show that under a finite longitudinal field the system exhibits a one-third magnetization plateau, consistent with a classical spin liquid state of reduced entropy. An anneal-pause-quench protocol is then used to extract an experimental ensemble of states resulting from the equilibration of the model at finite transverse and longitudinal field. This allows us to construct a partial phase diagram and confirm that the system exits the constrained Hilbert space of the classical spin liquid when subjected to a transverse field. We connect our results to previous theoretical results and quantum Monte Carlo simulation, which helps us confirm the validity of the quantum simulation realized here, thereby extracting insight into the performance of the D-Wave quantum annealer to simulate non-trivial quantum systems in equilibrium., Comment: 12 + 6 pages, 7 + 8 figures, updated paper to be published in PRB

We introduce a new model for multivariate probabilistic time series prediction, designed to flexibly address a range of tasks including forecasting, interpolation, and their combinations. Building on copula theory, we propose a simplified objective for the recently-introduced transformer-based attentional copulas (TACTiS), wherein the number of distributional parameters now scales linearly with the number of variables instead of factorially. The new objective requires the introduction of a training curriculum, which goes hand-in-hand with necessary changes to the original architecture. We show that the resulting model has significantly better training dynamics and achieves state-of-the-art performance across diverse real-world forecasting tasks, while maintaining the flexibility of prior work, such as seamless handling of unaligned and unevenly-sampled time series. Code is made available at https://github.com/ServiceNow/TACTiS., Comment: 28 pages, 15 figures, The Twelfth International Conference on Learning Representations (ICLR 2024)

The exploration of memristors' behavior at cryogenic temperatures has become crucial due to the growing interest in quantum computing and cryogenic electronics. In this context, our study focuses on the characterization at cryogenic temperatures (4.2 K) of TiO$_\textrm{2-x}$-based memristors fabricated with a CMOS-compatible etch-back process. We demonstrate a so-called cryogenic reforming (CR) technique performed at 4.2 K to overcome the well-known metal-insulator transition (MIT) which limits the analog behavior of memristors at low temperatures. This cryogenic reforming process was found to be reproducible and led to a durable suppression of the MIT. This process allowed to reduce by approximately 20% the voltages required to perform DC resistive switching at 4.2 K. Additionally, conduction mechanism studies of memristors before and after cryogenic reforming from 4.2 K to 300 K revealed different behaviors above 100 K, indicating a potential change in the conductive filament stoichiometry. The reformed devices exhibit a conductance level that is 50 times higher than ambient-formed memristor, and the conduction drop between 300 K and 4.2 K is 100 times smaller, indicating the effectiveness of the reforming process. More importantly, CR enables analog programming at 4.2 K with typical read voltages. Suppressing the MIT improved the analog switching dynamics of the memristor leading to approximately 250% larger on/off ratios during long-term depression (LTD)/long-term potentiation (LTP) resistance tuning. This enhancement opens up the possibility of using TiO$_{\textrm{2-x}}$-based memristors to be used as synapses in neuromorphic computing at cryogenic temperatures.

Neural decoders for quantum error correction (QEC) rely on neural networks to classify syndromes extracted from error correction codes and find appropriate recovery operators to protect logical information against errors. Its ability to adapt to hardware noise and long-term drifts make neural decoders a promising candidate for inclusion in a fault-tolerant quantum architecture. However, given their limited scalability, it is prudent that small-scale (local) neural decoders are treated as first stages of multi-stage decoding schemes for fault-tolerant quantum computers with millions of qubits. In this case, minimizing the decoding time to match the stabilization measurements frequency and a tight co-integration with the QPUs is highly desired. Cryogenic realizations of neural decoders can not only improve the performance of higher stage decoders, but they can minimize communication delays, and alleviate wiring bottlenecks. In this work, we design and analyze a neural decoder based on an in-memory computation (IMC) architecture, where crossbar arrays of resistive memory devices are employed to both store the synaptic weights of the neural decoder and perform analog matrix-vector multiplications. In simulations supported by experimental measurements, we investigate the impact of TiOx-based memristive devices' non-idealities on decoding fidelity. We develop hardware-aware re-training methods to mitigate the fidelity loss, restoring the ideal decoder's pseudo-threshold for the distance-3 surface code. This work provides a pathway to scalable, fast, and low-power cryogenic IMC hardware for integrated fault-tolerant QEC.

The practical utility of causality in decision-making is widespread and brought about by the intertwining of causal discovery and causal inference. Nevertheless, a notable gap exists in the evaluation of causal discovery methods, where insufficient emphasis is placed on downstream inference. To address this gap, we evaluate seven established baseline causal discovery methods including a newly proposed method based on GFlowNets, on the downstream task of treatment effect estimation. Through the implementation of a distribution-level evaluation, we offer valuable and unique insights into the efficacy of these causal discovery methods for treatment effect estimation, considering both synthetic and real-world scenarios, as well as low-data scenarios. The results of our study demonstrate that some of the algorithms studied are able to effectively capture a wide range of useful and diverse ATE modes, while some tend to learn many low-probability modes which impacts the (unrelaxed) recall and precision., Comment: Peer-reviewed and Accepted to ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling

Understanding the causal relationships that underlie a system is a fundamental prerequisite to accurate decision-making. In this work, we explore how expert knowledge can be used to improve the data-driven identification of causal graphs, beyond Markov equivalence classes. In doing so, we consider a setting where we can query an expert about the orientation of causal relationships between variables, but where the expert may provide erroneous information. We propose strategies for amending such expert knowledge based on consistency properties, e.g., acyclicity and conditional independencies in the equivalence class. We then report a case study, on real data, where a large language model is used as an imperfect expert., Comment: Peer reviewed and accepted for presentation at the Structured Probabilistic Inference & Generative Modeling (SPIGM) workshop at ICML 2023, Hawaii, USA

Rule-based models, such as decision trees, appeal to practitioners due to their interpretable nature. However, the learning algorithms that produce such models are often vulnerable to spurious associations and thus, they are not guaranteed to extract causally-relevant insights. In this work, we build on ideas from the invariant causal prediction literature to propose Invariant Causal Set Covering Machines, an extension of the classical Set Covering Machine algorithm for conjunctions/disjunctions of binary-valued rules that provably avoids spurious associations. We demonstrate both theoretically and empirically that our method can identify the causal parents of a variable of interest in polynomial time.

Recent progress in self-supervision has shown that pre-training large neural networks on vast amounts of unsupervised data can lead to substantial increases in generalization to downstream tasks. Such models, recently coined foundation models, have been transformational to the field of natural language processing. Variants have also been proposed for image data, but their applicability to remote sensing tasks is limited. To stimulate the development of foundation models for Earth monitoring, we propose a benchmark comprised of six classification and six segmentation tasks, which were carefully curated and adapted to be both relevant to the field and well-suited for model evaluation. We accompany this benchmark with a robust methodology for evaluating models and reporting aggregated results to enable a reliable assessment of progress. Finally, we report results for 20 baselines to gain information about the performance of existing models. We believe that this benchmark will be a driver of progress across a variety of Earth monitoring tasks., Comment: arXiv admin note: text overlap with arXiv:2112.00570

Passive resistive random access memory (ReRAM) crossbar arrays, a promising emerging technology used for analog matrix-vector multiplications, are far superior to their active (1T1R) counterparts in terms of the integration density. However, current transfers of neural network weights into the conductance state of the memory devices in the crossbar architecture are accompanied by significant losses in precision due to hardware variabilities such as sneak path currents, biasing scheme effects and conductance tuning imprecision. In this work, training approaches that adapt techniques such as dropout, the reparametrization trick and regularization to TiO2 crossbar variabilities are proposed in order to generate models that are better adapted to their hardware transfers. The viability of this approach is demonstrated by comparing the outputs and precision of the proposed hardware-aware network with those of a regular fully connected network over a few thousand weight transfers using the half moons dataset in a simulation based on experimental data. For the neural network trained using the proposed hardware-aware method, 79.5% of the test set's data points can be classified with an accuracy of 95% or higher, while only 18.5% of the test set's data points can be classified with this accuracy by the regularly trained neural network., Comment: 15 pages, 11 figures

In this work we report a study and a co-design methodology of an analog SNN crossbar output circuit designed in a 28nm FD-SOI technology node that comprises a tunable current attenuator and a leak-integrate and fire neurons that would enable the integration of emerging non-volatile memories (eNVMs) for synaptic arrays based on various technologies including phase change (PCRAM), oxide-based (OxRAM), spin transfer and spin orbit torque magnetic memories (STT, SOT-MRAM). Circuit SPICE simulation results and eNVM experimental data are used to showcase and estimate the neurons fan-in for each type of eNVM considering the technology constraints and design trade-offs that set its limits such as membrane capacitance and supply voltage, etc., Comment: 7 pages, 6 figures

Methodical aerial population surveys monitoring critically endangered species in Canadian North Atlantic waters are instrumental in influencing government policies both in economic and conservational efforts. The primary factor hindering the success of these missions is poor visibility caused by glare. This paper builds off our foundational paper [1] and pushes the envelope toward a data-driven glare modelling system. Said data-driven system makes use of meteorological and astronomical data to assist aircraft in navigating in order to mitigate acquisition errors and optimize the quality of acquired data. It is found that reliably extracting aircraft orientation is critical to our approach, to that end, we present a GPS alignment methodology which makes use of the fusion of two GPS signals. Using the complementary strengths and weaknesses of these two signals a synthetic interpolation of fused data is used to generate more reliable flight tracks, substantially improving glare modelling. This methodology could be applied to any other applications with similar signal restrictions., Comment: 12 pages, 11 figures, IEEE/ION Position Location and Navigation Symposium (PLANS 2023)

Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good discrimination in the non-asymptotic regime. In this paper, we provide the first systematic finite-sample study of proper scoring rules for time-series forecasting evaluation. Through a power analysis, we identify the "region of reliability" of a scoring rule, i.e., the set of practical conditions where it can be relied on to identify forecasting errors. We carry out our analysis on a comprehensive synthetic benchmark, specifically designed to test several key discrepancies between ground-truth and forecast distributions, and we gauge the generalizability of our findings to real-world tasks with an application to an electricity production problem. Our results reveal critical shortcomings in the evaluation of multivariate probabilistic forecasts as commonly performed in the literature., Comment: 47 pages, 37 figures, camera-ready version, Fortieth International Conference on Machine Learning (ICML 2023)

Critically endangered species in Canadian North Atlantic waters are systematically surveyed to estimate species populations which influence governing policies. Due to its impact on policy, population accuracy is important. This paper lays the foundation towards a data-driven glare modelling system, which will allow surveyors to preemptively minimize glare. Surveyors use a detection function to estimate megafauna populations which are not explicitly seen. A goal of the research is to maximize useful imagery collected, to that end we will use our glare model to predict glare and optimize for glare-free data collection. To build this model, we leverage a small labelled dataset to perform semi-supervised learning. The large dataset is labelled with a Cascading Random Forest Model using a na\"ive pseudo-labelling approach. A reflectance model is used, which pinpoints features of interest, to populate our datasets which allows for context-aware machine learning models. The pseudo-labelled dataset is used on two models: a Multilayer Perceptron and a Recurrent Neural Network. With this paper, we lay the foundation for data-driven mission planning; a glare modelling system which allows surveyors to preemptively minimize glare and reduces survey reliance on the detection function as an estimator of whale populations during periods of poor subsurface visibility., Comment: 15 pages, 4 figures, 5th ICPR Workshop on Computer Vison for Automated Analysis of Underwater Imagery (CVAUI 2022)