Search

The coldest Y spectral type brown dwarfs are similar in mass and temperature to cool and warm ($\sim$200 -- 400 K) giant exoplanets. We can therefore use their atmospheres as proxies for planetary atmospheres, testing our understanding of physics and chemistry for these complex, cool worlds. At these cold temperatures, their atmospheres are cold enough for water clouds to form, and chemical timescales increase, increasing the likelihood of disequilibrium chemistry compared to warmer classes of planets. JWST observations are revolutionizing the characterization of these worlds with high signal-to-noise, moderate resolution near- and mid-infrared spectra. The spectra have been used to measure the abundances of prominent species like water, methane, and ammonia; species that trace chemical reactions like carbon monoxide; and even isotopologues of carbon monoxide and ammonia. Here, we present atmospheric retrieval results using both published fixed-slit (GTO program 1230) and new averaged time series observations (GO program 2327) of the coldest known Y dwarf, WISE 0855-0714 (using NIRSpec G395M spectra), which has an effective temperature of $\sim$ 264 K. We present a detection of deuterium in an atmosphere outside of the solar system via a relative measurement of deuterated methane (CH$_{3}$D) and standard methane. From this, we infer the D/H ratio of a substellar object outside the solar system for the first time. We also present a well-constrained part-per-billion abundance of phosphine (PH$_{3}$). We discuss our interpretation of these results and the implications for brown dwarf and giant exoplanet formation and evolution., Comment: 17 pages, 7 figures, submitted to the Astrophysical Journal Letters

H5N1 highly pathogenic avian influenza (HPAI) has been recently circulating in previously unseen patterns. As the underlying causes are uncertain, we need a better understanding of the drivers of virus circulation, as they underpin the spread, and inform about the impact of HPAI. We combine infection and bird population data, using Random Forest models to quantify the relationship between spatially aggregated values of possible infection risk factors and viral phylogenetic data from HPAI outbreaks to define scales of interactions that influence virus circulation. As genetic proximity is correlated to transmission proximity, the genetic relationships of sequences and relevant variables indicate the scale and resolution of interactions driving transmission. We used H5N1 Sequences from Great Britain divided into two periods: Dec 2020-May 2022, and Jun-Oct 2022. Final model with selected spatial scales shows that for the first period game bird abundance and geographical distance were the strongest predictors of genetic distance, and for the second period, these were geographical distance, collection date and farm count. Variables related to infections have different spatial scales for analysed periods, while variables describing the environment have more consistent scales. The differences in predictors for the periods bears further investigation, however may be related to differences in the virus, areas of spread, or seasonal factors. This is the first phylogenetic study indicating possible role for game birds in the circulation of HPAI and supports experimental studies but must be viewed with caution as our study does not indicate direct causation. We show that the scale of spatial aggregation is variable dependent and reflects the scale of underlying processes. In the future our findings may be used to study dynamics of HPAI and design targeted surveillance and intervention strategies.

We report the first successful ALMA follow-up observations of a secure $z > 10$ JWST-selected galaxy, by robustly detecting ($6.6\sigma$) the [OIII]$_{88\mu m}\,$ line in JADES-GS-z14-0 (hereafter GS-z14). The ALMA detection yields a spectroscopic redshift of $z=14.1793\pm0.0007$, and increases the precision on the prior redshift measurement of $z=14.32_{-0.20}^{+0.08}$ from NIRSpec by $\gtrsim$180$\times$. Moreover, the redshift is consistent with that previously determined from a tentative detection ($3.6\sigma$) of CIII]$_{1907,1909}$ ($z=14.178\pm0.013$), solidifying the redshift determination via multiple line detections. We measure a line luminosity of $L_\mathrm{[OIII]88} = (2.1 \pm 0.5)\times10^8\,L_\odot$, placing GS-z14 at the lower end, but within the scatter of, the local $L_\mathrm{[OIII]88}$-star formation rate relation. No dust continuum from GS-z14 is detected, suggesting an upper limit on the dust-to-stellar mass ratio of $< 2 \times 10^{-3}$, consistent with dust production from supernovae with a yield $y_d < 0.3\,M_\odot$. Combining a previous JWST/MIRI photometric measurement of the [OIII]$\lambda\lambda$4959,5007$\mathrm{\mathring{A}}$ and H$\beta$ lines with Cloudy models, we find GS-z14 to be surprisingly metal-enriched ($Z\sim0.05 - 0.2\,Z_\odot$) a mere $300\,\mathrm{Myr}$ after the Big Bang. The detection of a bright oxygen line in GS-z14 thus reinforces the notion that galaxies in the early Universe undergo rapid evolution., Comment: 11 Pages, 6 Figures, 1 Table. Submitted to The Astrophysical Journal

We present a new method to determine the star formation rate (SFR) density of the Universe at $z \gtrsim 5$ that includes the contribution of dust-obscured star formation. For this purpose, we use a [CII] (158 $\mu$m) selected sample of galaxies serendipitously identified in the fields of known $z\gtrsim 4.5$ objects to characterize the fraction of obscured SFR. The advantage of a [CII] selection is that our sample is SFR-selected, in contrast to a UV-selection that would be biased towards unobscured star formation. We obtain a sample of 23 [CII] emitters near star-forming (SF) galaxies and QSOs -- three of which we identify for the first time -- using previous literature and archival ALMA data. 18 of these serendipitously identified galaxies have sufficiently deep rest-UV data and are used to characterize the obscured fraction of the star formation in galaxies with SFRs $\gtrsim 30\ \text{M}_{\odot} \ \text{yr}^{-1}$. We find that [CII] emitters identified around SF galaxies have $\approx$63\% of their SFR obscured, while [CII] emitters around QSOs have $\approx$93\% of their SFR obscured. By forward modeling existing wide-area UV luminosity function (LF) determinations, we derive the intrinsic UV LF using our characterization of the obscured SFR. Integrating the intrinsic LF to $M_{UV}$ = $-$20 we find that the obscured SFRD contributes to $>3\%$ and $>10\%$ of the total SFRD at $z \sim 5$ and $z \sim 6$ based on our sample of companions galaxies near SFGs and QSOs, respectively. Our results suggest that dust obscuration is not negligible at $z\gtrsim 5$, further underlining the importance of far-IR observations of the $z\gtrsim 5$ Universe., Comment: Accepted for publication in MNRAS. 24 pages, 14 figures, 8 tables (including appendices)

Characterising drivers of SARS-CoV-2 circulation is crucial for understanding COVID-19 because of the severity of control measures adopted during the pandemic. Whole genome sequence data augmented with demographic metadata provides the best opportunity to do this. We use Random Forest Decision Tree models to analyse a combination of over 4000 SARS-CoV2 sequences from a densely sampled, mixed urban and rural population (Tayside) in Scotland in the period from August 2020 to July 2021, with fine scale geographical and socio-demographic metadata. Comparing periods in versus out of "lockdown" restrictions, we show using genetic distance relationships that individuals from more deprived areas are more likely to get infected during lockdown but less likely to spread the infection further. As disadvantaged communities were the most affected by both COVID-19 and its restrictions, our finding has important implications for informing future approaches to control future pandemics driven by similar respiratory infections., Comment: 186 pages, approx. 48 figures (including Methods ans Supplementary Information)

We use Gr\"obner bases to compute the Sinkhorn limit of a positive $3 \times 3$ matrix $A$, showing that the entries are algebraic numbers with degree at most 6. The polynomial equation satisfied by each entry is large, but we show that it has a natural representation in terms of linear combinations of products of minors of $A$. We then use this representation to interpolate a polynomial equation satisfied by an entry of the Sinkhorn limit of a positive $4 \times 4$ matrix. Finally, we use the PSLQ algorithm and 1.5 years of CPU time to formulate a conjecture, up to certain signs that we have not been able to identify, for a polynomial equation satisfied by an entry of the Sinkhorn limit of a positive $m \times n$ matrix. In particular, we conjecture that the entries are algebraic numbers with degree at most $\binom{m + n - 2}{m - 1}$. This degree has a combinatorial interpretation as the number of minors of an $(m - 1) \times (n - 1)$ matrix, and the coefficients reflect new combinatorial structure on sets of minor specifications., Comment: 25 pages; Mathematica package and documentation available as ancillary files

In reinforcement learning (RL), the consideration of multivariate reward signals has led to fundamental advancements in multi-objective decision-making, transfer learning, and representation learning. This work introduces the first oracle-free and computationally-tractable algorithms for provably convergent multivariate distributional dynamic programming and temporal difference learning. Our convergence rates match the familiar rates in the scalar reward setting, and additionally provide new insights into the fidelity of approximate return distribution representations as a function of the reward dimension. Surprisingly, when the reward dimension is larger than $1$, we show that standard analysis of categorical TD learning fails, which we resolve with a novel projection onto the space of mass-$1$ signed measures. Finally, with the aid of our technical results and simulations, we identify tradeoffs between distribution representations that influence the performance of multivariate distributional RL in practice.

We present new multi-frequency ALMA continuum observations of the massive [$\log_{10}(M_\star/M_\odot) = 10.3_{-0.2}^{+0.1}$], UV-luminous [$M_\mathrm{UV} = -21.7 \pm 0.2$] $z=7.31$ galaxy REBELS-25 in Bands 3, 4, 5, and 9. Combining the new observations with previously-taken data in Bands 6 and 8, we cover the dust continuum emission of the galaxy in six distinct bands -- spanning rest-frame $50-350\,\mu$m -- enabling simultaneous constraints on its dust mass ($M_\mathrm{dust}$), temperature ($T_\mathrm{dust}$) and emissivity index ($\beta_\mathrm{IR}$) via modified blackbody fitting. Given a fiducial model of optically thin emission, we infer a cold dust temperature of $T_\mathrm{dust} = 32_{-6}^{+9}\,$K and a high dust mass of $\log_{10}(M_\mathrm{dust}/M_\odot) = 8.2_{-0.4}^{+0.6}$, and moderately optically thick dust does not significantly alter these estimates. If we assume dust production is solely through supernovae (SNe), the inferred dust yield would be high, $y = 0.7_{-0.4}^{+2.3}\,M_\odot$ per SN. Consequently, we argue grain growth in the interstellar medium of REBELS-25 also contributes to its dust build-up. This is supported by the steep dust emissivity index $\beta_\mathrm{IR} = 2.5 \pm 0.4$ we measure for REBELS-25, as well as by its high stellar mass, dense interstellar medium, and metal-rich nature. Our results suggest that constraining the dust emissivity indices of high-redshift galaxies is important not only to mitigate systematic uncertainties in their dust masses and obscured star formation rates, but also to assess if dust properties evolve across cosmic time. We present an efficient observing setup to do so with ALMA, combining observations of the peak and Rayleigh-Jeans tail of the dust emission., Comment: 14 pages main text, 6 figures + appendices; re-submitted to MNRAS after a positive referee report

Christol and, independently, Denef and Lipshitz showed that an algebraic sequence of $p$-adic integers (or integers) is $p$-automatic when reduced modulo $p^\alpha$. Previously, the best known bound on the minimal automaton size for such a sequence was doubly exponential in $\alpha$. We improve this bound to the order of $p^{\alpha^3 h d}$, where $h$ and $d$ are the height and degree of the minimal annihilating polynomial modulo $p$. We achieve this bound by showing that all states in the automaton are naturally represented in a new numeration system. This significantly restricts the set of possible states. Since our approach embeds algebraic sequences as diagonals of rational functions, we also obtain bounds more generally for diagonals of multivariate rational functions., Comment: 43 pages, 1 figure, 2 tables; includes discussions of related results due to Beukers and Straub

Effective policy and intervention strategies to combat human trafficking for child sexual exploitation material (CSEM) production require accurate prevalence estimates. Traditional Network Scale Up Method (NSUM) models often necessitate standalone surveys for each geographic region, escalating costs and complexity. This study introduces a partially pooled NSUM model, using a hierarchical Bayesian framework that efficiently aggregates and utilizes data across multiple regions without increasing sample sizes. We developed this model for a novel national survey dataset from the Philippines and we demonstrate its ability to produce detailed municipal-level prevalence estimates of trafficking for CSEM production. Our results not only underscore the model's precision in estimating hidden populations but also highlight its potential for broader application in other areas of social science and public health research, offering significant implications for resource allocation and intervention planning.

Learning a good representation is a crucial challenge for Reinforcement Learning (RL) agents. Self-predictive learning provides means to jointly learn a latent representation and dynamics model by bootstrapping from future latent representations (BYOL). Recent work has developed theoretical insights into these algorithms by studying a continuous-time ODE model for self-predictive representation learning under the simplifying assumption that the algorithm depends on a fixed policy (BYOL-$\Pi$); this assumption is at odds with practical instantiations of such algorithms, which explicitly condition their predictions on future actions. In this work, we take a step towards bridging the gap between theory and practice by analyzing an action-conditional self-predictive objective (BYOL-AC) using the ODE framework, characterizing its convergence properties and highlighting important distinctions between the limiting solutions of the BYOL-$\Pi$ and BYOL-AC dynamics. We show how the two representations are related by a variance equation. This connection leads to a novel variance-like action-conditional objective (BYOL-VAR) and its corresponding ODE. We unify the study of all three objectives through two complementary lenses; a model-based perspective, where each objective is shown to be equivalent to a low-rank approximation of certain dynamics, and a model-free perspective, which establishes relationships between the objectives and their respective value, Q-value, and advantage function. Our empirical investigations, encompassing both linear function approximation and Deep RL environments, demonstrates that BYOL-AC is better overall in a variety of different settings.

Honour based abuse covers a wide range of family abuse including female genital mutilation and forced marriage. Safeguarding professionals need to identify where abuses are happening in their local community to best support those at risk of these crimes and take preventative action. However, there is little local data about these kinds of crime. To tackle this problem, we ran comparative judgement surveys to map abuses at local level. In previous comparative judgement studies, participants reported fatigue associated with comparisons between areas with similar levels of abuse. Allowing for ties reduces fatigue, but increase the computational complexity when fitting the model. We designed an efficient Markov Chain Monte Carlo algorithm to fit the model, allowing for a wide range of prior distributions on the model parameters. Working with South Yorkshire Police and Oxford Against Cutting, we mapped the risk of honour based abuse at community level in two counties in the UK., Comment: 17 pages, 6 figures, submitted

We present high resolution ($\sim0.14$" = 710 pc) ALMA [CII] 158$\mu$m and dust continuum follow-up observations of REBELS-25, a [CII]-luminous ($L_{\mathrm{[CII]}}=(1.7\pm0.2)\times 10^9 \mathrm{L_{\odot}}$) galaxy at redshift $z=7.3065\pm0.0001$. These high resolution, high signal-to-noise observations allow us to study the sub-kpc morphology and kinematics of this massive ($M_* = 8^{+4}_{-2} \times 10^9 \mathrm{M_{\odot}}$) star-forming (SFR$_{\mathrm{UV+IR}} = 199^{+101}_{-63} \mathrm{M_{\odot}} \mathrm{yr}^{-1}$) galaxy in the Epoch of Reionisation. By modelling the kinematics with $^{\mathrm{3D}}$BAROLO, we find it has a low velocity dispersion ($\bar{\sigma} = 33 \pm 9$ km s$^{-1}$) and a high ratio of ordered-to-random motion ($V_{\mathrm{rot, ~max}}/\bar{\sigma} = 11 ^{+8}_{-4}$), indicating that REBELS-25 is a dynamically cold disc. Additionally, we find that the [CII] distribution is well fit by a near-exponential disc model, with a S\'ersic index, $n$, of $1.3 \pm 0.2$, and we see tentative evidence of more complex non-axisymmetric structures suggestive of a bar in the [CII] and dust continuum emission. By comparing to other high spatial resolution cold gas kinematic studies, we find that dynamically cold discs seem to be more common in the high redshift Universe than expected based on prevailing galaxy formation theories, which typically predict more turbulent and dispersion-dominated galaxies in the early Universe as an outcome of merger activity, gas accretion and more intense feedback. This higher degree of rotational support seems instead to be consistent with recent cosmological simulations that have highlighted the contrast between cold and warm ionised gas tracers, particularly for massive galaxies. We therefore show that dynamically settled disc galaxies can form as early as 700 Myr after the Big Bang., Comment: Submitted to MNRAS

Objective: To report imaging protocol and scheduling variance in routine care of glioblastoma patients in order to demonstrate challenges of integrating deep-learning models in glioblastoma care pathways. Additionally, to understand the most common imaging studies and image contrasts to inform the development of potentially robust deep-learning models. Methods: MR imaging data were analysed from a random sample of five patients from the prospective cohort across five participating sites of the ZGBM consortium. Reported clinical and treatment data alongside DICOM header information were analysed to understand treatment pathway imaging schedules. Results: All sites perform all structural imaging at every stage in the pathway except for the presurgical study, where in some sites only contrast-enhanced T1-weighted imaging is performed. Diffusion MRI is the most common non-structural imaging type, performed at every site. Conclusion: The imaging protocol and scheduling varies across the UK, making it challenging to develop machine-learning models that could perform robustly at other centres. Structural imaging is performed most consistently across all centres. Advances in knowledge: Successful translation of deep-learning models will likely be based on structural post-treatment imaging unless there is significant effort made to standardise non-structural or peri-operative imaging protocols and schedules.

Beyond our solar system, aurorae have been inferred from radio observations of isolated brown dwarfs (e.g. Hallinan et al. 2006; Kao et al. 2023). Within our solar system, giant planets have auroral emission with signatures across the electromagnetic spectrum including infrared emission of H3+ and methane. Isolated brown dwarfs with auroral signatures in the radio have been searched for corresponding infrared features but have only had null detections (e.g. Gibbs et al. 2022). CWISEP J193518.59-154620.3. (W1935 for short) is an isolated brown dwarf with a temperature of ~482 K. Here we report JWST observations of strong methane emission from W1935 at 3.326 microns. Atmospheric modeling leads us to conclude that a temperature inversion of ~300 K centered at 1-10 millibar replicates the feature. This represents an atmospheric temperature inversion for a Jupiter-like atmosphere without irradiation from a host star. A plausible explanation for the strong inversion is heating by auroral processes, although other internal and/or external dynamical processes cannot be ruled out. The best fit model rules out the contribution of H3+ emission which is prominent in solar system gas giants however this is consistent with rapid destruction of H3+ at the higher pressure where the W1935 emission originates (e.g. Helling et al. 2019)., Comment: Accepted in Nature 9 February 2024

Diagnostic tests which can detect pre-clinical or sub-clinical infection, are one of the most powerful tools in our armoury of weapons to control infectious diseases. Considerable effort has been therefore paid to improving diagnostic testing for human, plant and animal diseases, including strategies for targeting the use of diagnostic tests towards individuals who are more likely to be infected. Here, we follow other recent proposals to further refine this concept, by using machine learning to assess the situational risk under which a diagnostic test is applied to augment its interpretation . We develop this to predict the occurrence of breakdowns of cattle herds due to bovine tuberculosis, exploiting the availability of exceptionally detailed testing records. We show that, without compromising test specificity, test sensitivity can be improved so that the proportion of infected herds detected by the skin test, improves by over 16 percentage points. While many risk factors are associated with increased risk of becoming infected, of note are several factors which suggest that, in some herds there is a higher risk of infection going undetected, including effects that are correlated to the veterinary practice conducting the test, and number of livestock moved off the herd.

Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task.

Stellar feedback in dwarf galaxies remains, to date, poorly explored, yet is crucial to understanding galaxy evolution in the early Universe. In particular, pre-supernova feedback has recently been found to play a significant role in regulating and disrupting star formation in larger spiral galaxies, but it remains uncertain if it also plays this role in dwarfs. We study the ionised gas properties and stellar content of individual star-forming regions across three nearby, low-metallicity, dwarf starburst galaxies (J0921, KKH046, and Leo P) to investigate how massive stars influence their surroundings and how this influence changes as a function of environment. To achieve this, we extracted integrated spectra of 30 HII regions from archival VLT/MUSE integral field spectroscopic observations of these three dwarf starburst galaxies. We fitted the HII regions' main emission lines with Gaussian profiles to derive their oxygen abundances, electron densities, and luminosities, and we used the Stochastically Lighting Up Galaxies (SLUG) code to derive the stellar mass, age, and bolometric luminosity of the stellar populations driving the HII regions. We then quantified two pre-supernova stellar feedback mechanisms, namely the direct radiation pressure and photoionisation feedback, and explored how feedback strength varies with HII region properties. Our findings suggest that stellar feedback has less of an impact on evolved regions, with both the pressure of the ionised gas and the direct radiation pressure decreasing as a function of HII region size. We also find that these stellar feedback mechanisms are dependent on the metallicity of the HII regions. These findings extend results from stellar feedback studies of more massive star-forming galaxies to the low-mass, low-metallicity regime., Comment: The paper has been published in A&A

This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible., Comment: Accepted to ICML 2024. First two authors contributed equally

We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023). Our analysis provides new theoretical results on categorical approaches to distributional RL, and also introduces a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. We also provide an experimental study comparing several model-based distributional RL algorithms, with several takeaways for practitioners., Comment: NeurIPS 2024

We introduce off-policy distributional Q($\lambda$), a new addition to the family of off-policy distributional evaluation algorithms. Off-policy distributional Q($\lambda$) does not apply importance sampling for off-policy learning, which introduces intriguing interactions with signed measures. Such unique properties distributional Q($\lambda$) from other existing alternatives such as distributional Retrace. We characterize the algorithmic properties of distributional Q($\lambda$) and validate theoretical insights with tabular experiments. We show how distributional Q($\lambda$)-C51, a combination of Q($\lambda$) with the C51 agent, exhibits promising results on deep RL benchmarks.

Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners., Comment: Accepted at ICML 2023 main conference

The increasing popularity of AI, particularly Large Language Models (LLMs), has significantly impacted various domains, including Software Engineering. This study explores the integration of AI tools in software engineering practices within a large organization. We focus on ANZ Bank, which employs over 5000 engineers covering all aspects of the software development life cycle. This paper details an experiment conducted using GitHub Copilot, a notable AI tool, within a controlled environment to evaluate its effectiveness in real-world engineering tasks. Additionally, this paper shares initial findings on the productivity improvements observed after GitHub Copilot was adopted on a large scale, with about 1000 engineers using it. ANZ Bank's six-week experiment with GitHub Copilot included two weeks of preparation and four weeks of active testing. The study evaluated participant sentiment and the tool's impact on productivity, code quality, and security. Initially, participants used GitHub Copilot for proposed use-cases, with their feedback gathered through regular surveys. In the second phase, they were divided into Control and Copilot groups, each tackling the same Python challenges, and their experiences were again surveyed. Results showed a notable boost in productivity and code quality with GitHub Copilot, though its impact on code security remained inconclusive. Participant responses were overall positive, confirming GitHub Copilot's effectiveness in large-scale software engineering environments. Early data from 1000 engineers also indicated a significant increase in productivity and job satisfaction., Comment: 16 pages, 4 figures. in proceeding for 10th International Conference on Software Engineering (SEC 2024)

We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.

Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences.