Search

As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM generations spanning 8 dimensions and 30 issues. Our evaluation task is a real-world example of a growing number of challenging problems where AI can complement and lift human performance. We introduce a novel protocol for scalable oversight that relies on AI Assistance and raters with relevant education. We evaluate several recent LLMs on a set of diverse climate questions. Our results point to a significant gap between surface and epistemological qualities of LLMs in the realm of climate communication.

Neural retrieval models have superseded classic bag-of-words methods such as BM25 as the retrieval framework of choice. However, neural systems lack the interpretability of bag-of-words models; it is not trivial to connect a query change to a change in the latent space that ultimately determines the retrieval results. To shed light on this embedding space, we learn a "query decoder" that, given a latent representation of a neural search engine, generates the corresponding query. We show that it is possible to decode a meaningful query from its latent representation and, when moving in the right direction in latent space, to decode a query that retrieves the relevant paragraph. In particular, the query decoder can be useful to understand "what should have been asked" to retrieve a particular paragraph from the collection. We employ the query decoder to generate a large synthetic dataset of query reformulations for MSMarco, leading to improved retrieval performance. On this data, we train a pseudo-relevance feedback (PRF) T5 model for the application of query suggestion that outperforms both query reformulation and PRF information retrieval baselines.

Learning to search is the task of building artificial agents that learn to autonomously use a search box to find information. So far, it has been shown that current language models can learn symbolic query reformulation policies, in combination with traditional term-based retrieval, but fall short of outperforming neural retrievers. We extend the previous learning to search setup to a hybrid environment, which accepts discrete query refinement operations, after a first-pass retrieval step via a dual encoder. Experiments on the BEIR task show that search agents, trained via behavioral cloning, outperform the underlying search system based on a combined dual encoder retriever and cross encoder reranker. Furthermore, we find that simple heuristic Hybrid Retrieval Environments (HRE) can improve baseline performance by several nDCG points. The search agent based on HRE (HARE) matches state-of-the-art performance, balanced in both zero-shot and in-domain evaluations, via interpretable actions, and at twice the speed.

This paper presents first successful steps in designing search agents that learn meta-strategies for iterative query refinement in information-seeking tasks. Our approach uses machine reading to guide the selection of refinement terms from aggregated search results. Agents are then empowered with simple but effective search operators to exert fine-grained and transparent control over queries and search results. We develop a novel way of generating synthetic search sessions, which leverages the power of transformer-based language models through (self-)supervised learning. We also present a reinforcement learning agent with dynamically constrained actions that learns interactive search strategies from scratch. Our search agents obtain retrieval and answer quality performance comparable to recent neural methods, using only a traditional term-based BM25 ranking function and interpretable discrete reranking and filtering actions., Comment: Published in Transactions on Machine Learning Research (06/2022)

We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment. The environment encapsulates a competitive machine reader based on BERT, providing candidate answers to questions, and possibly some context. To validate the realism of our formulation, we ask humans to play the role of a meta-answerer. With just a small snippet of text around an answer, humans can outperform the machine reader, improving recall. Similarly, a simple machine meta-answerer outperforms the environment, improving both precision and recall on the Natural Questions dataset. The system relies on joint training of answer scoring and the selection of conditioning information.

"Whether life exists beyond Earth remains a fundamental question driving our exploration of the Solar System. At Saturn’s moon Enceladus, plumes of oceanic material vented into space allow the investigation of the astrobiological potential of an ocean world, hinted at by Cassini, without the necessity of drilling through kilometers of ice crust. The Enceladus Orbilander is a flagship ($2.56B in fiscal year 2025 dollars) mission concept created for the 2023–2033 Planetary Science Decadal Survey. Orbilander takes full advantage of the opportunity provided by Enceladus’ plumes to search for signs of life. A single spacecraft both orbits and lands, capturing samples from four distinct reservoirs offered by the plumes. These samples, both particulate and vapor, are then analyzed by the Life Detection Suite (LDS), a set of five instruments conducting complementary and orthogonal biosignature-seeking measurements. To provide the context that specifically enhances interpretation of LDS measurements, geochemical and geophysical investigations are conducted both in orbit and on the surface. These reveal the physio-chemical state of the ocean and core as well as the processes involved in ejection of plume material and how these affect the ocean material analyzed by the LDS. The Orbilander can be delivered to the Saturn system via several launch vehicle and trajectory options, including a direct trajectory (7-year cruise), a ∆V-EGA trajectory (9-year cruise) and several options using an inner cruise with Venus and Earth flybys (10-year cruise). Upon Saturn Orbit Insertion, a 4-year moon tour pumps down the Orbilander’s orbit to intercept Enceladus. The most optimal arrival times balance the Jupiter flyby opportunities of the late 2030s and solar illumination at the Enceladus high southern latitudes where plume material is most abundant. This mission concept therefore targets project start in 2030. Upon Enceladus Orbit Insertion, the Orbilander begins a 1.5-year-long campaign of landing site reconnaissance, remote sensing science, and collecting sufficient plume sample to run all but one of the LDS measurements. After successful landing, the Orbilander spends 2 years on the surface conducting multiple LDS measurements with all five instruments on actively and passively collected plume material, as well as seismic investigations. The schedule laid out here is well-defined, but the mission also has operational and resource flexibility should additional reconnaissance be needed. As part of the design study, mission and development risks were identified and mitigation strategies proposed. Technologies key to achieving the life detection science objectives include instrumentation matured under programs like COLDTech and ICEE-2, such as aspects of the sampling system and microfluidic devices, as well as well-known techniques like high-resolution and separation-capable mass spectrometers. Autonomous onboard navigation is planned to maintain a halo orbit around Enceladus to enable passive sampling from orbit as well as reconnaissance measurements for use in site selection and landing. Terrain relative navigation is included to ensure safe landing, given that targeted areas may contain landing hazards. Continued development of radioisotope thermoelectric generator (RTG) technology and long-life batteries is essential for this long duration mission. The Enceladus Orbilander represents an optimal point in the trade space of science value versus cost, taking advantage of the extensive knowledge of Enceladus provided by Cassini, how well Enceladus lends itself to a search for life in material from its ocean, and the flexibility afforded by the innovative design developed by the APL team. By taking full advantage of Enceladus’ plumes both in orbit and on the surface, Orbilander represents a robust search for life with complementary and orthogonal biosignatures as well as contextual geophysical and geochemical measurements, determining not only whether Enceladus is inhabited (at levels up to 500,000× scarcer than in Earth’s oceans) but also why. "