Search

2D Matryoshka Training is an advanced embedding representation training approach designed to train an encoder model simultaneously across various layer-dimension setups. This method has demonstrated higher effectiveness in Semantic Text Similarity (STS) tasks over traditional training approaches when using sub-layers for embeddings. Despite its success, discrepancies exist between two published implementations, leading to varied comparative results with baseline models. In this reproducibility study, we implement and evaluate both versions of 2D Matryoshka Training on STS tasks and extend our analysis to retrieval tasks. Our findings indicate that while both versions achieve higher effectiveness than traditional Matryoshka training on sub-dimensions, and traditional full-sized model training approaches, they do not outperform models trained separately on specific sub-layer and sub-dimension setups. Moreover, these results generalize well to retrieval tasks, both in supervised (MSMARCO) and zero-shot (BEIR) settings. Further explorations of different loss computations reveals more suitable implementations for retrieval tasks, such as incorporating full-dimension loss and training on a broader range of target dimensions. Conversely, some intuitive approaches, such as fixing document encoders to full model outputs, do not yield improvements. Our reproduction code is available at https://github.com/ielab/2DMSE-Reproduce.

Effective approaches that can scale embedding model depth (i.e. layers) and embedding size allow for the creation of models that are highly scalable across different computational resources and task requirements. While the recently proposed 2D Matryoshka training approach can efficiently produce a single embedding model such that its sub-layers and sub-dimensions can measure text similarity, its effectiveness is significantly worse than if smaller models were trained separately. To address this issue, we propose Starbucks, a new training strategy for Matryoshka-like embedding models, which encompasses both the fine-tuning and pre-training phases. For the fine-tuning phase, we discover that, rather than sampling a random sub-layer and sub-dimensions for each training steps, providing a fixed list of layer-dimension pairs, from small size to large sizes, and computing the loss across all pairs significantly improves the effectiveness of 2D Matryoshka embedding models, bringing them on par with their separately trained counterparts. To further enhance performance, we introduce a new pre-training strategy, which applies masked autoencoder language modelling to sub-layers and sub-dimensions during pre-training, resulting in a stronger backbone for subsequent fine-tuning of the embedding model. Experimental results on both semantic text similarity and retrieval benchmarks demonstrate that the proposed pre-training and fine-tuning strategies significantly improved the effectiveness over 2D Matryoshka models, enabling Starbucks models to perform more efficiently and effectively than separately trained models.

The emergence of Vec2Text -- a method for text embedding inversion -- has raised serious privacy concerns for dense retrieval systems which use text embeddings. This threat comes from the ability for an attacker with access to embeddings to reconstruct the original text. In this paper, we take a new look at Vec2Text and investigate how much of a threat it poses to the different attacks of corpus poisoning, whereby an attacker injects adversarial passages into a retrieval corpus with the intention of misleading dense retrievers. Theoretically, Vec2Text is far more dangerous than previous attack methods because it does not need access to the embedding model's weights and it can efficiently generate many adversarial passages. We show that under certain conditions, corpus poisoning with Vec2Text can pose a serious threat to dense retriever system integrity and user experience by injecting adversarial passaged into top ranked positions. Code and data are made available at https://github.com/ielab/vec2text-corpus-poisoning, Comment: arXiv admin note: substantial text overlap with arXiv:2402.12784

Currently, neural-network processing in machine learning applications relies on layer synchronization, whereby neurons in a layer aggregate incoming currents from all neurons in the preceding layer, before evaluating their activation function. This is practiced even in artificial Spiking Neural Networks (SNNs), which are touted as consistent with neurobiology, in spite of processing in the brain being, in fact asynchronous. A truly asynchronous system however would allow all neurons to evaluate concurrently their threshold and emit spikes upon receiving any presynaptic current. Omitting layer synchronization is potentially beneficial, for latency and energy efficiency, but asynchronous execution of models previously trained with layer synchronization may entail a mismatch in network dynamics and performance. We present a study that documents and quantifies this problem in three datasets on our simulation environment that implements network asynchrony, and we show that models trained with layer synchronization either perform sub-optimally in absence of the synchronization, or they will fail to benefit from any energy and latency reduction, when such a mechanism is in place. We then "make ends meet" and address the problem with unlayered backprop, a novel backpropagation-based training method, for learning models suitable for asynchronous processing. We train with it models that use different neuron execution scheduling strategies, and we show that although their neurons are more reactive, these models consistently exhibit lower overall spike density (up to 50%), reach a correct decision faster (up to 2x) without integrating all spikes, and achieve superior accuracy (up to 10% higher). Our findings suggest that asynchronous event-based (neuromorphic) AI computing is indeed more efficient, but we need to seriously rethink how we train our SNN models, to benefit from it.

The Shared Task on Large-Scale Radiology Report Generation (RRG24) aims to expedite the development of assistive systems for interpreting and reporting on chest X-ray (CXR) images. This task challenges participants to develop models that generate the findings and impression sections of radiology reports from CXRs from a patient's study, using five different datasets. This paper outlines the e-Health CSIRO team's approach, which achieved multiple first-place finishes in RRG24. The core novelty of our approach lies in the addition of entropy regularisation to self-critical sequence training, to maintain a higher entropy in the token distribution. This prevents overfitting to common phrases and ensures a broader exploration of the vocabulary during training, essential for handling the diversity of the radiology reports in the RRG24 datasets. Our model is available on Hugging Face https://huggingface.co/aehrc/cxrmate-rrg24.

We propose a new statistical reduced complexity climate model. The centerpiece of the model consists of a set of physical equations for the global climate system which we show how to cast in non-linear state space form. The parameters in the model are estimated using the method of maximum likelihood with the likelihood function being evaluated by the extended Kalman filter. Our statistical framework is based on well-established methodology and is computationally feasible. In an empirical analysis, we estimate the parameters for a data set comprising the period 1959-2022. A likelihood ratio test sheds light on the most appropriate equation for converting the level of atmospheric concentration of carbon dioxide into radiative forcing. Using the estimated model, and different future paths of greenhouse gas emissions, we project global mean surface temperature until the year 2100. Our results illustrate the potential of combining statistical modelling with physical insights to arrive at rigorous statistical analyses of the climate system.

Clinical documentation is an important aspect of clinicians' daily work and often demands a significant amount of time. The BioNLP 2024 Shared Task on Streamlining Discharge Documentation (Discharge Me!) aims to alleviate this documentation burden by automatically generating discharge summary sections, including brief hospital course and discharge instruction, which are often time-consuming to synthesize and write manually. We approach the generation task by fine-tuning multiple open-sourced language models (LMs), including both decoder-only and encoder-decoder LMs, with various configurations on input context. We also examine different setups for decoding algorithms, model ensembling or merging, and model specialization. Our results show that conditioning on the content of discharge summary prior to the target sections is effective for the generation task. Furthermore, we find that smaller encoder-decoder LMs can work as well or even slightly better than larger decoder based LMs fine-tuned through LoRA. The model checkpoints from our team (aehrc) are openly available., Comment: BioNLP @ ACL 2024

The goal of screening prioritisation in systematic reviews is to identify relevant documents with high recall and rank them in early positions for review. This saves reviewing effort if paired with a stopping criterion, and speeds up review completion if performed alongside downstream tasks. Recent studies have shown that neural models have good potential on this task, but their time-consuming fine-tuning and inference discourage their widespread use for screening prioritisation. In this paper, we propose an alternative approach that still relies on neural models, but leverages dense representations and relevance feedback to enhance screening prioritisation, without the need for costly model fine-tuning and inference. This method exploits continuous relevance feedback from reviewers during document screening to efficiently update the dense query representation, which is then applied to rank the remaining documents to be screened. We evaluate this approach across the CLEF TAR datasets for this task. Results suggest that the investigated dense query-driven approach is more efficient than directly using neural models and shows promising effectiveness compared to previous methods developed on the considered datasets. Our code is available at https://github.com/ielab/dense-screening-feedback., Comment: Accepted at SIGIR 2024;typos corrected

The Simons Observatory (SO) is a Cosmic Microwave Background experiment located in the Atacama Desert in Chile. SO consists of three small aperture telescopes (SATs) and one large aperture telescope (LAT) with a total of 60,000 detectors in six frequency bands. As an observatory, SO encompasses hundreds of hardware components simultaneously running at different readout rates, all separate from its 60,000 detectors on-sky and their metadata. We provide an overview of commissioning SO's data acquisition software system for SAT-MF1, the first SAT deployed to the Atacama site. Additionally, we share insights from deploying data access software for all four telescopes, detailing how performance limitations affected data loading and quality investigations, which led to site-compatible software improvements., Comment: 15 pages, 9 figures, 3 tables. Presented at SPIE Astronomical Telescopes + Instrumentation 2024 Second submission: fix typos

The Simons Observatory (SO) is a cosmic microwave background (CMB) observatory consisting of three small aperture telescopes and one large aperture telescope. SO is located in the Atacama Desert in Chile at an elevation of 5180m. Distributed among the four telescopes are over 60,000 transition-edge sensor (TES) bolometers across six spectral bands centered between 27 and 280 GHz. A large collection of ancillary hardware devices which produce lower rate `housekeeping' data are used to support the detector data collection. We developed a distributed control system, which we call the observatory control system (ocs), to coordinate data collection among all systems within the observatory. ocs is a core component of the deployed site software, interfacing with all on-site hardware. Alongside ocs we utilize a combination of internally and externally developed open source projects to enable remote monitoring, data management, observation coordination, and data processing. Deployment of a majority of the software is done using Docker containers. The deployment of software packages is partially done via automated Ansible scripts, utilizing a GitOps based approach for updating infrastructure on site. We describe an overview of the software and computing systems deployed within SO, including how those systems are deployed and interact with each other. We also discuss the timing distribution system and its configuration as well as lessons learned during the deployment process and where we plan to make future improvements., Comment: 19 pages, 4 figures

The Simons Observatory (SO) is a group of modern telescopes dedicated to observing the polarized cosmic microwave background (CMB), transients, and more. The Observatory consists of four telescopes and instruments, with over 60,000 superconducting detectors in total, located at ~5,200 m altitude in the Atacama Desert of Chile. During observations, it is important to ensure the detectors, telescope platforms, calibration and receiver hardware, and site hardware are within operational bounds. To facilitate rapid response when problems arise with any part of the system, it is essential that alerts are generated and distributed to appropriate personnel if components exceed these bounds. Similarly, alerts are generated if the quality of the data has become degraded. In this paper, we describe the SO alarm system we developed within the larger Observatory Control System (OCS) framework, including the data sources, alert architecture, and implementation. We also present results from deploying the alarm system during the commissioning of the SO telescopes and receivers., Comment: 14 pages, 6 figures, 2 tables. To be presented at SPIE Astronomical Telescopes + Instrumentation 2024

This study investigates the integration of diverse patient data sources into multimodal language models for automated chest X-ray (CXR) report generation. Traditionally, CXR report generation relies solely on CXR images and limited radiology data, overlooking valuable information from patient health records, particularly from emergency departments. Utilising the MIMIC-CXR and MIMIC-IV-ED datasets, we incorporate detailed patient information such as aperiodic vital signs, medications, and clinical history to enhance diagnostic accuracy. We introduce a novel approach to transform these heterogeneous data sources into embeddings that prompt a multimodal language model, significantly enhancing the diagnostic accuracy of generated radiology reports. Our comprehensive evaluation demonstrates the benefits of using a broader set of patient data, underscoring the potential for enhanced diagnostic capabilities and better patient outcomes through the integration of multimodal data in CXR report generation.

The Simons Observatory (SO) is a next-generation ground-based telescope located in the Atacama Desert in Chile, designed to map the cosmic microwave background (CMB) with unprecedented precision. The observatory consists of three small aperture telescopes (SATs) and one large aperture telescope (LAT), each optimized for distinct but complementary scientific goals. To achieve these goals, optimized scan strategies have been defined for both the SATs and LAT. This paper describes a software system deployed in SO that effectively translates high-level scan strategies into realistic observing scripts executable by the telescope, taking into account realistic observational constraints. The data volume of SO also necessitates a scalable software infrastructure to support its daily data processing needs. This paper also outlines an automated workflow system for managing data packaging and daily data reduction at the site., Comment: 16 pages, 7 figures, 6 tables, to be presented at SPIE Astronomical Telescopes + Instrumentation 2024

In this paper we demonstrate a technique for developing high performance applications with strong correctness guarantees. We use a theorem prover to derive a high-level specification of the application that includes correctness invariants of our choice. After that, within the same theorem prover, we implement an extraction of the specified application into a high-performance language of our choice. Concretely, we are using Agda to specify a framework for automatic differentiation (reverse mode) that is focused on index-safe tensors. This framework comes with an optimiser for tensor expressions and the ability to translate these expressions into SaC and C. We specify a canonical convolutional neural network within the proposed framework, compute the derivatives needed for the training phase and then demonstrate that the generated code matches the performance of hand-written code when running on a multi-core machine.

A robotaxi dragged a pedestrian 20 feet down a San Francisco street on the evening of October 2, 2023, coming to rest with its rear wheel on that woman's legs. The mishap was complex, involving a first impact by a different, human-driven vehicle. The following weeks saw Cruise stand down its road operations amid allegations of withholding crucial mishap information from regulators. The pedestrian has survived her severe injuries, but the robotaxi industry is still wrestling with the aftermath. Key observations include that the robotaxi had multiple possible ways available to avoid initial impact with the pedestrian. Limitations to the computer driver's programming prevented it from recognizing a pedestrian was about to be hit in an adjacent lane, caused the robotaxi to lose tracking of and then in essence forget a pedestrian who was hit by an adjacent vehicle, and forget that the robotaxi had just run over a presumed pedestrian when beginning a subsequent repositioning maneuver. The computer driver was unable to detect the pedestrian being dragged even though her legs were partially in view of a robotaxi camera. Moreover, more conservative operational approaches could have avoided the dragging portion of the mishap entirely, such as waiting for remote confirmation before moving after a crash with a pedestrian, or operating the still-developing robotaxi technology with an in-vehicle safety driver rather than prioritizing driver-out deployment., Comment: 1 figure; preprint for IEEE Reliability Magazine

In many states, certificate-of-need (CON) laws prevent ambulatory surgical centers (ASCs) from entering the market or expanding their services. This paper estimates the causal effects of state ASC-CON law repeal on the accessibility of medical services statewide, as well as for rural areas. Our findings show that CON law repeals increase ASCs per capita by 44-47% statewide and 92-112% in rural areas. Repealing ASC-CON laws causes a continuous increase in ASCs per capita, an effect which levels off ten years after repeal. Contrary to the 'cream-skimming' hypothesis, we find no evidence that CON repeal is associated with hospital closures in rural areas. Rather, some regression models show that repeal is associated with fewer medical service reductions.

Cross-encoders distilled from large language models (LLMs) are often more effective re-rankers than cross-encoders fine-tuned on manually labeled data. However, the distilled models usually do not reach their teacher LLM's effectiveness. To investigate whether best practices for fine-tuning cross-encoders on manually labeled data (e.g., hard-negative sampling, deep sampling, and listwise loss functions) can help to improve LLM ranker distillation, we construct and release a new distillation dataset: Rank-DistiLLM. In our experiments, cross-encoders trained on Rank-DistiLLM reach the effectiveness of LLMs while being orders of magnitude more efficient. Our code and data is available at https://github.com/webis-de/msmarco-llm-distillation.

The Simons Observatory (SO) is a cosmic microwave background (CMB) survey experiment that includes small-aperture telescopes (SATs) observing from an altitude of 5,200 m in the Atacama Desert in Chile. The SO SATs will cover six spectral bands between 27 and 280 GHz to search for primordial B-modes to a sensitivity of $\sigma(r)=0.002$, with quantified systematic errors well below this value. Each SAT is a self-contained cryogenic telescope with a 35$^\circ$ field of view, 42 cm diameter optical aperture, 40 K half-wave plate, 1 K refractive optics, and $<0.1$ K focal plane that holds $>12,000$ TES detectors. We describe the nominal design of the SATs and present details about the integration and testing for one operating at 93 and 145 GHz.

Utilizing large language models (LLMs) for zero-shot document ranking is done in one of two ways: (1) prompt-based re-ranking methods, which require no further training but are only feasible for re-ranking a handful of candidate documents due to computational costs; and (2) unsupervised contrastive trained dense retrieval methods, which can retrieve relevant documents from the entire corpus but require a large amount of paired text data for contrastive training. In this paper, we propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. Our method only requires prompts to guide an LLM to generate query and document representations for effective document retrieval. Specifically, we prompt the LLMs to represent a given text using a single word, and then use the last token's hidden states and the corresponding logits associated with the prediction of the next token to construct a hybrid document retrieval system. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations given by the LLM. Our experimental evaluation on the MSMARCO, TREC deep learning and BEIR zero-shot document retrieval datasets illustrates that this simple prompt-based LLM retrieval method can achieve a similar or higher retrieval effectiveness than state-of-the-art LLM embedding methods that are trained with large amounts of unsupervised data, especially when using a larger LLM., Comment: EMNLP2024 main

Existing definitions and associated conceptual frameworks for computer-based system safety should be revisited in light of real-world experiences from deploying autonomous vehicles. Current terminology used by industry safety standards emphasizes mitigation of risk from specifically identified hazards, and carries assumptions based on human-supervised vehicle operation. Operation without a human driver dramatically increases the scope of safety concerns, especially due to operation in an open world environment, a requirement to self-enforce operational limits, participation in an ad hoc sociotechnical system of systems, and a requirement to conform to both legal and ethical constraints. Existing standards and terminology only partially address these new challenges. We propose updated definitions for core system safety concepts that encompass these additional considerations as a starting point for evolving safe-ty approaches to address these additional safety challenges. These results might additionally inform framing safety terminology for other autonomous system applications., Comment: 19 pages, SafeComp 2024 preprint with additional appendix

This study presents a statistical time-domain approach for identifying transitions between climate states, referred to as breakpoints, using well-established econometric tools. We analyze a 67.1 million year record of the oxygen isotope ratio delta-O-18 derived from benthic foraminifera. The dataset is presented in Westerhold et al. (2020), where the authors use recurrence analysis to identify six climate states. Fixing the number of breakpoints to five, our procedure results in breakpoint estimates that closely align with those identified by Westerhold et al. (2020). By treating the number of breakpoints as a parameter to be estimated, we provide the statistical justification for more than five breakpoints in the time series. Further, our approach offers the advantage of constructing confidence intervals for the breakpoints, and it allows for testing the number of breakpoints present in the time series., Comment: Authors in alphabetical order

Existing cross-encoder re-rankers can be categorized as pointwise, pairwise, or listwise models. Pair- and listwise models allow passage interactions, which usually makes them more effective than pointwise models but also less efficient and less robust to input order permutations. To enable efficient permutation-invariant passage interactions during re-ranking, we propose a new cross-encoder architecture with inter-passage attention: the Set-Encoder. In Cranfield-style experiments on TREC Deep Learning and TIREx, the Set-Encoder is as effective as state-of-the-art listwise models while improving efficiency and robustness to input permutations. Interestingly, a pointwise model is similarly effective, but when additionally requiring the models to consider novelty, the Set-Encoder is more effective than its pointwise counterpart and retains its advantageous properties compared to other listwise models. Our code and models are publicly available at https://github.com/webis-de/set-encoder.

Time series analysis of delta-O-18 and delta-C-13 measurements from benthic foraminifera for purposes of paleoclimatology is challenging. The time series reach back tens of millions of years, they are relatively sparse in the early record and relatively dense in the later, the time stamps of the observations are not evenly spaced, and there are instances of multiple different observations at the same time stamp. The time series appear non-stationary over most of the historical record with clearly visible temporary trends of varying directions. In this paper, we propose a continuous-time state-space framework to analyze the time series. State space models are uniquely suited for this purpose, since they can accommodate all the challenging features mentioned above. We specify univariate models and joint bivariate models for the two time series of delta-O-18 and delta-C-13. The models are estimated using maximum likelihood by way of the Kalman filter recursions. The suite of models we consider has an interpretation as an application of the Butterworth filter. We propose model specifications that take the origin of the data from different studies into account and that allow for a partition of the total period into sub-periods reflecting different climate states. The models can be used, for example, to impute evenly time-stamped values by way of Kalman filtering. They can also be used, in future work, to analyze the relation to proxies for CO2 concentrations., Comment: Authors in alphabetical order

We present an estimate of the Atacama Cosmology Telescope (ACT) detector polarization angle systematic uncertainty from optics perturbation analysis using polarization-sensitive ray tracing in CODE V optical design software. Uncertainties in polarization angle calibration in CMB measurements can limit constraints on cosmic birefringence and other cosmological parameters sensitive to polarization leakage. Our framework estimates the angle calibration systematic uncertainties from possible displacements in lens positions and orientations, and anti-reflection coating (ARC) thicknesses and refractive indices. With millimeter displacements in lens positions and percent-level perturbations in ARC thicknesses and indices from design, we find the total systematic uncertainty for three ACT detector arrays operating between 90--220 GHz to be at the tenth of degree scale. Reduced lens position and orientation uncertainties from physical measurements could lead to a reduction in the systematic uncertainty estimated with the framework presented here. This optical modeling may inform polarization angle systematic uncertainties for current and future microwave polarimeters, such as the CCAT Observatory, Simons Observatory, and CMB-S4., Comment: 14 pages, 8 figures, version accepted for publication in Applied Optics

The emergence of Vec2Text -- a method for text embedding inversion -- has raised serious privacy concerns for dense retrieval systems which use text embeddings, such as those offered by OpenAI and Cohere. This threat comes from the ability for a malicious attacker with access to embeddings to reconstruct the original text. In this paper, we investigate various factors related to embedding models that may impact text recoverability via Vec2Text. We explore factors such as distance metrics, pooling functions, bottleneck pre-training, training with noise addition, embedding quantization, and embedding dimensions, which were not considered in the original Vec2Text paper. Through a comprehensive analysis of these factors, our objective is to gain a deeper understanding of the key elements that affect the trade-offs between the text recoverability and retrieval effectiveness of dense retrieval systems, offering insights for practitioners designing privacy-aware dense retrieval systems. We also propose a simple embedding transformation fix that guarantees equal ranking effectiveness while mitigating the recoverability risk. Overall, this study reveals that Vec2Text could pose a threat to current dense retrieval systems, but there are some effective methods to patch such systems.

An October 2023 crash between a GM Cruise robotaxi and a pedestrian in San Francisco resulted not only in a severe injury, but also dramatic upheaval at that company that will likely have lasting effects throughout the industry. Is-sues stem not just from the loss events themselves, but also from how Cruise mishandled dealing with their robotaxi dragging a pedestrian under the vehicle after the initial post-crash stop. External investigation reports provide raw material describing the incident and critique the company's response from a regulatory point of view, but exclude safety engineering recommendations from scope. We highlight specific facts and relationships among events by tying together different pieces of the external report material. We then explore safety lessons that might be learned related to: recognizing and responding to nearby mishaps, building an accurate world model of a post-collision scenario, the in-adequacy of a so-called "minimal risk condition" strategy in complex situations, poor organizational discipline in responding to a mishap, overly aggressive post-collision automation choices that made a bad situation worse, and a reluctance to admit to a mishap causing much worse organizational harm down-stream., Comment: 15 pages, 2 figures

Federated search, which involves integrating results from multiple independent search engines, will become increasingly pivotal in the context of Retrieval-Augmented Generation pipelines empowering LLM-based applications such as chatbots. These systems often distribute queries among various search engines, ranging from specialized (e.g., PubMed) to general (e.g., Google), based on the nature of user utterances. A critical aspect of federated search is resource selection - the selection of appropriate resources prior to issuing the query to ensure high-quality and rapid responses, and contain costs associated with calling the external search engines. However, current SOTA resource selection methodologies primarily rely on feature-based learning approaches. These methods often involve the labour intensive and expensive creation of training labels for each resource. In contrast, LLMs have exhibited strong effectiveness as zero-shot methods across NLP and IR tasks. We hypothesise that in the context of federated search LLMs can assess the relevance of resources without the need for extensive predefined labels or features. In this paper, we propose ReSLLM. Our ReSLLM method exploits LLMs to drive the selection of resources in federated search in a zero-shot setting. In addition, we devise an unsupervised fine tuning protocol, the Synthetic Label Augmentation Tuning (SLAT), where the relevance of previously logged queries and snippets from resources is predicted using an off-the-shelf LLM and then in turn used to fine-tune ReSLLM with respect to resource selection. Our empirical evaluation and analysis details the factors influencing the effectiveness of LLMs in this context. The results showcase the merits of ReSLLM for resource selection: not only competitive effectiveness in the zero-shot setting, but also obtaining large when fine-tuned using SLAT-protocol.

This paper considers Pseudo-Relevance Feedback (PRF) methods for dense retrievers in a resource constrained environment such as that of cheap cloud instances or embedded systems (e.g., smartphones and smartwatches), where memory and CPU are limited and GPUs are not present. For this, we propose a transformer-based PRF method (TPRF), which has a much smaller memory footprint and faster inference time compared to other deep language models that employ PRF mechanisms, with a marginal effectiveness loss. TPRF learns how to effectively combine the relevance feedback signals from dense passage representations. Specifically, TPRF provides a mechanism for modelling relationships and weights between the query and the relevance feedback signals. The method is agnostic to the specific dense representation used and thus can be generally applied to any dense retriever.

Screening documents is a tedious and time-consuming aspect of high-recall retrieval tasks, such as compiling a systematic literature review, where the goal is to identify all relevant documents for a topic. To help streamline this process, many Technology-Assisted Review (TAR) methods leverage active learning techniques to reduce the number of documents requiring review. BERT-based models have shown high effectiveness in text classification, leading to interest in their potential use in TAR workflows. In this paper, we investigate recent work that examined the impact of further pre-training epochs on the effectiveness and efficiency of a BERT-based active learning pipeline. We first report that we could replicate the original experiments on two specific TAR datasets, confirming some of the findings: importantly, that further pre-training is critical to high effectiveness, but requires attention in terms of selecting the correct training epoch. We then investigate the generalisability of the pipeline on a different TAR task, that of medical systematic reviews. In this context, we show that there is no need for further pre-training if a domain-specific BERT backbone is used within the active learning pipeline. This finding provides practical implications for using the studied active learning pipeline within domain-specific TAR tasks., Comment: Accepted at ECIR 2024 (reproducibility)

Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches., Comment: Accepted to ECIR2024 full paper (findings)

We describe team ielab from CSIRO and The University of Queensland's approach to the 2023 TREC Clinical Trials Track. Our approach was to use neural rankers but to utilise Large Language Models to overcome the issue of lack of training data for such rankers. Specifically, we employ ChatGPT to generate relevant patient descriptions for randomly selected clinical trials from the corpus. This synthetic dataset, combined with human-annotated training data from previous years, is used to train both dense and sparse retrievers based on PubmedBERT. Additionally, a cross-encoder re-ranker is integrated into the system. To further enhance the effectiveness of our approach, we prompting GPT-4 as a TREC annotator to provide judgments on our run files. These judgments are subsequently employed to re-rank the results. This architecture tightly integrates strong PubmedBERT-based rankers with the aid of SOTA Large Language Models, demonstrating a new approach to clinical trial retrieval., Comment: TREC Notebook

The global fraction of anthropogenically emitted carbon dioxide (CO$_2$) that stays in the atmosphere, the CO$_2$ airborne fraction, has been fluctuating around a constant value over the period 1959 to 2022. The consensus estimate of the airborne fraction is around $44\%$; the remaining $56\%$ is absorbed by the oceanic and terrestrials biospheres. In this study, we show that the conventional estimator of the airborne fraction, based on a ratio of changes in atmospheric CO$_2$ concentrations and CO$_2$ emissions, suffers from a number of statistical deficiencies, such as non-existence of moments and a non-Gaussian limiting distribution. We propose an alternative regression-based estimator of the airborne fraction that does not suffer from these deficiencies. We show that the regression-based estimator has a Gaussian limiting distribution and reduces estimation uncertainty substantially. Our empirical analysis leads to an estimate of the airborne fraction over 1959--2022 of $47.0\%$ ($\pm 1.1\%$; $1 \sigma$), implying a higher, and better constrained, estimate than the current consensus. Using climate model output, we show that a regression-based approach provides sensible estimates of the airborne fraction, also in future scenarios where emissions are at or near zero.

In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. We make our codebase publicly available at https://github.com/ielab/llm-qlm., Comment: 5 pages

We propose a novel zero-shot document ranking approach based on Large Language Models (LLMs): the Setwise prompting approach. Our approach complements existing prompting approaches for LLM-based zero-shot ranking: Pointwise, Pairwise, and Listwise. Through the first-of-its-kind comparative evaluation within a consistent experimental framework and considering factors like model size, token consumption, latency, among others, we show that existing approaches are inherently characterised by trade-offs between effectiveness and efficiency. We find that while Pointwise approaches score high on efficiency, they suffer from poor effectiveness. Conversely, Pairwise approaches demonstrate superior effectiveness but incur high computational overhead. Our Setwise approach, instead, reduces the number of LLM inferences and the amount of prompt token consumption during the ranking procedure, compared to previous methods. This significantly improves the efficiency of LLM-based zero-shot ranking, while also retaining high zero-shot ranking effectiveness. We make our code and results publicly available at \url{https://github.com/ielab/llm-rankers}., Comment: SIGIR2024 full paper

We present the requirements, design and evaluation of the cryogenic continuously rotating half-wave plate (CHWP) for the Simons Observatory (SO). SO is a cosmic microwave background (CMB) polarization experiment at Parque Astron\'{o}mico Atacama in northern Chile that covers a wide range of angular scales using both small (0.42 m) and large (6 m) aperture telescopes. In particular, the small aperture telescopes (SATs) focus on large angular scales for primordial B-mode polarization. To this end, the SATs employ a CHWP to modulate the polarization of the incident light at 8 Hz, suppressing atmospheric $1/f$ noise and mitigating systematic uncertainties that would otherwise arise due to the differential response of detectors sensitive to orthogonal polarizations. The CHWP consists of a 505 mm diameter achromatic sapphire HWP and a cryogenic rotation mechanism, both of which are cooled down to $\sim$50 K to reduce detector thermal loading. Under normal operation the HWP is suspended by a superconducting magnetic bearing and rotates with a constant 2 Hz frequency, controlled by an electromagnetic synchronous motor. We find that the number of superconductors and magnets that make up the superconducting magnetic bearing are important design parameters, especially for the rotation mechanism's vibration performance. The rotation angle is detected through an angular encoder with a noise level of 0.07 $\mu$rad$\sqrt{\mathrm{s}}$. During a cooldown, the rotor is held in place by a grip-and-release mechanism that serves as both an alignment device and a thermal path. In this paper we provide an overview of the SO SAT CHWP: its requirements, hardware design, and laboratory performance., Comment: 19 pages, 21 figures

Can ChatGPT provide evidence to support its answers? Does the evidence it suggests actually exist and does it really support its answer? We investigate these questions using a collection of domain-specific knowledge-based questions, specifically prompting ChatGPT to provide both an answer and supporting evidence in the form of references to external sources. We also investigate how different prompts impact answers and evidence. We find that ChatGPT provides correct or partially correct answers in about half of the cases (50.6% of the times), but its suggested references only exist 14% of the times. We further provide insights on the generated references that reveal common traits among the references that ChatGPT generates, and show how even if a reference provided by the model does exist, this reference often does not support the claims ChatGPT attributes to it. Our findings are important because (1) they are the first systematic analysis of the references created by ChatGPT in its answers; (2) they suggest that the model may leverage good quality information in producing correct answers, but is unable to attribute real evidence to support its answers. Prompts, raw result files and manual analysis are made publicly available.

Screening prioritisation in medical systematic reviews aims to rank the set of documents retrieved by complex Boolean queries. Prioritising the most important documents ensures that subsequent review steps can be carried out more efficiently and effectively. The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. However, the final title is only formulated at the end of the review process, which makes this approach impractical as it relies on ex post facto information. At the time of screening, only a rough working title is available, with which the BERT-based ranker performs significantly worse than with the final title. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based generative large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title., Comment: Preprints for Accepted paper in SIGIR-AP-2023, note that this is updated from ACM published paper. The working title was wrong in the ACM-published version due to a bug in data preprocessing; however, this does not have any influence on the final conclusion/observation made from the paper

Radiologists face high burnout rates, partially due to the increasing volume of Chest X-rays (CXRs) requiring interpretation and reporting. Automated CXR report generation holds promise for reducing this burden and improving patient care. While current models show potential, their diagnostic accuracy is limited. Our proposed CXR report generator integrates elements of the radiologist workflow and introduces a novel reward for reinforcement learning. Our approach leverages longitudinal data from a patient's prior CXR study and effectively handles cases where no prior study exist, thus mirroring the radiologist's workflow. In contrast, existing models typically lack this flexibility, often requiring prior studies for the model to function optimally. Our approach also incorporates all CXRs from a patient's study and distinguishes between report sections through section embeddings. Our reward for reinforcement learning leverages CXR-BERT, which forces our model to learn the clinical semantics of radiology reporting. We conduct experiments on publicly available datasets -- MIMIC-CXR and Open-i IU X-ray -- with metrics shown to more closely correlate with radiologists' assessment of reporting. Results from our study demonstrate that the proposed model generates reports that are more aligned with radiologists' reports than state-of-the-art models, such as those utilising large language models, reinforcement learning, and multi-task learning. The proposed model improves the diagnostic accuracy of CXR report generation, which could one day reduce radiologists' workload and enhance patient care. Our Hugging Face checkpoint (https://huggingface.co/aehrc/cxrmate) and code (https://github.com/aehrc/cxrmate) are publicly available.

Observations of the millimeter sky contain valuable information on a number of signals, including the blackbody cosmic microwave background (CMB), Galactic emissions, and the Compton-$y$ distortion due to the thermal Sunyaev-Zel'dovich (tSZ) effect. Extracting new insight into cosmological and astrophysical questions often requires combining multi-wavelength observations to spectrally isolate one component. In this work, we present a new arcminute-resolution Compton-$y$ map, which traces out the line-of-sight-integrated electron pressure, as well as maps of the CMB in intensity and E-mode polarization, across a third of the sky (around 13,000 sq.~deg.). We produce these through a joint analysis of data from the Atacama Cosmology Telescope (ACT) Data Release 4 and 6 at frequencies of roughly 93, 148, and 225 GHz, together with data from the \textit{Planck} satellite at frequencies between 30 GHz and 545 GHz. We present detailed verification of an internal linear combination pipeline implemented in a needlet frame that allows us to efficiently suppress Galactic contamination and account for spatial variations in the ACT instrument noise. These maps provide a significant advance, in noise levels and resolution, over the existing \textit{Planck} component-separated maps and will enable a host of science goals including studies of cluster and galaxy astrophysics, inferences of the cosmic velocity field, primordial non-Gaussianity searches, and gravitational lensing reconstruction of the CMB., Comment: The Compton-y map and associated products will be made publicly available upon publication of the paper. The CMB T and E mode maps will be made available when the DR6 maps are made public

This paper introduces a checksum algorithm that provides a new point in the performance/complexity/effectiveness checksum tradeoff space. It has better fault detection properties than single-sum and dual-sum modular addition checksums. It is also simpler to compute efficiently than a cyclic redundancy check (CRC) due to exploiting commonly available hardware and programming language support for unsigned integer division. The key idea is to compute a single running sum, but introduce a left shift by the size (in bits) of the modulus before performing the modular reduction after each addition step. This approach provides a Hamming Distance of 3 for longer data word lengths than dual-sum approaches such as the Fletcher checksum. Moreover, it provides this capability using a single running sum that is only twice the size of the final computed check value, while providing fault detection capabilities even better than large-block variants of dual-sum approaches that require larger division operations. A variant that includes a parity bit achieves Hamming Distance 4 for the same size check value, approximating the fault detection capabilities of a good CRC for about half the data word length attainable by such a CRC., Comment: 13 pages, 6 figures, example C source code

We present cosmological constraints from a gravitational lensing mass map covering 9400 sq. deg. reconstructed from CMB measurements made by the Atacama Cosmology Telescope (ACT) from 2017 to 2021. In combination with BAO measurements (from SDSS and 6dF), we obtain the amplitude of matter fluctuations $\sigma_8 = 0.819 \pm 0.015$ at 1.8% precision, $S_8\equiv\sigma_8({\Omega_{\rm m}}/0.3)^{0.5}=0.840\pm0.028$ and the Hubble constant $H_0= (68.3 \pm 1.1)\, \text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$ at 1.6% precision. A joint constraint with CMB lensing measured by the Planck satellite yields even more precise values: $\sigma_8 = 0.812 \pm 0.013$, $S_8\equiv\sigma_8({\Omega_{\rm m}}/0.3)^{0.5}=0.831\pm0.023$ and $H_0= (68.1 \pm 1.0)\, \text{km}\,\text{s}^{-1}\,\text{Mpc}^{-1}$. These measurements agree well with $\Lambda$CDM-model extrapolations from the CMB anisotropies measured by Planck. To compare these constraints to those from the KiDS, DES, and HSC galaxy surveys, we revisit those data sets with a uniform set of assumptions, and find $S_8$ from all three surveys are lower than that from ACT+Planck lensing by varying levels ranging from 1.7-2.1$\sigma$. These results motivate further measurements and comparison, not just between the CMB anisotropies and galaxy lensing, but also between CMB lensing probing $z\sim 0.5-5$ on mostly-linear scales and galaxy lensing at $z\sim 0.5$ on smaller scales. We combine our CMB lensing measurements with CMB anisotropies to constrain extensions of $\Lambda$CDM, limiting the sum of the neutrino masses to $\sum m_{\nu} < 0.13$ eV (95% c.l.), for example. Our results provide independent confirmation that the universe is spatially flat, conforms with general relativity, and is described remarkably well by the $\Lambda$CDM model, while paving a promising path for neutrino physics with gravitational lensing from upcoming ground-based CMB surveys., Comment: 32 pages, 17 figures, replaced with version accepted in ApJ (Feb 2024). Cosmological likelihood data and mass maps are public here: https://lambda.gsfc.nasa.gov/product/act/actadv_prod_table.html ; likelihood software is here: https://github.com/ACTCollaboration/act_dr6_lenslike . Also see companion papers Qu et al and MacCrann et al

We present new measurements of cosmic microwave background (CMB) lensing over $9400$ sq. deg. of the sky. These lensing measurements are derived from the Atacama Cosmology Telescope (ACT) Data Release 6 (DR6) CMB dataset, which consists of five seasons of ACT CMB temperature and polarization observations. We determine the amplitude of the CMB lensing power spectrum at $2.3\%$ precision ($43\sigma$ significance) using a novel pipeline that minimizes sensitivity to foregrounds and to noise properties. To ensure our results are robust, we analyze an extensive set of null tests, consistency tests, and systematic error estimates and employ a blinded analysis framework. The baseline spectrum is well fit by a lensing amplitude of $A_{\mathrm{lens}}=1.013\pm0.023$ relative to the Planck 2018 CMB power spectra best-fit $\Lambda$CDM model and $A_{\mathrm{lens}}=1.005\pm0.023$ relative to the $\text{ACT DR4} + \text{WMAP}$ best-fit model. From our lensing power spectrum measurement, we derive constraints on the parameter combination $S^{\mathrm{CMBL}}_8 \equiv \sigma_8 \left({\Omega_m}/{0.3}\right)^{0.25}$ of $S^{\mathrm{CMBL}}_8= 0.818\pm0.022$ from ACT DR6 CMB lensing alone and $S^{\mathrm{CMBL}}_8= 0.813\pm0.018$ when combining ACT DR6 and Planck NPIPE CMB lensing power spectra. These results are in excellent agreement with $\Lambda$CDM model constraints from Planck or $\text{ACT DR4} + \text{WMAP}$ CMB power spectrum measurements. Our lensing measurements from redshifts $z\sim0.5$--$5$ are thus fully consistent with $\Lambda$CDM structure growth predictions based on CMB anisotropies probing primarily $z\sim1100$. We find no evidence for a suppression of the amplitude of cosmic structure at low redshifts, Comment: 45+22 pages, 50 figures. v2 matches with published version in ApJ. Cosmological likelihood data and lensing maps are here: https://lambda.gsfc.nasa.gov/product/act/actadv_prod_table.html ; likelihood software is here: https://github.com/ACTCollaboration/act_dr6_lenslike . Also see companion papers Madhavacheril et al and MacCrann et al

Checksum algorithms are widely employed due to their use of a simple algorithm with fast computational speed to provide a basic detection capability for corrupted data. This paper describes the benefits of adding the design parameter of increased data block size for modular addition checksums, combined with an empirical approach to modulus selection. A longer processing block size with the right modulus can provide significantly better fault detection performance with no change in the number of bytes used to store the check value. In particular, a large-block dual-sum approach provides Hamming Distance 3-class fault detection performance for many times the data word length capability of previously studied Fletcher and Adler checksums. Moduli of 253 and 65525 are identified as being particularly effective for general-purpose checksum use., Comment: 21 pages, 15 figures

Generative pre-trained language models (GPLMs) like ChatGPT encode in the model's parameters knowledge the models observe during the pre-training phase. This knowledge is then used at inference to address the task specified by the user in their prompt. For example, for the question-answering task, the GPLMs leverage the knowledge and linguistic patterns learned at training to produce an answer to a user question. Aside from the knowledge encoded in the model itself, answers produced by GPLMs can also leverage knowledge provided in the prompts. For example, a GPLM can be integrated into a retrieve-then-generate paradigm where a search engine is used to retrieve documents relevant to the question; the content of the documents is then transferred to the GPLM via the prompt. In this paper we study the differences in answer correctness generated by ChatGPT when leveraging the model's knowledge alone vs. in combination with the prompt knowledge. We study this in the context of consumers seeking health advice from the model. Aside from measuring the effectiveness of ChatGPT in this context, we show that the knowledge passed in the prompt can overturn the knowledge encoded in the model and this is, in our experiments, to the detriment of answer correctness. This work has important implications for the development of more robust and transparent question-answering systems based on generative pre-trained language models.

Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable.

Decisions in agriculture are increasingly data-driven; however, valuable agricultural knowledge is often locked away in free-text reports, manuals and journal articles. Specialised search systems are needed that can mine agricultural information to provide relevant answers to users' questions. This paper presents AgAsk -- an agent able to answer natural language agriculture questions by mining scientific documents. We carefully survey and analyse farmers' information needs. On the basis of these needs we release an information retrieval test collection comprising real questions, a large collection of scientific documents split in passages, and ground truth relevance assessments indicating which passages are relevant to each question. We implement and evaluate a number of information retrieval models to answer farmers questions, including two state-of-the-art neural ranking models. We show that neural rankers are highly effective at matching passages to questions in this context. Finally, we propose a deployment architecture for AgAsk that includes a client based on the Telegram messaging platform and retrieval model deployed on commodity hardware. The test collection we provide is intended to stimulate more research in methods to match natural language to answers in scientific documents. While the retrieval models were evaluated in the agriculture domain, they are generalisable and of interest to others working on similar problems. The test collection is available at: \url{https://github.com/ielab/agvaluate}., Comment: 17 pages, submitted to IJDL

Medical systematic reviews typically require assessing all the documents retrieved by a search. The reason is two-fold: the task aims for ``total recall''; and documents retrieved using Boolean search are an unordered set, and thus it is unclear how an assessor could examine only a subset. Screening prioritisation is the process of ranking the (unordered) set of retrieved documents, allowing assessors to begin the downstream processes of the systematic review creation earlier, leading to earlier completion of the review, or even avoiding screening documents ranked least relevant. Screening prioritisation requires highly effective ranking methods. Pre-trained language models are state-of-the-art on many IR tasks but have yet to be applied to systematic review screening prioritisation. In this paper, we apply several pre-trained language models to the systematic review document ranking task, both directly and fine-tuned. An empirical analysis compares how effective neural methods compare to traditional methods for this task. We also investigate different types of document representations for neural methods and their impact on ranking performance. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods. However, BERT rankers and existing methods can actually be complementary, and thus, further improvements may be achieved if used in conjunction.

The Internet of Things (IoT) is growing fast. In 2018, there was approximately one connected device per person on earth and the number has been growing ever since. The devices interact with the environment via different modalities at the same time using sensors and actuators making the programs parallel. Yet, writing this type of programs is difficult because the devices have little computation power and memory, the platforms are heterogeneous and the languages are low level. Task Oriented Programming (TOP) is a declarative programming language paradigm that is used to express coordination of work, collaboration of users and systems, the distribution of shared data and the human-computer interaction. The mTask language is a specialized, yet full-fledged, multi-backend TOP language for IoT devices. With the bytecode interpretation backend and the integration with iTask, tasks can be executed on the device dynamically. This means that -- according to the current state of affairs -- tasks can be tailor-made at run time, compiled to device-agnostic bytecode and shipped to the device for interpretation. Tasks sent to the device are fully integrated in iTask to allow every form of interaction with the tasks such as observation of the task value and interaction with Shared Data Sources (SDSs). The entire IoT application -- both server and devices -- are programmed in a single language, albeit using two embedded Domain Specific Languages (EDSLs)., Comment: 53 pages, to be published in: Central European Functional Programming School: 8th Summer School, CEFP 2019, Budapest, Hungary, 17-21 July 2019, Revised Selected Papers, Springer, Cham

A Trial within Cohorts (TwiCs) study design is a trial design that uses the infrastructure of an observational cohort study to initiate a randomized trial. Upon cohort enrollment, cohort participants provided consent for being randomized in future studies without being informed. Once a new treatment is available, eligible cohort participants are randomly assigned to this new treatment or standard of care. Patients randomized to the treatment arm are offered this new treatment, which they can refuse. Patients who refuse will receive standard of care instead. Patients randomized to the standard of care arm, receive no information about the trial and continue receiving standard of care as part the cohort study. Standard cohort measures are used for outcome comparisons. The TwiCs study design aims to overcome some issues encountered in standard Randomized Controlled Trials (RCTs). An example of an issue in standard RCTs is the slow patient accrual, which a TwiCs study aims to improve by selecting patients using a cohort and only offering the intervention to patients in the active arm. In oncology, the TwiCs study design has gained increasing interest during the last decade. Despite its potential advantages over RCTs, the TwiCs study design is concerned with several methodological challenges that need careful consideration when planning a TwiCs study. In this article, we focus on these methodological challenges and reflect on them using experiences from TwiCs studies initiated in oncology. Important methodological challenges that are discussed are the timing of randomization, the issue of non-compliance (refusal) after randomization in the intervention arm, and the definition of the intention-to-treat effect in a TwiCs study and how this effect is related to its counterpart in standard RCTs.