Search

Abstract Objective An important criterion for selecting clinical trials to be compared in systematic reviews and meta-analyses is that they measure the same outcomes. However, this represents a challenge as there is a wide variety of outcomes, and it is difficult to standardize them for comparing clinical trials containing them. To address this challenge, we utilized our annotated dataset, which includes 211 abstracts of clinical trials related to glaucoma and type 2 diabetes mellitus. We then developed a tool that provides an overview of the annotated clinical trial information and enables users to group them by outcomes. Results Using our visualization tool, we formed groups of outcomes and their respective clinical trials. We were able to determine the most common outcomes in clinical trials for these diseases. As a case study on diabetes, we compared our outcomes with those consented by diabetes stakeholders and found that many of the grouped outcomes are aligned with the consented ones. This demonstrates that tools such as the one presented can help standardize clinical outcomes, which in turn help in the synthesis of clinical trials. Finally, we also offer some recommendations that could help in the automation of clinical trials based on outcome standardization.

BackgroundIn the field of structured information extraction, there are typically semantic and syntactic constraints on the output of information extraction (IE) systems. These constraints, however, can typically not be guaranteed using standard (fine-tuned) encoder-decoder architectures. This has led to the development of constrained decoding approaches which allow, e.g., to specify constraints in form of context-free grammars. An open question is in how far an IE system can be effectively guided by a domain-specific grammar to ensure that the output structures follow the requirements of a certain domain data model.MethodsIn this work we experimentally investigate the influence of grammar-constrained decoding as well as pointer generators on the performance of a domain-specific information extraction system. For this, we consider fine-tuned encoder-decoder models, Longformer and Flan-T5 in particular, and experimentally investigate whether the addition of grammar-constrained decoding and pointer generators improve information extraction results. Toward this goal, we consider the task of inducing structured representations from abstracts describing clinical trials, relying on the C-TrO ontology to semantically describe the clinical trials and their results. We frame the task as a slot filling problem where certain slots of templates need to be filled with token sequences occurring in the input text. We use a dataset comprising 211 annotated clinical trial abstracts about type 2 diabetes and glaucoma for training and evaluation. Our focus is on settings in which the available training data is in the order of a few hundred training examples, which we consider as a low-resource setting.ResultsIn all our experiments we could demonstrate the positive impact of grammar-constrained decoding, with an increase in F1 score of pp 0.351 (absolute score 0.413) and pp 0.425 (absolute score 0.47) for the best-performing models on type 2 diabetes and glaucoma datasets, respectively. The addition of the pointer generators had a detrimental impact on the results, decreasing F1 scores by pp 0.15 (absolute score 0.263) and pp 0.198 (absolute score 0.272) for the best-performing pointer generator models on type 2 diabetes and glaucoma datasets, respectively.ConclusionThe experimental results indicate that encoder-decoder models used for structure prediction for information extraction tasks in low-resource settings clearly benefit from grammar-constrained decoding guiding the output generation. In contrast, the evaluated pointer generator models decreased the performance drastically in some cases. Moreover, the performance of the pointer models appears to depend both on the used base model as well as the function used for aggregating the attention values. How the size of large language models affects the performance benefit of grammar-constrained decoding remains to be more structurally investigated in future work.