Search

Background: Recent advancements in large language models (LLMs) offer potential benefits in healthcare, particularly in processing extensive patient records. However, existing benchmarks do not fully assess LLMs' capability in handling real-world, lengthy clinical data. Methods: We present the LongHealth benchmark, comprising 20 detailed fictional patient cases across various diseases, with each case containing 5,090 to 6,754 words. The benchmark challenges LLMs with 400 multiple-choice questions in three categories: information extraction, negation, and sorting, challenging LLMs to extract and interpret information from large clinical documents. Results: We evaluated nine open-source LLMs with a minimum of 16,000 tokens and also included OpenAI's proprietary and cost-efficient GPT-3.5 Turbo for comparison. The highest accuracy was observed for Mixtral-8x7B-Instruct-v0.1, particularly in tasks focused on information retrieval from single and multiple patient documents. However, all models struggled significantly in tasks requiring the identification of missing information, highlighting a critical area for improvement in clinical data interpretation. Conclusion: While LLMs show considerable potential for processing long clinical documents, their current accuracy levels are insufficient for reliable clinical use, especially in scenarios requiring the identification of missing information. The LongHealth benchmark provides a more realistic assessment of LLMs in a healthcare setting and highlights the need for further model refinement for safe and effective clinical application. We make the benchmark and evaluation code publicly available., Comment: 11 pages, 3 figures, 5 tables

In this paper we prove motivic versions of the Langlands-Shelstad Fundamental Lemma and Ng\^o's Geometric Stabilization. To achieve this, we follow the strategy from the recent proof by Groechenig, Wyss and Ziegler which avoided the use of perverse sheaves using instead $p$-adic integration and Tate duality. We make a key use of a construction of Denef and Loeser which assigns a virtual motive to any definable set in the theory of pseudo-finite fields., Comment: 53 pages

As large language models (LLMs) like OpenAI's GPT series continue to make strides, we witness the emergence of artificial intelligence applications in an ever-expanding range of fields. In medicine, these LLMs hold considerable promise for improving medical workflows, diagnostics, patient care, and education. Yet, there is an urgent need for open-source models that can be deployed on-premises to safeguard patient privacy. In our work, we present an innovative dataset consisting of over 160,000 entries, specifically crafted to fine-tune LLMs for effective medical applications. We investigate the impact of fine-tuning these datasets on publicly accessible pre-trained LLMs, and subsequently, we juxtapose the performance of pre-trained-only models against the fine-tuned models concerning the examinations that future medical doctors must pass to achieve certification.

This paper presents medBERTde, a pre-trained German BERT model specifically designed for the German medical domain. The model has been trained on a large corpus of 4.7 Million German medical documents and has been shown to achieve new state-of-the-art performance on eight different medical benchmarks covering a wide range of disciplines and medical document types. In addition to evaluating the overall performance of the model, this paper also conducts a more in-depth analysis of its capabilities. We investigate the impact of data deduplication on the model's performance, as well as the potential benefits of using more efficient tokenization methods. Our results indicate that domain-specific models such as medBERTde are particularly useful for longer texts, and that deduplication of training data does not necessarily lead to improved performance. Furthermore, we found that efficient tokenization plays only a minor role in improving model performance, and attribute most of the improved performance to the large amount of training data. To encourage further research, the pre-trained model weights and new benchmarks based on radiological data are made publicly available for use by the scientific community., Comment: Keno K. Bressem and Jens-Michalis Papaioannou and Paul Grundmann contributed equally

The use of deep neural models for diagnosis prediction from clinical text has shown promising results. However, in clinical practice such models must not only be accurate, but provide doctors with interpretable and helpful results. We introduce ProtoPatient, a novel method based on prototypical networks and label-wise attention with both of these abilities. ProtoPatient makes predictions based on parts of the text that are similar to prototypical patients - providing justifications that doctors understand. We evaluate the model on two publicly available clinical datasets and show that it outperforms existing baselines. Quantitative and qualitative evaluations with medical doctors further demonstrate that the model provides valuable explanations for clinical decision support., Comment: AACL-IJCNLP 2022 Main Conference (Long Paper)

We prove a general finiteness statement for the ordered abelian group of tropical functions on skeleta in Berkovich analytifications of algebraic varieties. Our approach consists in working in the framework of stable completions of algebraic varieties, a model-theoretic version of Berkovich analytifications, for which we prove a similar result, of which the former one is a consequence., Comment: 47 pages

Clinical phenotyping enables the automatic extraction of clinical conditions from patient records, which can be beneficial to doctors and clinics worldwide. However, current state-of-the-art models are mostly applicable to clinical notes written in English. We therefore investigate cross-lingual knowledge transfer strategies to execute this task for clinics that do not use the English language and have a small amount of in-domain data available. We evaluate these strategies for a Greek and a Spanish clinic leveraging clinical notes from different clinical domains such as cardiology, oncology and the ICU. Our results reveal two strategies that outperform the state-of-the-art: Translation-based methods in combination with domain-specific encoders and cross-lingual encoders plus adapters. We find that these strategies perform especially well for classifying rare phenotypes and we advise on which method to prefer in which situation. Our results show that using multilingual data overall improves clinical phenotyping models and can compensate for data sparseness., Comment: LREC 2022 submmision: January 2022

In this work, we propose a dynamical systems perspective on the modeling of sepsis and its organ-damaging consequences. We develop a functional two-layer network model for sepsis based upon the interaction of parenchymal cells and immune cells via cytokines, and the coevolutionary dynamics of parenchymal, immune cells, and cytokines. By means of the simple paradigmatic model of phase oscillators in a two-layer system, we analyze the emergence of organ threatening interactions between the dysregulated immune system and the parenchyma. We demonstrate that the complex cellular cooperation between parenchyma and stroma (immune layer) either in the physiological or in the pathological case can be related to dynamical patterns of the network. In this way we explain sepsis by the dysregulation of the healthy homeostatic state (frequency synchronized) leading to a pathological state (desynchronized or multifrequency cluster) in the parenchyma. We provide insight into the complex stabilizing and destabilizing interplay of parenchyma and stroma by determining critical interaction parameters. The coupled dynamics of parenchymal cells (metabolism) and nonspecific immune cells (response of the innate immune system) is represented by nodes of a duplex layer. Cytokine interaction is modeled by adaptive coupling weights between nodes representing immune cells (with fast adaptation timescale) and parenchymal cells (slow adaptation timescale), and between pairs of parenchymal and immune cells in the duplex network (fixed bidirectional coupling). The proposed model allows for a functional description of organ dysfunction in sepsis and the recurrence risk in a plausible pathophysiological context.

Due to the decarbonisation of energy use, the power system is expected to become the backbone of all energy sectors and thus the basic critical infrastructure. High penetration with distributed energy resources demands the coordination of a large number of prosumers, partly controlled by home energy management systems (HEMS), to be designed in such a way that the power system's operational limits are not violated. On the grid level, distribution management systems (DMS) try to keep the power system in the normal operational state. On the prosumer level, distributed HEMS optimise the internal power flows by using batteries, photovoltaic generators, or flexible loads optimally. The vision of the ODiS (Organic Distribution System) initiative is to develop an architecture to operate a distribution grid reliably, with high resiliency, and fully autonomously by developing "organic" HEMS and DMS which possess multiple self-* capabilities. Thus, ODiS seeks answers to the following question: How can we create the most appropriate models, techniques, and algorithms to develop novel kinds of self-configuring, self-organising, self-healing, and self-optimising DMS that are integrally coupled with the distributed HEMS? In this article, the vision of ODiS is presented in detail based on a thorough review of the state of the art., Comment: Preprint submitted to Energies

Decision support systems based on clinical notes have the potential to improve patient care by pointing doctors towards overseen risks. Predicting a patient's outcome is an essential part of such systems, for which the use of deep neural networks has shown promising results. However, the patterns learned by these networks are mostly opaque and previous work revealed flaws regarding the reproduction of unintended biases. We thus introduce an extendable testing framework that evaluates the behavior of clinical outcome models regarding changes of the input. The framework helps to understand learned patterns and their influence on model decisions. In this work, we apply it to analyse the change in behavior with regard to the patient characteristics gender, age and ethnicity. Our evaluation of three current clinical NLP models demonstrates the concrete effects of these characteristics on the models' decisions. They show that model behavior varies drastically even when fine-tuned on the same data and that allegedly best-performing models have not always learned the most medically plausible patterns., Comment: NeurIPS 2021 Research2Clinics Workshop, Bridging the Gap: From Machine Learning Research to Clinical Practice

Retrieving answer passages from long documents is a complex task requiring semantic understanding of both discourse and document context. We approach this challenge specifically in a clinical scenario, where doctors retrieve cohorts of patients based on diagnoses and other latent medical aspects. We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. In addition, we contribute a novel retrieval dataset based on clinical notes to simulate this scenario on a large corpus of clinical notes. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. From our extensive evaluation on MIMIC-III and three other healthcare datasets, we report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages. This makes the model powerful especially in zero-shot scenarios where only limited training data is available.

In this study, we provide a dynamical systems perspective to the modelling of pathological states induced by tumors or infection. A unified disease model is established using the innate immune system as the reference point. We propose a two-layer network model for carcinogenesis and sepsis based upon the interaction of parenchymal cells and immune cells via cytokines, and the co-evolutionary dynamics of parenchymal, immune cells, and cytokines. Our aim is to show that the complex cellular cooperation between parenchyma and stroma (immune layer) in the physiological and pathological case can be qualitatively and functionally described by a simple paradigmatic model of phase oscillators. By this, we explain carcinogenesis, tumor progression, and sepsis by destabilization of the healthy homeostatic state (frequency synchronized), and emergence of a pathological state (desynchronized or multifrequency cluster). The coupled dynamics of parenchymal cells (metabolism) and nonspecific immune cells (reaction of innate immune system) are represented by nodes of a duplex layer. The cytokine interaction is modeled by adaptive coupling weights between the nodes representing the immune cells (with fast adaptation time scale) and the parenchymal cells (slow adaptation time scale) and between the pairs of parenchymal and immune cells in the duplex network (fixed bidirectional coupling). Thereby, carcinogenesis, organ dysfunction in sepsis, and recurrence risk can be described in a correct functional context., Comment: 27 pages, 9 figures

Magnetic fluctuations is the leading candidate for pairing in cuprate, iron-based and heavy fermion superconductors. This view is challenged by the recent discovery of nodeless superconductivity in CeCu$_2$Si$_2$, and calls for a detailed understanding of the corresponding magnetic fluctuations. Here, we mapped out the magnetic excitations in \ys{superconducting (S-type)} CeCu$_2$Si$_2$ using inelastic neutron scattering, finding a strongly asymmetric dispersion for $E\lesssim1.5$~meV, which at higher energies evolve into broad columnar magnetic excitations that extend to $E\gtrsim 5$ meV. While low-energy magnetic excitations exhibit marked three-dimensional characteristics, the high-energy magnetic excitations in CeCu$_2$Si$_2$ are almost two-dimensional, reminiscent of paramagnons found in cuprate and iron-based superconductors. By comparing our experimental findings with calculations in the random-phase approximation,we find that the magnetic excitations in CeCu$_2$Si$_2$ arise from quasiparticles associated with its heavy electron band, which are also responsible for superconductivity. Our results provide a basis for understanding magnetism and superconductivity in CeCu$_2$Si$_2$, and demonstrate the utility of neutron scattering in probing band renormalization in heavy fermion metals.

We report detailed thermodynamic studies on high-quality single crystals of the centrosymmetric skyrmion-hosting intermetallic Gd2PdSi3 by means of high-resolution capacitance dilatometry in fields up to 15 T which are complemented by specific heat and magnetization studies. Our dilatometric measurements show magnetoelastic effects associated with antiferromagnetic order at TN1 = 22.3 K and TN2 = 19.7 K, as well as strong field effects in an applied magnetic field of 15 Tup to 200 K (150 K) for B||c (B||a*, i.e. B perp c). The data allow us to complete the magneticphase diagram, including a new feature at T* approx 12 K below which a new degree of freedom becomesrelevant. For the first time, the magnetic B vs. T phase diagram for the a*-axis is also reported. Gr\"uneisen analysis shows the onset of magnetic contributions around 60 K, i.e., well above TN1. Uniaxial pressure dependencies of opposite sign, -1.3 K/GPa and 0.3 K/GPa, are extracted for the out-of-plane and in-plane directions at TN1. For T* we obtain dT*/dpc= 1.4 K/GPa. In particular we elucidate thermodynamic properties of the recently discovered skyrmion lattice phase and show that it is strongly enhanced by uniaxial pressure.

Physical data layout is an important performance factor for modern databases. Clustering, i.e., storing similar values in proximity, can lead to performance gains in several ways. We present an automated model to determine beneficial clustering columns and a clustering algorithm for the column-oriented, memory-resident database Hyrise. To automatically select clustering columns, the model analyzes the database's workload and provides estimates by how much certain clustering columns would impact the workload's latency. We evaluate the precision of the model's estimates, as well as the overall quality of its clustering suggestions. To apply a determined clustering configuration, we developed an online clustering algorithm. The clustering algorithm supports an arbitrary number of clustering dimensions. We show that the algorithm is robust against concurrently running data modifying queries. We obtain a 5% latency reduction for the TPC-H benchmark when clustering the lineitem table and a 4% latency reduction for the TPC-DS benchmark when clustering the store_sales table.

Outcome prediction from clinical text can prevent doctors from overlooking possible risks and help hospitals to plan capacities. We simulate patients at admission time, when decision support can be especially valuable, and contribute a novel admission to discharge task with four common outcome prediction targets: Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction. The ideal system should infer outcomes based on symptoms, pre-conditions and risk factors of a patient. We evaluate the effectiveness of language models to handle this scenario and propose clinical outcome pre-training to integrate knowledge about patient outcomes from multiple public sources. We further present a simple method to incorporate ICD code hierarchy into the models. We show that our approach improves performance on the outcome tasks against several baselines. A detailed analysis reveals further strengths of the model, including transferability, but also weaknesses such as handling of vital values and inconsistencies in the underlying data., Comment: EACL 2021

Explainability and interpretability are two important concepts, the absence of which can and should impede the application of well-performing neural networks to real-world problems. At the same time, they are difficult to incorporate into the large, black-box models that achieve state-of-the-art results in a multitude of NLP tasks. Bidirectional Encoder Representations from Transformers (BERT) is one such black-box model. It has become a staple architecture to solve many different NLP tasks and has inspired a number of related Transformer models. Understanding how these models draw conclusions is crucial for both their improvement and application. We contribute to this challenge by presenting VisBERT, a tool for visualizing the contextual token representations within BERT for the task of (multi-hop) Question Answering. Instead of analyzing attention weights, we focus on the hidden states resulting from each encoder block within the BERT model. This way we can observe how the semantic representations are transformed throughout the layers of the model. VisBERT enables users to get insights about the model's internal state and to explore its inference steps or potential shortcomings. The tool allows us to identify distinct phases in BERT's transformations that are similar to a traditional NLP pipeline and offer insights during failed predictions., Comment: Published in WWW '20: Companion Proceedings of the Web Conference 2020

The $^2S_{1/2} - ^2P_{1/2}$ and $^2S_{1/2} - ^2P_{3/2}$ transitions in Li-like carbon ions stored and cooled at a velocity of $\beta \approx 0.47$ in the Experimental Storage Ring (ESR) at the GSI Helmholtz Centre in Darmstadt have been investigated in a laser spectroscopy experiment. Resonance wavelengths were obtained using a new continuous-wave UV laser system and a novel extreme UV (XUV) detection system to detect forward emitted fluorescence photons. The results obtained for the two transitions are compared to existing experimental and theoretical data. A discrepancy found in an earlier laser spectroscopy measurement at the ESR with results from plasma spectroscopy and interferometry has been resolved and agreement between experiment and theory is confirmed., Comment: 12 pages, 6 figures

We report on experimental investigations of proton acceleration from solid foils irradiated with PW-class laser-pulses, where highest proton cut-off energies were achieved for temporal pulse parameters that varied significantly from those of an ideally Fourier transform limited (FTL) pulse. Controlled spectral phase modulation of the driver laser by means of an acousto-optic programmable dispersive filter enabled us to manipulate the temporal shape of the last picoseconds around the main pulse and to study the effect on proton acceleration from thin foil targets. The results show that applying positive third order dispersion values to short pulses is favourable for proton acceleration and can lead to maximum energies of 70 MeV in target normal direction at 18 J laser energy for thin plastic foils, significantly enhancing the maximum energy compared to ideally compressed FTL pulses. The paper further proves the robustness and applicability of this enhancement effect for the use of different target materials and thicknesses as well as laser energy and temporal intensity contrast settings. We demonstrate that application relevant proton beam quality was reliably achieved over many months of operation with appropriate control of spectral phase and temporal contrast conditions using a state-of-the-art high-repetition rate PW laser system.

We present Contextual Discourse Vectors (CDV), a distributed document representation for efficient answer retrieval from long healthcare documents. Our approach is based on structured query tuples of entities and aspects from free text and medical taxonomies. Our model leverages a dual encoder architecture with hierarchical LSTM layers and multi-task training to encode the position of clinical entities and aspects alongside the document discourse. We use our continuous representations to resolve queries with short latency using approximate nearest neighbor search on sentence level. We apply the CDV model for retrieving coherent answer passages from nine English public health resources from the Web, addressing both patients and medical professionals. Because there is no end-to-end training data available for all application scenarios, we train our model with self-supervised data from Wikipedia. We show that our generalized model significantly outperforms several state-of-the-art baselines for healthcare passage ranking and is able to adapt to heterogeneous domains without additional fine-tuning., Comment: The Web Conference 2020 (WWW '20)

We provide additional statistical background for the methodology developed in the clinical analysis of knee osteoarthritis in "A Precision Medicine Approach to Develop and Internally Validate Optimal Exercise and Weight Loss Treatments for Overweight and Obese Adults with Knee Osteoarthritis" (Jiang et al. 2020). Jiang et al. 2020 proposed a pipeline to learn optimal treatment rules with precision medicine models and compared them with zero-order models with a Z-test. The model performance was based on value functions, a scalar that predicts the future reward of each decision rule. The jackknife (i.e., leave-one-out cross validation) method was applied to estimate the value function and its variance of several outcomes in IDEA. IDEA is a randomized clinical trial studying three interventions (exercise (E), dietary weight loss (D), and D+E) on overweight and obese participants with knee osteoarthritis. In this report, we expand the discussion and justification with additional statistical background. We elaborate more on the background of precision medicine, the derivation of the jackknife estimator of value function and its estimated variance, the consistency property of jackknife estimator, as well as additional simulation results that reflect more of the performance of jackknife estimators. We recommend reading Jiang et al. 2020 for clinical application and interpretation of the optimal ITR of knee osteoarthritis as well as the overall understanding of the pipeline and recommend using this article to understand the underlying statistical derivation and methodology.

We prove that the moduli spaces of twisted $\mathrm{SL}_n$ and $\mathrm{PGL}_n$-Higgs bundles on a smooth projective curve have the same (stringy) class in the Grothendieck ring of rational Chow motives. On the level of Hodge numbers this was conjectured by Hausel and Thaddeus, and recently proven by Groechenig, Ziegler and the second author. To adapt their argument, which relies on p-adic integration, we use a version of motivic integration with values in rational Chow motives and the geometry of N\'eron models to evaluate such integrals on Hitchin fibers., Comment: 37 pages, comments welcome

We explain how non-archimedean integrals considered by Chambert-Loir and Ducros naturally arise in asymptotics of families of complex integrals. To perform this analysis we work over a non-standard model of the field of complex numbers, which is endowed at the same time with an archimedean and a non-archimedean norm. Our main result states the existence of a natural morphism between bicomplexes of archimedean and non-archimedean forms which is compatible with integration., Comment: 69 pages

We generalize the tree doubling and Christofides algorithm, the two most common approximations for TSP, to parameterized approximations for ATSP. The parameters we consider for the respective parameterizations are upper bounded by the number of asymmetric distances in the given instance, which yields algorithms to efficiently compute constant factor approximations also for moderately asymmetric TSP instances. As generalization of the Christofides algorithm, we derive a parameterized 2.5-approximation, where the parameter is the size of a vertex cover for the subgraph induced by the asymmetric edges. Our generalization of the tree doubling algorithm gives a parameterized 3-approximation, where the parameter is the number of asymmetric edges in a given minimum spanning arborescence. Both algorithms are also stated in the form of additive lossy kernelizations, which allows to combine them with known polynomial time approximations for ATSP. Further, we combine them with a notion of symmetry relaxation which allows to trade approximation guarantee for runtime. We complement our results by experimental evaluations, which show that generalized tree-doubling frequently outperforms generalized Christofides with respect to parameter size.

In the series R2PdSi3, Nd2PdSi3 is an anomalous compound in the sense that it exhibits ferromagnetic order unlike other members in this family. The magnetic ordering temperature is also unusually high compared to the expected value for a Nd-based system, assuming 4f localization. Here, we have studied the electronic structure of single crystalline Nd2PdSi3 employing high resolution photoemission spectroscopy and ab initio band structure calculations. Theoretical results obtained for the effective electron correlation strength of 6 eV corroborate well with the experimental valence band spectra. While there is significant Pd 4d-Nd 4f hybridization, the states near the Fermi level are found to be dominated by hybridized Nd 4f-Si 3p states. Nd 3d core level spectrum exhibits multiple features manifesting strong final state effects due to electron correlation, charge transfer and collective excitations. These results serve as one of the rare demonstrations of hybridization of Nd 4$f$ states with the conduction electrons possibly responsible for the exoticity of this compound.

Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks. However, understanding of their internal functioning is still insufficient and unsatisfactory. In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states. Unlike previous research, which mainly focuses on explaining Transformer models by their attention weights, we argue that hidden states contain equally valuable information. Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task. We inspect how QA models transform token vectors in order to find the correct answer. To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer. Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process. Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks. The system can therefore implicitly incorporate task-specific information into its token representations. Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers., Comment: Accepted at CIKM 2019

This paper proposes a model predictive control (MPC)-based energy management system to address communication failures in an islanded microgrid (MG). The energy management system relies on a communication network to monitor and control power generation in the units. A communication failure to a unit inhibits the ability of the MPC to change the power of the unit. However, this unit is electrically connected to the network and ignoring this aspect could have adverse effect in the operation of the microgrid. Therefore, this paper considers an MPC design that includes the electrical connectivity of the unit during communication failures. This paper also highlights the benefit of including the lower layer control behaviour of the MG to withstand communication failures. The proposed approaches are validated with a case study., Comment: accepted for publication in CDC 2019