Search

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP., Comment: Technical Report

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

The rapid development of Large Language Models (LLMs) for healthcare applications has spurred calls for holistic evaluation beyond frequently-cited benchmarks like USMLE, to better reflect real-world performance. While real-world assessments are valuable indicators of utility, they often lag behind the pace of LLM evolution, likely rendering findings obsolete upon deployment. This temporal disconnect necessitates a comprehensive upfront evaluation that can guide model selection for specific clinical applications. We introduce MEDIC, a framework assessing LLMs across five critical dimensions of clinical competence: medical reasoning, ethics and bias, data and language understanding, in-context learning, and clinical safety. MEDIC features a novel cross-examination framework quantifying LLM performance across areas like coverage and hallucination detection, without requiring reference outputs. We apply MEDIC to evaluate LLMs on medical question-answering, safety, summarization, note generation, and other tasks. Our results show performance disparities across model sizes, baseline vs medically finetuned models, and have implications on model selection for applications requiring specific model strengths, such as low hallucination or lower cost of inference. MEDIC's multifaceted evaluation reveals these performance trade-offs, bridging the gap between theoretical capabilities and practical implementation in healthcare settings, ensuring that the most promising models are identified and adapted for diverse healthcare applications., Comment: Technical report

Cardiac arrest remains a leading cause of death worldwide, necessitating proactive measures for early detection and intervention. This project aims to develop and assess predictive models for the timely identification of cardiac arrest incidents, utilizing a comprehensive dataset of clinical parameters and patient histories. Employing machine learning (ML) algorithms like XGBoost, Gradient Boosting, and Naive Bayes, alongside a deep learning (DL) approach with Recurrent Neural Networks (RNNs), we aim to enhance early detection capabilities. Rigorous experimentation and validation revealed the superior performance of the RNN model, which effectively captures complex temporal dependencies within the data. Our findings highlight the efficacy of these models in accurately predicting cardiac arrest likelihood, emphasizing the potential for improved patient care through early risk stratification and personalized interventions. By leveraging advanced analytics, healthcare providers can proactively mitigate cardiac arrest risk, optimize resource allocation, and improve patient outcomes. This research highlights the transformative potential of machine learning and deep learning techniques in managing cardiovascular risk and advances the field of predictive healthcare analytics.

The analytical expression of the density-dependent binding energy per nucleon for the relativistic mean field (RMF), also known as the relativistic energy density functional (Relativistic-EDF), is used to obtain the isospin-dependent symmetry energy and its components for the isotopic chain of Sc, Ti, V, and Cr nuclei. The procedure of the coherent density fluctuation model is employed to formulate the Relativistic-EDF and Brueckner energy density functional (Brueckner-EDF) at local density. A few signatures of shell and/or sub-shell closure are observed in the symmetry energy and its components, i.e., surface and volume symmetry energy, far from the beta-stable region for odd-A Sc and V, and even-even Ti and Cr nuclei with non-linear NL3 and G3 parameter sets. A comparison is made with the results obtained from Relativistic-EDF and Brueckner-EDF with both NL3 and G3 for the considered isotopic chains. We find Relativistic-EDF outperforms the Brueckner-EDF in predicting the shell and/or sub-shell closure of neutron-rich isotopes at N = 50 for these atomic nuclei. Moreover, a relative comparison has been made for the results obtained with the non-linear NL3 and G3 parameter sets., Comment: 18 pages, 4 figures

Med42-v2 introduces a suite of clinical large language models (LLMs) designed to address the limitations of generic models in healthcare settings. These models are built on Llama3 architecture and fine-tuned using specialized clinical data. They underwent multi-stage preference alignment to effectively respond to natural prompts. While generic models are often preference-aligned to avoid answering clinical queries as a precaution, Med42-v2 is specifically trained to overcome this limitation, enabling its use in clinical settings. Med42-v2 models demonstrate superior performance compared to the original Llama3 models in both 8B and 70B parameter configurations and GPT-4 across various medical benchmarks. These LLMs are developed to understand clinical queries, perform reasoning tasks, and provide valuable assistance in clinical environments. The models are now publicly available at \href{https://huggingface.co/m42-health}{https://huggingface.co/m42-health}.

As large language models (LLMs) continue to evolve, the need for robust and standardized evaluation benchmarks becomes paramount. Evaluating the performance of these models is a complex challenge that requires careful consideration of various linguistic tasks, model architectures, and benchmarking methodologies. In recent years, various frameworks have emerged as noteworthy contributions to the field, offering comprehensive evaluation tests and benchmarks for assessing the capabilities of LLMs across diverse domains. This paper provides an exploration and critical analysis of some of these evaluation methodologies, shedding light on their strengths, limitations, and impact on advancing the state-of-the-art in natural language processing., Comment: 15 pages, 3 figures

Selective adsorptive capture and separation of chemically inert Kr and Xe noble gases with very low ppmv concentrations in air and industrial off-gases constitute an important technological challenge. Here, using a synergistic combination of experiment and theory, the microporous crystalline vanadomolybdates (MoVOx) as highly selective Kr sorbents are studied in detail. By varying the Mo/V ratios, we show for the first time that their one-dimensional pores can be fine-tuned for the size-selective adsorption of Kr over the larger Xe with selectivities reaching >100. Using extensive electronic structure calculations and grand canonical Monte-Carlo simulations, the competition between Kr uptake with CO2 and N2 was also investigated. As most materials reported so far are selective toward the larger, more polarizable Xe than Kr, this work constitutes an important step toward robust Kr-selective sorbent materials. This work highlights the potential use of porous crystalline transition metal oxides as energy-efficient and selective noble gas capture sorbents for industrial applications.

This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications., Comment: Published at AAAI 2024 Spring Symposium - Clinical Foundation Models

In this theoretical study, we have derived a simplified analytical expression for the binding energy per nucleon as a function of density and isospin asymmetry within the relativistic mean-field model. We have generated a new parameterization for the density-dependent DD-ME2 parameter set using the Relativistic-Hartree-Bogoliubov approach. Moreover, this work attempts to revisit the prior polynomial fitting in [Phy. Rev. C 103, 024305 (2021)] for the non-linear NL3 force parameter to provide a simplified set of equations for the energy density functional which is used for calculating the surface properties of finite nuclei. The current study improves the existing fitting procedure by effectively proposing a simpler model that provides comparably precise results while lowering the computational expense. To study the surface properties of finite nuclei with these parameterizations, we have adopted the coherent density fluctuation model, which effectively translates the quantities of nuclear matter from momentum space to coordinate space at local density. The isospin properties, such as symmetry energy and its surface and volume components, slope parameter, finite nuclear incompressibility, and surface incompressibility for even-even nuclei, are calculated for different mass regions. Moreover, we have studied the effect of density, weight function, and choice of relativistic force parameters on the surface properties. The consequence of this work will help to determine the properties of nuclei along the nuclear landscape and can facilitate an improved understanding of the island of stability, heavy-ion collision, and nucleosynthesis, among others.

Advancing ultrafast high-repetition-rate lasers to shortest pulse durations comprising only a few optical cycles while pushing their energy into the multi-millijoule regime opens a route towards terawatt-class peak powers at unprecedented average power. We explore this route via efficient post-compression of high-energy 1.2 ps pulses from an Ytterbium InnoSlab laser to 9.6 fs duration using gas-filled multi-pass cells (MPCs) at a repetition rate of 1 kHz. Employing dual-stage compression with a second MPC stage supporting a close-to-octave-spanning bandwidth enabled by dispersion-matched dielectric mirrors, a record compression factor of 125 is reached at 70% overall efficiency, delivering 6.7 mJ pulses with a peak power of about 0.3 TW. Moreover, we show that post-compression can improve the temporal contrast at picosecond delay by at least one order of magnitude. Our results demonstrate efficient conversion of multi-millijoule picosecond lasers to high-peak-power few-cycle sources, opening up new parameter regimes for laser plasma physics, high energy physics, biomedicine and attosecond science.

This paper introduces a set of formally defined and transparent problems for reinforcement learning algorithms with the following characteristics: (1) variable degrees of observability (non-Markov observations), (2) distal and sparse rewards, (3) variable and hierarchical reward structure, (4) multiple-task generation, (5) variable problem complexity. The environment provides 1D or 2D categorical observations, and takes actions as input. The core structure of the CT-graph is a multi-branch tree graph with arbitrary branching factor, depth, and observation sets that can be varied to increase the dimensions of the problem in a controllable and measurable way. Two main categories of states, decision states and wait states, are devised to create a hierarchy of importance among observations, typical of real-world problems. A large observation set can produce a vast set of histories that impairs memory-augmented agents. Variable reward functions allow for the easy creation of multiple tasks and the ability of an agent to efficiently adapt in dynamic scenarios where tasks with controllable degrees of similarities are presented. Challenging complexity levels can be easily achieved due to the exponential growth of the graph. The problem formulation and accompanying code provide a fast, transparent, and mathematically defined set of configurable tests to compare the performance of reinforcement learning algorithms, in particular in lifelong learning settings.

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future., Comment: To appear in Neural Networks