Search

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications., (© 2023. The Author(s).)

Background: Catheter ablation for atrial fibrillation (AF) is the most commonly performed electrophysiological procedure. To improve healthcare utilization, we aimed to compare the efficacy, efficiency, and safety of a minimalistic, streamlined single catheter ablation approach using a high-power short-duration ablation index-guided protocol (HPSD) vs. a control single-catheter protocol (SP)., Methods: Pulmonary vein isolation (PVI) with a single transseptal puncture without a multipolar mapping catheter was performed in 91 patients. Left atrial mapping was performed with the ablation catheter, only. Pacing maneuvers were used to confirm exit block. Procedural characteristics and success rates were compared using HPSD (n = 34) vs. a control (n = 57) ablation protocol. Freedom from recurrence was defined as a 1-year absence of AF episodes > 30 s, beyond the 3-month blanking period., Results: Using the HPSD protocol the median procedure and RF ablation time were significantly shorter compared to the SP, 84 (IQR 76-100) vs. 118 min (IQR 104-141) and 1036 (898-1184) vs. 1949s (IQR 1693-2261), respectively, p < .001 for all. First-pass PVI was achieved using the HPSD protocol in 88% and using the SP in 87% of patients, p = 1.0. No procedural complications were observed. High-sensitivity cardiac troponin levels were significantly higher in patients using the HPSD protocol compared to the SP. At 12 months follow-up, 87% patients remained free from AF with no differences between groups., Conclusions: A minimalistic, HPSD ablation index-guided PVI with a single-catheter approach is very efficient, safe, and associated with excellent clinical outcomes at 1 year., (© 2022. The Author(s).)