Search

Background/aims: This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases., Methods: We tested GPT-4 on 422 Journal of the American Medical Association Ophthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort., Results: Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020)., Conclusion: Improved prompting enhances GPT-4's performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Evidence on the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model (LLM), in the ophthalmology question-answering domain is needed., Methods: We tested GPT-4 on two 260-question multiple choice question sets from the Basic and Clinical Science Course (BCSC) Self-Assessment Program and the OphthoQuestions question banks. We compared the accuracy of GPT-4 models with varying temperatures (creativity setting) and evaluated their responses in a subset of questions. We also compared the best-performing GPT-4 model to GPT-3.5 and to historical human performance., Results: GPT-4-0.3 (GPT-4 with a temperature of 0.3) achieved the highest accuracy among GPT-4 models, with 75.8% on the BCSC set and 70.0% on the OphthoQuestions set. The combined accuracy was 72.9%, which represents an 18.3% raw improvement in accuracy compared with GPT-3.5 (p<0.001). Human graders preferred responses from models with a temperature higher than 0 (more creative). Exam section, question difficulty and cognitive level were all predictive of GPT-4-0.3 answer accuracy. GPT-4-0.3's performance was numerically superior to human performance on the BCSC (75.8% vs 73.3%) and OphthoQuestions (70.0% vs 63.0%), but the difference was not statistically significant (p=0.55 and p=0.09)., Conclusion: GPT-4, an LLM trained on non-ophthalmology-specific data, performs significantly better than its predecessor on simulated ophthalmology board-style exams. Remarkably, its performance tended to be superior to historical human performance, but that difference was not statistically significant in our study., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ.)

Background/rationale: Artificial intelligence (AI)-based clinical decision support tools, being developed across multiple fields in medicine, need to be evaluated for their impact on the treatment and outcomes of patients as well as optimisation of the clinical workflow. The RAZORBILL study will investigate the impact of advanced AI segmentation algorithms on the disease activity assessment in patients with neovascular age-related macular degeneration (nAMD) by enriching three-dimensional (3D) retinal optical coherence tomography (OCT) scans with automated fluid and layer quantification measurements., Methods: RAZORBILL is an observational, multicentre, multinational, open-label study, comprising two phases: (a) clinical data collection (phase I): an observational study design, which enforces neither strict visit schedule nor mandated treatment regimen was chosen as an appropriate design to collect data in a real-world clinical setting to enable evaluation in phase II and (b) OCT enrichment analysis (phase II): de-identified 3D OCT scans will be evaluated for disease activity. Within this evaluation, investigators will review the scans once enriched with segmentation results (i.e., highlighted and quantified pathological fluid volumes) and once in its original (i.e., non-enriched) state. This review will be performed using an integrated crossover design, where investigators are used as their own controls allowing the analysis to account for differences in expertise and individual disease activity definitions., Conclusions: In order to apply novel AI tools to routine clinical care, their benefit as well as operational feasibility need to be carefully investigated. RAZORBILL will inform on the value of AI-based clinical decision support tools. It will clarify if these can be implemented in clinical treatment of patients with nAMD and whether it allows for optimisation of individualised treatment in routine clinical care., Competing Interests: Competing interests: FGH reports grants and personal fees from Novartis, during the conduct of the study; grants and personal fees from Acucela, Allergan, Apellis, Bayer, Bioeq/Formycon, Geuder, Heidelberg Engineering, IvericBio, Kanghong, Roche, and Zeiss; personal fees from Grayburg Vision, Oxurion, and Pixium; grants from Biogen, and Optos, outside the submitted work. RAG reports personal fees from Allergan, Bausch & Lomb, Nidek, and Novartis, outside the submitted work. FB reports personal fees from Allergan, Bayer, Boehringer-Ingelheim, Fidia Sooft, Hofmann La Roche, Novartis, NTC Pharma, Sifi, Thrombogenics, and Zeiss, outside the submitted work. RD reports grants and personal fees from Bayer, and personal fees from Novartis, outside the submitted work. LO reports grants from Novartis, and personal fees from Bayer, outside the submitted work. DP has nothing to disclose. GS reports grants and personal fees from Centervue, Nidek and Novartis; grants from Carl Zeiss Meditec, Heidelberg Engineering, Ocular Instruments, Optos, Optovue, Quantel Medical, and Topcon; personal fees from Apellis, Allergan, Astellas, Bayer, Boheringer, Genentech, Graybug, Iveric, Roche, Chengdu Kanghong Biotechnology, Kyoto Drug Discovery & Development, outside the submitted work. AW reports grants from Novartis, outside the submitted work. DL reports full-time employment of Novartis Pharma AG. AC reports full-time employment and shareholder of Novartis Pharma AG. BG reports full-time employment at Novartis Pharma GmbH and is a shareholder of Novartis Pharma AG., (© Author(s) (or their employer(s)) 2023. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Aims: Automated machine learning (AutoML) is a novel tool in artificial intelligence (AI). This study assessed the discriminative performance of AutoML in differentiating retinal vein occlusion (RVO), retinitis pigmentosa (RP) and retinal detachment (RD) from normal fundi using ultra-widefield (UWF) pseudocolour fundus images., Methods: Two ophthalmologists without coding experience carried out AutoML model design using a publicly available image data set (2137 labelled images). The data set was reviewed for low-quality and mislabeled images and then uploaded to the Google Cloud AutoML Vision platform for training and testing. We designed multiple binary models to differentiate RVO, RP and RD from normal fundi and compared them to bespoke models obtained from the literature. We then devised a multiclass model to detect RVO, RP and RD. Saliency maps were generated to assess the interpretability of the model., Results: The AutoML models demonstrated high diagnostic properties in the binary classification tasks that were generally comparable to bespoke deep-learning models (area under the precision-recall curve (AUPRC) 0.921-1, sensitivity 84.91%-89.77%, specificity 78.72%-100%). The multiclass AutoML model had an AUPRC of 0.876, a sensitivity of 77.93% and a positive predictive value of 82.59%. The per-label sensitivity and specificity, respectively, were normal fundi (91.49%, 86.75%), RVO (83.02%, 92.50%), RP (72.00%, 100%) and RD (79.55%,96.80%)., Conclusion: AutoML models created by ophthalmologists without coding experience can detect RVO, RP and RD in UWF images with very good diagnostic accuracy. The performance was comparable to bespoke deep-learning models derived by AI experts for RVO and RP but not for RD., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2023. No commercial re-use. See rights and permissions. Published by BMJ.)

Background/aims: To evaluate the non-invasive measurement of ocular rigidity (OR), an important biomechanical property of the eye, as a predictor of intraocular pressure (IOP) elevation after anti-vascular endothelial growth factor (anti-VEGF) intravitreal injection (IVI)., Methods: Subjects requiring IVI of anti-VEGF for a pre-existing retinal condition were enrolled in this prospective cross-sectional study. OR was assessed in 18 eyes of 18 participants by measurement of pulsatile choroidal volume change using video-rate optical coherence tomography, and pulsatile IOP change using dynamic contour tonometry. IOP was measured using Tono-Pen XL before and immediately following the injection and was correlated with OR., Results: The average increase in IOP following IVI was 19±9 mm Hg, with a range of 7-33 mm Hg. The Spearman correlation coefficient between OR and IOP elevation following IVI was 0.796 (p<0.001), showing higher IOP elevation in more rigid eyes. A regression line was also calculated to predict the IOP spike based on the OR coefficient, such that IOP spike=664.17 mm Hg·µL×OR + 4.59 mm Hg., Conclusion: This study shows a strong positive correlation between OR and acute IOP elevation following IVI. These findings indicate that the non-invasive measurement of OR could be an effective tool in identifying patients at risk of IOP spikes following IVI., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2021. No commercial re-use. See rights and permissions. Published by BMJ.)