Urology consultants versus large language models: Potentials and hazards for medical advice in urology

Abstract Background Current interest surrounding large language models (LLMs) will lead to an increase in their use for medical advice. Although LLMs offer huge potential, they also pose potential misinformation hazards. Objective This study evaluates three LLMs answering urology‐themed clinical case‐based questions by comparing the quality of answers to those provided by urology consultants. Methods Forty‐five case‐based questions were answered by consultants and LLMs (ChatGPT 3.5, ChatGPT 4, Bard). Answers were blindly rated using a six‐step Likert scale by four consultants in the categories: ‘medical adequacy’, ‘conciseness’, ‘coherence’ and ‘comprehensibility’. Possible misinformation hazards were identified; a modified Turing test was included, and the character count was matched. Results Higher ratings in every category were recorded for the consultants. LLMs' overall performance in language‐focused categories (coherence and comprehensibility) was relatively high. Medical adequacy was significantly poorer compared with the consultants. Possible misinformation hazards were identified in 2.8% to 18.9% of answers generated by LLMs compared with