Search

The emergence of large language models (LLM) with remarkable performance such as ChatGPT and GPT-4, has led to an unprecedented uptake in the population. One of their most promising and studied applications concerns education due to their ability to understand and generate human-like text, creating a multitude of opportunities for enhancing educational practices and outcomes. The objective of this study is twofold: to assess the accuracy of ChatGPT/GPT-4 in answering rheumatology questions from the access exam to specialized medical training in Spain (MIR), and to evaluate the medical reasoning followed by these LLM to answer those questions. A dataset, RheumaMIR, of 145 rheumatology-related questions, extracted from the exams held between 2010 and 2023, was created for that purpose, used as a prompt for the LLM, and was publicly distributed. Six rheumatologists with clinical and teaching experience evaluated the clinical reasoning of the chatbots using a 5-point Likert scale and their degree of agreement was analyzed. The association between variables that could influence the models' accuracy (i.e., year of the exam question, disease addressed, type of question and genre) was studied. ChatGPT demonstrated a high level of performance in both accuracy, 66.43%, and clinical reasoning, median (Q1–Q3), 4.5 (2.33–4.67). However, GPT-4 showed better performance with an accuracy score of 93.71% and a median clinical reasoning value of 4.67 (4.5–4.83). These findings suggest that LLM may serve as valuable tools in rheumatology education, aiding in exam preparation and supplementing traditional teaching methods. [ABSTRACT FROM AUTHOR]

The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model's overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. Material and methods: We studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4's new image analysis capability. Results: GPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (p < 0.001). English translations had a slightly enhanced performance. GPT-4 scored 26.1% of the questions with images in English. The results were worse when the questions were in Spanish, 13.0%, although the differences were not statistically significant (p = 0.250). Among medical specialties, GPT-4 achieved a 100% correct response rate in several areas, and the Pharmacology, Critical Care, and Infectious Diseases specialties showed lower performance. The error analysis revealed that while a 13.2% error rate existed, the gravest categories, such as "error requiring intervention to sustain life" and "error resulting in death", had a 0% rate. Conclusions: GPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model's high success rate is commendable, understanding the error severity is critical, especially when considering AI's potential role in real-world medical practice and its implications for patient safety. [ABSTRACT FROM AUTHOR]

The open launch of new artificial intelligence tools such as ChatGPT-3.5 (Generated Pre-trained Transformer) in November 2022 by the company OpenAI -and then its update to version GPT-4 in March 2023-poses new opportunities and challenges for journalism, and especially for professionals specifically focused on information verification. This research aims to understand and analyze the perceptions generated by the irruption of ChatGPT among fact-checking professionals in Spain with the aim of identifying disadvantages and advantages in its use, professional implications and desired functionalities. The study uses qualitative methodology with in-depth interviews with professionals from all Spanish fact-checking platforms belonging to the International Factchecking Network (IFCN) and the European Digital Media Observatory (EDMO). The results conclude that the use of ChatGPT presents notable ambivalences. On the one hand, there are perceived drawbacks in issues such as the transparency and reliability of sources, the scope of the data, and the format of the responses generated. However, fact-checkers also point to a possible auxiliary use of the chatbot in the tasks of gathering information, detecting falsehoods, and producing denials. The irruption of ChatGPT has a direct impact on the work routines of the fact-checkers, which can be made more difficult, reinforced or extended. Fact-checking professionals perceive themselves as "context agents" in a new ecosystem that also obliges them to further diversify their fields of action in the fight against disinformation and to accelerate the implementation of media education actions that empower citizens in the responsible use of artificial intelligence. [ABSTRACT FROM AUTHOR]

Communicative applications of artificial intelligence (AI) have burst into the lives of millions of people through products such as ChatGPT, Midjourney, or Replika, with the ability to generate texts and images and even understand natural language and socialize with humans. This study addresses attitudes toward communicative AI, as well as the sociodemographic and personality factors that are related to its positive perception and usage. For this purpose, we use data from a survey conducted at the end of February 2023 among adult residents in Spain (N = 821). Our results suggest that there is a slight predominance of negative attitudes toward communicative AI, which include cognitive and affective elements such as the perception of dehumanizing aspects and job-related threats, caution, or fear. However, a relevant portion of our respondents is not familiar with communicative AI or generalizes their answers toward other applications of AI. Individuals with higher educational level, greater openness to experience, and lower level of futurephobia show more positive attitudes toward these technologies. In terms of adoption, men, younger people, and those with more favorable attitudes toward AI are more likely to decide to use it, while those scoring higher in agreeableness tend to use it less frequently once they have decided to use it. We discuss these results under the premise that early adopters may play a central role in the attitudes of late majority and laggards, as well as in the future development of communicative AI and its regulatory framework. [ABSTRACT FROM AUTHOR]

Artificial Intelligence (AI) chatbots are able to explain complex concepts using plain language. The aim of this study was to assess the accuracy of three AI chatbots answering common questions related to contact lens (CL) wear. Three open access AI chatbots were compared: Perplexity, Open Assistant and ChatGPT 3.5. Ten general CL questions were asked to all AI chatbots on the same day in two different countries, with the questions asked in Spanish from Spain and in English from the U.K. Two independent optometrists with experience working in each country assessed the accuracy of the answers provided. Also, the AI chatbots' responses were assessed if their outputs showed any bias towards (or against) any eye care professional (ECP). The answers obtained by the same AI chatbots were different in Spain and the U.K. Also, statistically significant differences were found between the AI chatbots for accuracy. In the U.K., ChatGPT 3.5 was the most and Open Assistant least accurate (p < 0.01). In Spain, Perplexity and ChatGPT were statistically more accurate than Open Assistant (p < 0.01). All the AI chatbots presented bias, except ChatGPT 3.5 in Spain. AI chatbots do not always consider local CL legislation, and their accuracy seems to be dependent on the language used to interact with them. Hence, at this time, although some AI chatbots might be a good source of information for general CL related questions, they cannot replace an ECP. [ABSTRACT FROM AUTHOR]