Search

Objectives To evaluate demographic biases in diagnostic accuracy and health advice between generative artificial intelligence (AI) (ChatGPT GPT-4) and traditional symptom checkers like WebMD. Materials and Methods Combination symptom and demographic vignettes were developed for 27 most common symptom complaints. Standardized prompts, written from a patient perspective, with varying demographic permutations of age, sex, and race/ethnicity were entered into ChatGPT (GPT-4) between July and August 2023. In total, 3 runs of 540 ChatGPT prompts were compared to the corresponding WebMD Symptom Checker output using a mixed-methods approach. In addition to diagnostic correctness, the associated text generated by ChatGPT was analyzed for readability (using Flesch-Kincaid Grade Level) and qualitative aspects like disclaimers and demographic tailoring. Results ChatGPT matched WebMD in 91% of diagnoses, with a 24% top diagnosis match rate. Diagnostic accuracy was not significantly different across demographic groups, including age, race/ethnicity, and sex. ChatGPT's urgent care recommendations and demographic tailoring were presented significantly more to 75-year-olds versus 25-year-olds (P  < .01) but were not statistically different among race/ethnicity and sex groups. The GPT text was suitable for college students, with no significant demographic variability. Discussion The use of non-health-tailored generative AI, like ChatGPT, for simple symptom-checking functions provides comparable diagnostic accuracy to commercially available symptom checkers and does not demonstrate significant demographic bias in this setting. The text accompanying differential diagnoses, however, suggests demographic tailoring that could potentially introduce bias. Conclusion These results highlight the need for continued rigorous evaluation of AI-driven medical platforms, focusing on demographic biases to ensure equitable care. [ABSTRACT FROM AUTHOR]

Background: Engaging researchers as research subjects is key to informing the development of effective and relevant research practices. It is important to understand how best to engage researchers as research subjects. Methods: A 24 factorial experiment, as part of a Multiphase Optimization Strategy, was performed to evaluate effects of four recruitment strategy components on participant opening of an emailed survey link and survey completion. Participants were members of three US-based national health research consortia. A stratified simple random sample was used to assign potential survey participants to one of 16 recruitment scenarios. Recruitment strategy components were intended to address both intrinsic and extrinsic sources of motivation, including: $50 gift, $1,000 raffle, altruistic messaging, and egoistic messaging. Multivariable generalized linear regression analyses adjusting for consortium estimated component effects on outcomes. Potential interactions among components were tested. Results are reported as adjusted odds ratios (aOR) with 95% confidence intervals (95% CI). Results: Surveys were collected from June to December 2023. A total of 418 participants were included from the consortia, with final analytical sample of 400 eligible participants. Out of the final sample, 82% (341) opened the survey link and 35% (147) completed the survey. Altruistic messaging increased the odds of opening the survey (aOR 2.02, 95% CI: 1.35–2.69, p = 0.033), while egoistic messaging significantly reduced the odds of opening the survey (aOR 0.56, 95%CI 0.38–0.75, p = 0.08). The receipt of egoistic messaging increased the odds of completing the survey once opened (aOR 1.81, 95%CI: 1.39–2.23, p < 0.05). There was a significant negative interaction effect between the altruistic appeal and egoistic messaging strategies for survey completion outcome. Monetary incentives did not a have a significant impact on survey completion. Conclusion: Intrinsic motivation is likely to be a greater driver of health researcher participation in survey research than extrinsic motivation. Altruistic and egoistic messaging may differentially impact initial interest and survey completion and when combined may lead to improved rates of recruitment, but not survey completion. Further research is needed to determine how to best optimize message content and whether the effects observed are modified by survey burden. [ABSTRACT FROM AUTHOR]