Comparison of generative AI performance on undergraduate and postgraduate written assessments in the biomedical sciences.

The value of generative AI tools in higher education has received considerable attention. Although there are many proponents of its value as a learning tool, many are concerned with the issues regarding academic integrity and its use by students to compose written assessments. This study evaluates and compares the output of three commonly used generative AI tools, ChatGPT, Bing and Bard. Each AI tool was prompted with an essay question from undergraduate (UG) level 4 (year 1), level 5 (year 2), level 6 (year 3) and postgraduate (PG) level 7 biomedical sciences courses. Anonymised AI generated output was then evaluated by four independent markers, according to specifed marking criteria and matched to the Frameworks for Higher Education Qualifcations (FHEQ) of UK level descriptors. Percentage scores and ordinal grades were given for each marking criteria across AI generated papers, inter-rater reliability was calculated using Kendall's coefcient of concordance and generative AI performance ranked. Across all UG and PG levels, ChatGPT performed better than Bing or Bard in areas of scientifc accuracy, scientifc detail and context. All AI tools performed consistently well at PG level compared to UG level, although only ChatGPT consistently met levels of high attainment at all UG levels. ChatGPT and Bing did not provide adequate references, while Bing falsifed references. In conclusion, generative AI tools are useful for providing scientifc information consistent with the academic standards required of students in written assignments. These fndings have broad implications for the design, implementation and grading of written assessments in higher education. [ABSTRACT FROM AUTHOR]