Back to Search Start Over

Qualitative Research Methods for Large Language Models: Conducting Semi-Structured Interviews with ChatGPT and BARD on Computer Science Education

Authors :
Andreas Dengel
Rupert Gehrlein
David Fernes
Sebastian Görlich
Jonas Maurer
Hai Hoang Pham
Gabriel Großmann
Niklas Dietrich genannt Eisermann
Source :
Informatics, Vol 10, Iss 4, p 78 (2023)
Publication Year :
2023
Publisher :
MDPI AG, 2023.

Abstract

In the current era of artificial intelligence, large language models such as ChatGPT and BARD are being increasingly used for various applications, such as language translation, text generation, and human-like conversation. The fact that these models consist of large amounts of data, including many different opinions and perspectives, could introduce the possibility of a new qualitative research approach: Due to the probabilistic character of their answers, “interviewing” these large language models could give insights into public opinions in a way that otherwise only interviews with large groups of subjects could deliver. However, it is not yet clear if qualitative content analysis research methods can be applied to interviews with these models. Evaluating the applicability of qualitative research methods to interviews with large language models could foster our understanding of their abilities and limitations. In this paper, we examine the applicability of qualitative content analysis research methods to interviews with ChatGPT in English, ChatGPT in German, and BARD in English on the relevance of computer science in K-12 education, which was used as an exemplary topic. We found that the answers produced by these models strongly depended on the provided context, and the same model could produce heavily differing results for the same questions. From these results and the insights throughout the process, we formulated guidelines for conducting and analyzing interviews with large language models. Our findings suggest that qualitative content analysis research methods can indeed be applied to interviews with large language models, but with careful consideration of contextual factors that may affect the responses produced by these models. The guidelines we provide can aid researchers and practitioners in conducting more nuanced and insightful interviews with large language models. From an overall view of our results, we generally do not recommend using interviews with large language models for research purposes, due to their highly unpredictable results. However, we suggest using these models as exploration tools for gaining different perspectives on research topics and for testing interview guidelines before conducting real-world interviews.

Details

Language :
English
ISSN :
22279709
Volume :
10
Issue :
4
Database :
Directory of Open Access Journals
Journal :
Informatics
Publication Type :
Academic Journal
Accession number :
edsdoj.9e7efd2d790c461a850216652d4ccb54
Document Type :
article
Full Text :
https://doi.org/10.3390/informatics10040078