From hype to insight: Exploring ChatGPT's early footprint in education via altmetrics and bibliometrics.

Background: The emergence of ChatGPT in the education literature represents a transformative phase in educational technology research, marked by a surge in publications driven by initial research interest in new topics and media hype. While these publications highlight ChatGPT's potential in education, concerns arise regarding their quality, methodology, and uniqueness. Objective: Our study employs unconventional methods by combining altmetrics and bibliometrics to explore ChatGPT in education comprehensively. Methods: Two scholarly databases, Web of Science and Altmetric, were adopted to retrieve publications with citations and those mentioned on social media, respectively. We used a search query, "ChatGPT," and set the publication date between November 30th, 2022, and August 31st, 2023. Both datasets were within the education‐related domains. Through a filtering process, we identified three publication categories: 49 papers with both altmetrics and citations, 60 with altmetrics only, and 66 with citations only. Descriptive statistical analysis was conducted on all three lists of papers, further dividing the entire collection into three distinct periods. All the selected papers underwent detailed coding regarding open access, paper types, subject domains, and learner levels. Furthermore, we analysed the keywords occurring and visualized clusters of the co‐occurring keywords. Results and Conclusions: An intriguing finding is the significant correlation between media/social media mentions and academic citations in ChatGPT in education papers, underscoring the transformative potential of ChatGPT and the urgency of its incorporation into practice. Our keyword analysis also reveals distinctions between the themes of the papers that received both mentions and citations and those that received only citations but no mentions. Additionally, we noticed a limitation that authors' choice of keywords might be influenced by individual subjective judgements, potentially skewing results in thematic analysis based solely on author‐assigned keywords such as keyword co‐occurrence analysis. Henceforth, we advocate for developing a standardized keyword taxonomy in the educational technology field and integrating Large Language Models to enhance keyword analysis in altmetric and bibliometric tools. This study reveals that ChatGPT in education literature is evolving from rapid publication to rigorous research. Lay Description: What is currently known about this topic?: ChatGPT in education has seen a surge in publications driven by media hype.Early publications tend to lack rigour and reiterate known advantages/disadvantages.Literature reviews on ChatGPT in education have limitations in scope and depth.Some studies have explored altmetrics and bibliometrics in other fields. What does this paper add?: Combines altmetrics and bibliometrics to analyse publications of ChatGPT in education.Addresses challenges in the discourse by offering unconventional analysis methods.Identifies publication trends and investigates the relationship between media attention and citations.Determines key themes in the literature through keyword co‐occurrence analysis. Implications for practice/or policy: Expectations of continued growth in ChatGPT literature but with evolving publication trends.Distinctive characteristics of ChatGPT in education challenge keyword analysis.Proposes the development of a unified keyword taxonomy for clarity in the field.Suggests enhancing altmetrics and bibliometrics tools using Large Language Models. [ABSTRACT FROM AUTHOR]