Search

Authoring of help content within educational technologies is labor intensive, requiring many iterations of content creation, refining, and proofreading. In this paper, we conduct an efficacy evaluation of ChatGPT-generated help using a 3 x 4 study design (N = 274) to compare the learning gains of ChatGPT to human tutor-authored help across four mathematics problem subject areas. Participants are randomly assigned to one of three hint conditions (control, human tutor, or ChatGPT) paired with one of four randomly assigned subject areas (Elementary Algebra, Intermediate Algebra, College Algebra, or Statistics). We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help. Notably, ChatGPT-generated help failed quality checks on 32% of problems. This was, however, reducible to nearly 0% for algebra problems and 13% for statistics problems after applying self-consistency, a "hallucination" mitigation technique for Large Language Models. [ABSTRACT FROM AUTHOR]

Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). They have a powerful discursive ability that can be used to explain decisions. In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges. [ABSTRACT FROM AUTHOR]

Copyright of TE & ET: Revista Iberoamericana de Tecnología en Educación y Educación en Tecnología is the property of School of Computer Science, La Pata University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

ChatGPT, a state-of-the-art large language model (LLM), is revolutionizing the AI field by exhibiting humanlike skills in a range of tasks that include understanding and answering natural language questions, translating languages, writing code, passing professional exams, and even composing poetry, among its other abilities. ChatGPT has gained an immense popularity since its launch, amassing 100 million active monthly users in just two months, thereby establishing itself as the fastest-growing consumer application to date. This paper discusses the reasons for its success as well as the future prospects of similar large language models (LLMs), with an emphasis on their potential impact on forecasting, a specialized and domain-specific field. This is achieved by first comparing the correctness of the answers of the standard ChatGPT and a custom one, trained using published papers from a subfield of forecasting where the answers to the questions asked are known, allowing us to determine their correctness compared to those of the two ChatGPT versions. Then, we also compare the responses of the two versions on how judgmental adjustments to the statistical/ML forecasts should be applied by firms to improve their accuracy. The paper concludes by considering the future of LLMs and their impact on all aspects of our life and work, as well as on the field of forecasting specifically. Finally, the conclusion section is generated by ChatGPT, which was provided with a condensed version of this paper and asked to write a four-paragraph conclusion.

Artificial intelligence (AI) has permeated all human activities, bringing about significant changes and creating new scientific and ethical challenges. The field of education could not be an exception to this development. OpenAI's unveiling of ChatGPT, their large language model (LLM), has sparked significant interest in the potential applications of this technology in education. This paper aims to contribute to the ongoing discussion on the role of AI in education and its potential implications for the future of learning by exploring how LLMs could be utilized in the teaching of mathematics in higher education and how they compare to the currently widely used computer algebra systems (CAS) and other mathematical tools. It argues that these innovative tools have the potential to provide functional and pedagogical opportunities that may influence changes in curriculum and assessment approaches. [ABSTRACT FROM AUTHOR]

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness inLLMgenerations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problemsolving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility. [ABSTRACT FROM AUTHOR]

The limitations of traditional education models and the advancement of technology have revealed the need to transform the learning experience. The "Flipped Learning" approach, born out of this need, is a model where students study learning materials in advance and participate in more interactive and hands-on activities in the classroom. This research discusses how the Flipped Learning model can be combined with ChatGPT. ChatGPT is a language model developed by OpenAI, and it can interact with students thanks to its natural language processing capability. In this way, combined with the Flipped Learning model, it can provide students with a more individualized and interactive learning experience. From this viewpoint, in this research, a model called "ChatGPT Aided Flipped Learning Model (CAFLM)" has been developed, which enables ChatGPT to be used in the "Flipped Learning" learning environment. CAFLM has three main phases: 1) Preparation, 2) Outside classroom teaching, and 3) inside classroom teaching. At these phases, the tasks of the student and the teacher are explained in detail. Sample activities and use cases are provided on how ChatGPT can be integrated into the Flipped Learning model and help students. To experience the model, activities developed for mathematics and science lessons were carried out using ChatGPT. The experiences have shown that the Flipped Learning model can be combined with Chatbots such as ChatGPT to provide students with a more effective and personalized learning experience. CAFLM can enable students to be more active in the learning process and contribute to innovative approaches in education. [ABSTRACT FROM AUTHOR]

The use of and research on the large language model (LLM) Generative Pretrained Transformer (GPT) is growing steadily, especially in mathematics education. As students and teachers worldwide increasingly use this AI model for teaching and learning mathematics, the question of the quality of the generated output becomes important. Consequently, this study evaluates AIsupported mathematical problem solving with different GPT versions when the LLM is subjected to prompt techniques. To assess the mathematics educational quality (content related and process related) of the LLM's output, we facilitated four prompt techniques and investigated their effects in model validations (N = 1,080) using three mathematical problem-based tasks. Subsequently, human raters scored the mathematics educational quality of AI output. The results showed that the content-related quality of AI-supported problem solving was not significantly affected by using various prompt techniques across GPT versions. However, certain prompt techniques, particular Chain-of-Thought and Ask-me-Anything, notably improved process-related quality. [ABSTRACT FROM AUTHOR]

This research explored Large Language Models potential uses on formative assessment for mathematical problem-solving process. The study provides a conceptual analysis of feedback and how the use of these models is related in the context of formative assessment for Linear Algebra problems. Particularly, the performance of a popular model known as ChatGPT in mathematical problems fails on reasoning, proofs, model construction, among others. Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve student's achievement of intended instructional outcomes. The study analyzed and evaluated feedback provided to engineering students in their solutions, from both, instructors and ChatGPT, against fine-grained criteria of a formative feedback model that includes affective aspects. Considering preliminary outputs, and to improve performance of feedback from both agents' instructors and ChatGPT, we developed a framework for formative assessment in mathematical problemsolving using a Large Language Model (LLM). We designed a framework to generate prompts, supported by common Linear Algebra mistakes within the context of concept development and problem-solving strategies. In this framework, the instructor acts as an agent to verify tasks in a math problem assigned to students, establishing a virtuous cycle of learning of queries supported by ChatGPT. Results revealed potentialities and challenges on how to improve feedback on graduate-level math problems, by which both educators and students adapt teaching and learning strategies. [ABSTRACT FROM AUTHOR]

The study explores the role of ChatGPT in the development of a micro-level framework for Educational Technology (EdTech). This study employed an inductive research design particularly a Constructivist Grounded Theory (CGT) approach from thirty-four private secondary high school teachers in the Philippines. To gather insights from specific participants, a semi-structured interview format was utilized. The instrument went through face validation from two experts well-versed in education research, instrumentation, and three-stage prompts. In addition, using JAMOVI software, Fleiss’ Kappa was used with a value of 0.715 indicating a substantial agreement among the expert validators. The interviews were designed to be open-ended, allowing for a flexible exploration of the research topic then subjected for data analysis with the aid of QDA Miner lite software. The findings revealed three (3) distinct milieux: pedagogical support, professional development, and ethical sense. In the (a) pedagogical support milieu, ChatGPT emerges as a virtual compass, offering real-time assistance, explanations, and guidance for teachers. It demonstrates the capability to answer questions, clarify concepts, and provide step-by-step support, crafted to individual students’ needs and learning styles; within the (b) redesigning milieu, it serves as a catalyst for adapting and recalibrating assessment and teaching strategies to suit learning purpose; and (c) ethical sense milieu highlights various implications, including data privacy, plagiarism, transparency, accountability, and overly reliant to the AI tool. Thus, this paper offers a micro-level framework anchored to the ChatGPT’s role in EdTech and aims to stimulate further scholarly discourse in the field.

In an age where artificial intelligence is reshaping the landscape of education and problem solving, our study unveils the secrets behind three digital wizards, ChatGPT-3.5, ChatGPT-4, and Google Bard, as they engage in a thrilling showdown of mathematical and logical prowess. We assess the ability of the chatbots to understand the given problem, employ appropriate algorithms or methods to solve it, and generate coherent responses with correct answers. We conducted our study using a set of 30 questions. These questions were carefully crafted to be clear, unambiguous, and fully described using plain text only. Each question has a unique and well-defined correct answer. The questions were divided into two sets of 15: Set A consists of "Original" problems that cannot be found online, while Set B includes "Published" problems that are readily available online, often with their solutions. Each question was presented to each chatbot three times in May 2023. We recorded and analyzed their responses, highlighting their strengths and weaknesses. Our findings indicate that chatbots can provide accurate solutions for straightforward arithmetic, algebraic expressions, and basic logic puzzles, although they may not be consistently accurate in every attempt. However, for more complex mathematical problems or advanced logic tasks, the chatbots' answers, although they appear convincing, may not be reliable. Furthermore, consistency is a concern as chatbots often provide conflicting answers when presented with the same question multiple times. To evaluate and compare the performance of the three chatbots, we conducted a quantitative analysis by scoring their final answers based on correctness. Our results show that ChatGPT-4 performs better than ChatGPT-3.5 in both sets of questions. Bard ranks third in the original questions of Set A, trailing behind the other two chatbots. However, Bard achieves the best performance, taking first place in the published questions of Set B. This is likely due to Bard's direct access to the internet, unlike the ChatGPT chatbots, which, due to their designs, do not have external communication capabilities. [ABSTRACT FROM AUTHOR]

Artificial Intelligence (AI) has been widely used in the social and legal fields, and ChatGPT, after AI painting, has once again set off a wave of discussion on whether AI and its generated works can obtain legal protection. Starting from the theoretical orientation of the origin of ChatGPT legal governance, this paper proposes the legal positioning and layered governance framework of ChatGPT application. It explores the role mechanism of ChatGPT empowering legal modernization and combs through the realistic dilemmas of ChatGPT-generated content data compliance legalization. To effectively analyze legal risks in the process of the ChatGPT application, data crawling technology and SMOTE oversampling technology are utilized to obtain ChatGPT application data and produce datasets. The Stacking integration strategy is introduced to combine the Random Forest in the Decision Tree Algorithm, GBDT algorithm, and Support Vector Machine to construct the legal risk prediction model of the ChatGPT application. For the effectiveness of the model, the ChatGPT application dataset is used to analyze the accuracy, ROC curve, and AUC value, which provides a reference for improving the legal system related to the ChatGPT application. The results show that the accuracy of the SVM classifier reaches 0.839, the correctness of the GBDT model is 0.947, and the AUC value of ChatGPT legal risk prediction based on the Stacking integration strategy is 0.947. Based on the inspiration of the decision tree algorithm, the improvement of the legal system related to the ChatGPT application should be improved in terms of generating content and allocating risk. Based on the insights of the decision tree algorithm, the improvement of the legal system related to the ChatGPT application should be made based on the dimensions of generation content and risk allocation.

This paper discusses the multi-scenario application of ChatGPT-based language modeling in English language teaching, and empirical experiments are conducted to support the research findings. The study includes constructing and analyzing English composition scoring and similarity detection models. The BERT-BiLSTM algorithm was utilized and compared to the Word2Vec-BiLSTM model. The BERT-BiLSTM-based English composition scoring model has a high correlation and consistency with the original scores, with an average correlation of 0.72 and a consistency of 82%. Conversely, the Word2Vec-BiLSTM model has a lesser correlation and consistency. We created a model and used different K values for the experiment to detect English composition similarity. The correctness, recall, and F1 measures were higher at a K value 200, with F1 values fluctuating between 89.35% and 95.14%. These support the high accuracy and efficiency of ChatGPT-based language modeling in English language teaching.

In this paper, we present a rigorous mathematical examination of generative pre-trained transformer (GPT) models and their autoregressive self-supervised learning mechanisms. We begin by defining natural language space and knowledge space, which are two key concepts for understanding the dimensionality reduction process in GPT-based large language models (LLMs). By exploring projection functions and their inverses, we establish a framework for analyzing the language generation capabilities of these models. We then investigate the GPT representation space, examining its implications for the models’ approximation properties. Finally, we discuss the limitations and challenges of GPT models and their learning mechanisms, considering trade-offs between complexity and generalization, as well as the implications of incomplete inverse projection functions. Our findings demonstrate that GPT models possess the capability to encode knowledge into low-dimensional vectors through their autoregressive self-supervised learning mechanism. This comprehensive analysis provides a solid mathematical foundation for future advancements in GPT-based LLMs, promising advancements in natural language processing tasks such as language translation, text summarization, and question answering due to improved understanding and optimization of model training and performance.

In this paper, we present a comprehensive mathematical analysis of the hallucination phenomenon in generative pretrained transformer (GPT) models. We rigorously define and measure hallucination and creativity using concepts from probability theory and information theory. By introducing a parametric family of GPT models, we characterize the trade-off between hallucination and creativity and identify an optimal balance that maximizes model performance across various tasks. Our work offers a novel mathematical framework for understanding the origins and implications of hallucination in GPT models and paves the way for future research and development in the field of large language models (LLMs).