Automatic Text Analysis of Reflective Essays to Quantify the Impact of the Modification of a Mechanical Engineering Course.

Students reflective essays in engineering education provide insight and context for instructional modification and assessment. However, the assessment of reflective essays numbering in thousands can be time-consuming This is notably important when trying to find specific changes in focus from one essay to another and measur ing how strong those changes are across multiple corpora of essays In this paper we describe and demonstrate an au tomated text analysis method for the at-scale, corpus normalized analysis of reflective essays. We apply it to quantitatively measure whether the modification of an undergraduate mechanical engineering course had the conjectured impact of a stronger emphasis on teamwork. Our analytical method is a "pipeline" composed of Text Mining (TM), Natural Language Processing (NLP), and Recurrence Quantification Analysis (ROA). We use this method to measure the presence of a specific thematic element in reflective essays to confirm the impact of the modification of a team-driven, model-based engineering design course. The original course and its modification were visualized using Sandovals conjecture mapping framework. The novel innovation of this approach is that the input (text from hundreds of reflective essays, sourced one at a time) when passed through this pipeline quickly produces a quantitative indication of the presence of thematic el ements and their recurence normalized across a corpus of hundreds of essays. A comparison of this quantitative indicator across separate corpora (each corpus of essays is for a different year) of reflective essays signaled a change in student focus toward the conjectured outcome We conclude that the TM-NLP-ROA pipeline can be applied for quick and at-scale extraction of the relative magnitude of thematic statements from reflective essays We observe that our conjectured redesign had the impact that we desired [ABSTRACT FROM AUTHOR]