Search

Problem solving is a signature skill of engineers. Here, problem solving is employed when students apply course concepts to reverse engineer YouTube videos and solve new student-written, homework-style problems (YouTube problems). Replacing textbook problems with YouTube problems, this research focuses on examining the rigor of YouTube problems as well as students' problem-solving skills on textbook and YouTube problems. A quasi-experimental, treatment/control group design was employed and data was collected and evaluated using multiple measurement instruments. First, rigor of homework problems was examined using the NASA Task Load Index. Also, problem solving was assessed using a previously-developed rubric called PROCESS: Problem definition, Representing the problem, Organizing the information, Calculations, Evaluating the solution, Solution communication, and Self-assessment. PROCESS was modified to independently measure completeness and accuracy of student responses, as well as identify errors committed in material and energy balances. In the treatment group, students were assigned ten textbook problems and nine YouTube problems. While the control group obtained higher PROCESS scores at the beginning of the study, both groups exhibited similar problemsolving skills near the end. Also, the rigor of student-written YouTube problems was similar to textbook problems related to the same course concepts. [ABSTRACT FROM AUTHOR]

This evidence-based practice paper discusses the preliminary validation of a project modified version of the Promoting Problem Solving Proficiency in First Year Engineering (PROCESS). The full rating plan required four raters to use the PROCESS to assess the problem-solving ability of ~70 engineering students randomly selected from two undergraduate cohorts at two Midwest universities. The many-facet Rasch measurement model has the psychometric properties to determine if there are any characteristics other than problem-solving that influence the scores assigned to students, such as rater bias or differential item functioning. Prior to implementing the full rating plan, the analysis examined how raters interacted with the six items on the modified PROCESS when scoring a random selection of 20 students' solutions to one textbook homework problem. Follow up inter-rater reliability meetings enabled rater discussion of rationale for discrepancies observed in the ratings. Differences in conceptions of the latent construct of problem-solving were resolved by recourse to the theoretical framework that informed the development of the PROCESS. This iterative process resulted in substantial increases in construct validity and measurement reliability when raters completed another round of assessment. Evidence indicated that raters increased their understanding of how rating scale categories related to levels of the latent construct. This paper describes the impacts and benefits this method of psychometric evaluation of rater-mediated assessments hold for the implementation of the full rating plan of student outcomes, as well as for the field of engineering education more broadly. [ABSTRACT FROM AUTHOR]