Search

We aim to graphically analyze the depth of conceptual understanding behind the Force Concept Inventory (FCI) responses of students, focusing on three questions (questions 1, 15, and 28). In our study, we created and implemented subquestions to clarify and quantify the students' reasoning steps in reaching their responses to the original FCI questions. We used alluvial diagrams to visualize the responses of students on the subquestions and original FCI questions, representing the transition rates across the question options as flows. The flows represent common patterns of consistency and inconsistency in the students' conceptual understandings across the original FCI question and the subquestions. By combining subquestions and alluvial diagrams as we did, we are able to efficiently analyze the depth of understanding for a given situation and to visualize the existence of a specific misconception. For example, we found that even among those who correctly answered FCI question 15 about Newton's third law, many students incorrectly answered a corresponding subquestion asking about whether Newton's third law holds at the end of the interaction between the two objects. This misconception turned out to be fairly robust, as these same students had a similar answering pattern for question 28. Alluvial diagrams can be used for a broad range of applications, including analysis of multiple-tier tests, testlets (e.g., FCI questions 8 through 11), tests for representational consistency (e.g., a representational variant of the FCI), and so on.

Our study investigates whether confidence correlates with consistency in reasoning, specifically about radioactive decay. In prior work, we developed and tested a survey designed to measure consistency of student reasoning about radioactive decay by comparing responses to three prompts that are isomorphic, meaning that, despite having different surface features, they can all be answered appropriately with the understanding that radioactive decay occurs at random. In this paper, we compare (i) student patterns on these isomorphic prompts with (ii) confidence ratings that students provided together with their responses. Our research question is "to what extent does student confidence correlate with consistency in reasoning about radioactive decay?" We have found that there is no significant correlation, suggesting that more confident students are not more likely to be consistent. One reason why this finding is relevant is that the misconceptions model attributes consistency to student ideas (as opposed to the pieces model, which describes student ideas as potentially being context dependent). Our findings suggest that it is premature to describe a student idea as a misconception, even if the student is confident in that idea.

This paper presents improvements made to a computerized adaptive testing (CAT)-based version of the FCI (FCI-CAT) in regards to test security and test efficiency. First, we will discuss measures to enhance test security by controlling for item overexposure, decreasing the risk that respondents may (i) memorize the content of a pretest for use on the post-test or (ii) share information about the items with their classmates who take the assessment later. Second, we will discuss measures to enhance test efficiency, so that a shorter test length can yield a desired accuracy and precision of the measurement. Specifically, we utilized collateral information in the form of a pretest proficiency estimate of each respondent for selecting items and estimating respondent proficiency level in the post-test. To shorten the total testing time further, we also allowed the test lengths to be different for the pre- and post-test. To analyze how these improvements affect the accuracy and precision (which we measure in terms of root-mean-square error) of Cohen's d, we conducted a Monte Carlo simulation and a "post hoc" simulation. Then, we calculated the minimal test length of the FCI-CAT whose accuracy and precision are equivalent to that of the paper-and-pencil version of the FCI. Consequently, we obtained the following three findings: (i) By using collateral information, we can achieve the accuracy and precision of the full-length FCI with fewer items via the FCI-CAT. (ii) For a class size of 40, we can control for test security while still reducing the sum of the pre- and post-test lengths of the FCI-CAT to a total of 33 items (17 items on the pretest and 16 items on the post-test), thereby reducing the testing time to 55%. (iii) If one's goal is to maximize test efficiency, the pretest length should be slightly larger than the post-test length. On the other hand, if the goal is to maximize test security, the pretest length should be smaller and the post-test length should be larger. If one desires a balance of these two goals, it would be reasonable to choose equal pre- and post-test lengths.

As a method to shorten the test time of the Force Concept Inventory (FCI), we suggest the use of computerized adaptive testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. In so doing, the test length can be significantly shortened. As a step to develop a CAT-based version of the FCI (FCI-CAT), we examined the optimal test length of the FCI-CAT such that accuracy and precision [which we measure in terms of bias, standard error, and root-mean-square error (RMSE)] of Cohen's d would be comparable to that of the full FCI for a given class size. First, we estimated the item parameters of the FCI items based on the three-parameter logistic model of item response theory, which are used in the algorithm of CAT. For this estimation, we used 2882 responses of Japanese university students. Second, we conducted a Monte Carlo simulation to analyze how the bias, standard error, and RMSE of Cohen's "d" depend upon the test length. Third, we conducted a "post hoc" simulation to examine the consistency of the Monte Carlo results with what would have been obtained using empirical responses. For this comparison, we used 86 pairs of pre- and post- test responses of Japanese university students. As a result, we found that for a class size of 40, we may reduce the test length of the FCI-CAT to 15-19 items, thereby reducing the test time of the FCI to 50%-63%, with an accompanying decrease in accuracy and precision of only 5%-10%. The results of the Monte Carlo study and the "post hoc" simulation were consistent, which supports the adequacy of our Monte Carlo study and its relevance in terms of administering the FCI-CAT in real classrooms.

Professional problem-solving practice in physics and engineering relies on mathematical sense making--reasoning that leverages coherence between formal mathematics and conceptual understanding. A key question for physics education is how well current instructional approaches develop students' mathematical sense making. We introduce an assessment paradigm that operationalizes a typically unmeasured dimension of mathematical sense making: use of calculations on qualitative problems and use of conceptual arguments on quantitative problems. Three assessment items embodying this calculation-concept crossover assessment paradigm illustrate how mathematical sense making can positively benefit students' problem solving, leading to more efficient, insightful, and accurate solutions. These three assessment items were used to evaluate the efficacy of an instructional approach focused on developing students' mathematical sense making skills. In a quasi-experimental study, three parallel lecture sections of first-semester, introductory physics were compared: two mathematical sense making sections, one with an experienced instructor and one with a novice instructor, and a traditionally taught section, as a control group. Compared to the control group, mathematical sense making groups used calculation-concept crossover approaches more often and gave more correct answers on the crossover assessment items, but they did not give more correct answers to associated standard problems. In addition, although students' postcourse epistemological views on problem-solving coherence were associated with their crossover use, they did not fully account for the differences in crossover approach use between the mathematical sense making and control groups. These results demonstrate a new assessment paradigm for detecting a typically unmeasured dimension of mathematical sense making and provide evidence that a targeted instructional approach can enhance engagement with mathematical sense making in introductory physics.

In this study, we analyze the systematic error from false positives of the Force Concept Inventory (FCI). We compare the systematic errors of question 6 (Q.6), Q.7, and Q.16, for which clearly erroneous reasoning has been found, with Q.5, for which clearly erroneous reasoning has not been found. We determine whether or not a correct response to a given FCI question is a false positive using subquestions. In addition to the 30 original questions, subquestions were introduced for Q.5, Q.6, Q.7, and Q.16. This modified version of the FCI was administered to 1145 university students in Japan from 2015 to 2017. In this paper, we discuss our finding that the systematic errors of Q.6, Q.7, and Q.16 are much larger than that of Q.5 for students with mid-level FCI scores. Furthermore, we find that, averaged over the data sample, the sum of the false positives from Q.5, Q.6, Q.7, and Q.16 is about 10% of the FCI score of a midlevel student.

In this paper, we show data from the Colorado Learning Attitudes about Science Survey that suggests that Georgetown physics majors become increasingly expert in their attitudes towards physics learning and knowing after taking a course that combines two reformed curricula, Matter and Interactions (M&I) and Tutorials in Introductory Physics (TIPs). This occurs even though the two curricula do not send a consistent epistemological message to students. We analyze interview video data of two of these students to illustrate examples of this growth. We examine video data of one of these students in a tutorial session to describe a possible mechanism that may have contributed to the growth. Finally, we compare this qualitative video data with quantitative data from the newly developed Perceptions of Physics Classes survey and discuss aggregate responses to this survey in considering the ways in which other students developed more expert-like attitudes in this course. We conclude that the attitudinal growth observed cannot be explained simply "as the result of" either M&I or of TIPs but rather find the most plausible explanation to be that the growth is an emergent phenomena produced by M&I and TIPs working together in concert with other factors.

Much research in engineering and physics education has focused on improving students' problem-solving skills. This research has led to the development of step-by-step problem-solving strategies and grading rubrics to assess a student's expertise in solving problems using these strategies. These rubrics value "communication" between the student's qualitative description of the physical situation and the student's formal mathematical descriptions (usually equations) at two points: when initially setting up the equations, and when evaluating the final mathematical answer for meaning and plausibility. We argue that (i) neither the rubrics nor the associated problem-solving strategies explicitly value this kind of communication "during" mathematical manipulations of the chosen equations, and (ii) such communication is an aspect of problem-solving expertise. To make this argument, we present a case study of two students, Alex and Pat, solving the same kinematics problem in clinical interviews. We argue that Pat's solution, which connects manipulation of equations to their physical interpretation, is more expert-like than Alex's solution, which uses equations more algorithmically. We then show that the types of problem-solving rubrics currently available do not discriminate between these two types of solutions. We conclude that problem-solving rubrics should be revised or repurposed to more accurately assess problem-solving expertise. (Contains 2 tables, 1 figure and 4 footnotes.)