Search

As a method to shorten the test time of the Force Concept Inventory (FCI), we suggest the use of computerized adaptive testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. In so doing, the test length can be significantly shortened. As a step to develop a CAT-based version of the FCI (FCI-CAT), we examined the optimal test length of the FCI-CAT such that accuracy and precision [which we measure in terms of bias, standard error, and root-mean-square error (RMSE)] of Cohen’s d would be comparable to that of the full FCI for a given class size. First, we estimated the item parameters of the FCI items based on the three-parameter logistic model of item response theory, which are used in the algorithm of CAT. For this estimation, we used 2882 responses of Japanese university students. Second, we conducted a Monte Carlo simulation to analyze how the bias, standard error, and RMSE of Cohen’s d depend upon the test length. Third, we conducted a post hoc simulation to examine the consistency of the Monte Carlo results with what would have been obtained using empirical responses. For this comparison, we used 86 pairs of pre- and post- test responses of Japanese university students. As a result, we found that for a class size of 40, we may reduce the test length of the FCI-CAT to 15–19 items, thereby reducing the test time of the FCI to 50%–63%, with an accompanying decrease in accuracy and precision of only 5%–10%. The results of the Monte Carlo study and the post hoc simulation were consistent, which supports the adequacy of our Monte Carlo study and its relevance in terms of administering the FCI-CAT in real classrooms.

As a method to shorten the test time of the Force Concept Inventory (FCI), Computerized Adaptive Testing (CAT) is suggested. CAT is a test administered on a computer, where items (i.e. questions) are selected based on the responses of examinees to prior items. As a step of the development, we conducted analyses to find an optimal way for the administration of CAT with the FCI. Specifically, since CAT is based on Item Response Theory (IRT), we examined which IRT model is the most preferable. Using 2812 responses of the FCI of Japanese students, we estimated the item parameters of the One-, Two-, Three-, and Four-parameter logistic model of IRT and then we evaluated the statistics for the goodness of fit and for model selection. Based on the analysis, we suggest that the Two-parameter logistic model is the most preferable for the administration of CAT with the FCI.

We analyze the measurement error due to false positives of the Force Concept Inventory (FCI) focusing on four questions (Q.5, Q.6, Q.7, and Q.16). We determine whether or not a correct response to a given FCI question is a false positive using subquestions. Using the data of 1145 university students in Japan from 2015 to 2017, we find that the sum of the systematic error from the false positives of Q.5, Q.6, Q.7, and Q.16 is about 10% of the FCI score of a mid-level student. We consider what degree the error influences the measures of the effectiveness of a course, namely, the average normalized gain and Cohens’ d. Using a set of simulated data, we show that Cohens’ d is less sensitive to the systematic error due to false positives than the average normalized gain.