Search

Multiple choice writing guidelines are decidedly split on the use of "none of the above" (NOTA), with some authors discouraging and others advocating its use. Moreover, empirical studies of NOTA have produced mixed results. Generally, these studies have utilized NOTA as either the correct response or a distractor and assessed its effect on difficulty and discrimination. In these studies, NOTA commonly yields increased difficulty when it is used as the correct response, and no change in discrimination regardless of usage. However, when NOTA is implemented as a distractor, rarely is consideration given to the distractor that could have been written in its place. Here, we systematically replaced each distractor in a series of questions with NOTA across different versions of an Introductory Psychology examination. This approach allowed us to quantify the quality of each distractor based on its relative discrimination index and assess the effect of NOTA relative to the quality of distractor it replaced. Moreover, our use of large Introductory Psychology examinations afforded highly stable difficulty and discrimination estimates. We found that NOTA increased question difficulty only when it was the correct response, with no effect on difficulty of replacing any distractor type with NOTA. Moreover, we found that NOTA decreased discrimination when it replaced the most effective distractors, with no effect on discrimination of replacing either the correct response or lowest quality distractor with NOTA. These results replicate the common finding that inclusion of NOTA as the correct response increases question difficulty by equally luring high-performing and low-performing students toward distractors. Moreover, we have shown that including NOTA as a distractor can reduce discrimination if used in lieu of a well written alternative, suggesting that multiple choice authors should avoid using NOTA on multiple choice tests.

This study examined whether audio feedback provided to undergraduates (N=51) about one paper would prove beneficial in terms of improving their grades on another, unrelated paper of the same type. We examined this issue both in terms of student "beliefs" about learning transfer, as well as their "actual ability" to transfer what had been learned on one assignment to another, subsequent assignment. Results indicated that students believed that they would be able to transfer what they had learned via audio feedback. Moreover, results also suggested that students actually did generalize the overarching comments about content and structure made in the audio files to a subsequent paper, the content of which differed substantially from the initial one. Both students and teaching assistants demonstrated very favourable responses to this type of feedback, suggesting that it was both clear and comprehensive.

Because multiple-choice testing is so widespread in higher education, we assessed the quality of items used on classroom tests by carrying out a statistical item analysis. We examined undergraduates' responses to 1198 multiple-choice items on sixteen classroom tests in various disciplines. The mean item discrimination coefficient was +0.25, with more than 30% of items having unsatisfactory coefficients less than +0.20. Of the 3819 distractors, 45% were flawed either because less than 5% of examinees selected them or because their selection was positively rather than negatively correlated with test scores. In three tests, more than 40% of the items had an unsatisfactory discrimination coefficient, and in six tests, more than half of the distractors were flawed. Discriminatory power suffered dramatically when the selection of one or more distractors was positively correlated with test scores, but it was only minimally affected by the presence of distractors that were selected by less than 5% of examinees. Our findings indicate that there is considerable room for improvement in the quality of many multiple-choice tests. We suggest that instructors consider improving the quality of their multiple-choice tests by conducting an item analysis and by modifying distractors that impair the discriminatory power of items. (Contains 5 figures and 4 tables.)

Multiple-choice questions are widely used in higher education and have some important advantages over constructed-response test questions. It seems, however, that many teachers underestimate the value of multiple-choice questions, believing them to be useful only for assessing how well students can memorize information, but not for assessing higher-order cognitive skills. Several strategies are presented for generating multiple-choice questions that can effectively assess students' ability to understand, apply, analyze, and evaluate information.

The authors assessed the effects of using "none of the above" as an option in a 40-item, general-knowledge multiple-choice test administered to undergraduate students. Examinees who selected "none of the above" were given an incentive to write the correct answer to the question posed. Using "none of the above" as the keyed option made items much more difficult (d = -1.11). Furthermore, 45% of the time that examinees correctly selected "none of the above," they wrote either a wrong answer (19%) or no answer (26%), and rescoring items to deny credit in these cases caused item discrimination to fall (d = -0.35). Thus, when "none of the above" is the keyed option, credit earned by examinees with knowledge deficiencies can make items appear to have more discriminatory power than is actually the case. The authors recommend that "none of the above" should not be used as an option in multiple-choice items.

The Monty Hall dilemma (MHD) is a remarkably difficult probability problem with a counterintuitive solution. Undergraduate students used an interactive digital learning object that provided a set-based, animated explanation of the solution to the MHD and let them play games designed to increase understanding of the solution. More than 60% of users could explain the solution to the classic three-door version of the MHD and generalized the solution to an extremely difficult six-door version of the game. Because the MHD has relevance to important aspects of science in general and psychology in particular, some instructors may find the learning object, which is freely available at www.brocku.ca/psychology/people/dibattista.htm, to be of substantial value in their teaching. (Contains 2 figures.)

The authors examined how the grading scheme affects learning and students' reactions to the Immediate Feedback Assessment Technique (IFAT), an answer form providing immediate feedback on multiple-choice questions. Undergraduate students (N = 141) took a general-knowledge multiple-choice test of low, medium, or high difficulty. They used the IFAT with a number-correct (NC), partial-credit (PC), or correction-for-guessing (CG) grading scheme. After 1 week, they retook the test using a traditional response form with an NC scheme. When the authors used the NC or PC scheme for Test 1, the number of correct answers increased by more than 30% on Test 2. However, the increase was only about half as large with the CG scheme, suggesting that it interferes with the IFAT's learning benefits. Responses to questionnaire items provided no strong clues regarding the origin of this difference. Participants in all treatment conditions had positive attitudes toward the IFAT. (Contains 5 tables.)

The authors examined the relationship between the reactions of undergraduate students to using the Immediate Feedback Assessment Technique (IFAT), an answer form that provides immediate feedback on multiple-choice questions, for the first time on a major examination and their levels of test anxiety and trait anxiety. They also assessed whether students with higher levels of test anxiety and trait anxiety might be disadvantaged relative to other students by use of the IFAT in a testing situation. They found that preference of undergraduates (N =185) for the IFAT was not related to test anxiety, nor did evidence indicate that the IFAT put students with higher levels of test anxiety at a disadvantage with respect to test performance. Using the IFAT did not generally increase test-related anxiety, and for a majority of students, immediate feedback actually reduced it. Nineteen percent of students felt that immediate feedback interfered with their test performance but would nevertheless still prefer to use the IFAT in future tests. Potential concerns that test-anxious students may either dislike the IFAT or be disadvantaged by its use appear unwarranted and should not deter instructors from adopting the IFAT.

Undergraduates completed a questionnaire after using the Immediate Feedback Assessment Technique (IFAT), a commercially available answer form for multiple-choice (MC) testing that can be used easily and conveniently with large classes. This simple new technique for MC testing provides immediate feedback for each item in an answer-until-correct format and permits the earning of partial credit when the student's initial response is incorrect. Reaction to the IFAT was extremely positive, with students saying it was easy to use and contributed to their learning, and they indicated a strong desire to use the IFAT for all MC tests. Liking for the IFAT was not related to either personal characteristics or test performance variables, indicating that it has broad appeal to students.