Search

We developed assessment tasks aligned to the Next Generation Science Standards (NGSS) that require students to use argumentation and explanation practices along with disciplinary core ideas and crosscutting concepts to make sense of energy-related phenomena. Scoring rubrics were created to evaluate students' ability to make accurate claims, cite evidence, use relevant science ideas, and combine those elements to formulate well-reasoned arguments and explanations. We present an analysis of data to investigate the validity and reliability of our rubrics. Due to school closures caused by the COVID-19 pandemic, data were collected using Amazon Mechanical Turk (MTurk). The MTurk data were scored by two researchers to evaluate the inter-rater reliability. Data were then analyzed using Rasch modeling. Results show that rubric categories associated with stating claims, citing evidence, applying science ideas, and formulating coherent, well-reasoned arguments and explanations fit well to the Rasch model, and that rubric categories followed a hierarchy of difficulty. In this hierarchy, applying science ideas and formulating well-reasoned statements were more difficult than citing evidence, which were all more difficult than stating a claim. The ability to locate a student along this hierarchy allows for our tasks to be used to better understand a student's ability to write arguments and explanations of energy-related phenomena.

We have developed assessment tasks aligned to NGSS that require students to use practices along with disciplinary core ideas to make sense of energy-related phenomena. In this paper, we present an analysis of field test data and feedback from expert reviewers on the validity and reliability of a set of elementary school tasks. These tasks focused on assessing students' ability to write explanations or arguments about energy-related phenomena. Field test data were scored using rubrics based on the claim, evidence, reasoning (CER) framework. Using Rasch modeling, we evaluated the reliability of the task's rubric categories. We found that rubric categories fit well to the Rasch model. Categories were found to cluster in a hierarchy of difficulty in which reasoning and applying science idea categories were more difficult than evidence, which were more difficult than claim. The observed hierarchy in difficulty of CER categories is consistent with other studies and validates the tasks as measures of the CER framework. In addition, a panel of experts agreed that the tasks were aligned to the targeted NGSS practices and ideas. Overall, our results show that our procedure for task development resulted in valid and reliable NGSS-aligned assessment tasks.

This study is part of a larger effort to investigate linguistic features of assessment items that might explain differential performance of English learners on science assessments. An analysis of a large set of multiple-choice items suggested two features that held promise in reducing the performance gap: the presence of contrast words and the reduction of 1+ clauses. A subset of 48 items were modified and used in a study with 638 students to investigate the effect of these features. For half of those items, we reduced the number of 1+ clauses; for the other half, we removed contrast words. We analyzed the results from two subsets of students: 358 9th grade native-English and native-Spanish speaking students from across the U.S. and 138 8th grade students from one teacher at a school with 50% Spanish speaking students who agreed to participate in follow-up focus group discussions about the items. We compared student performance on the modified and original items and found little to no support for the selected linguistic features as affecting English learners' scores differentially, suggesting that performance was more strongly dependent on content knowledge than on the linguistic features of the items. The follow-up discussions also supported the idea that students' performance was strongly related to their content knowledge level.

This paper describes the development and validation of a set of three assessment instruments that can be used to assess students' progress on the energy concept (ASPECt) from fourth through twelfth grade. Rasch analysis techniques were used throughout the development process to guide the construction of an item bank and the selection of items for inclusion on the instruments. Rasch analysis techniques were also used to validate the instrument and design support materials to aid in the interpretation of student performance. A cross-sectional analysis was performed to determine the match between the instruments and students and revealed that currently the difficulties of intermediate and advanced instruments are greater than the average performance level of students in grades four through twelve. Wright maps and option probability curves were created to help users interpret student performance. [Paper presented at the AERA Annual Conference in New York, NY, April 13-17, 2018.]

This study tests a hypothesized learning progression for the concept of energy. It looks at 14 specific ideas under the categories of (i) Energy Forms and Transformations; (ii) Energy Transfer; (iii) Energy Dissipation and Degradation; and (iv) Energy Conservation. It then examines students' growth of understanding within each of these ideas at three levels of increasing conceptual complexity. The basic level of the model focuses on simple energy relationships and easily observable effects of energy processes; the intermediate level focuses on more complex energy concepts and applications; and the advanced level focuses on still more complex energy concepts, often requiring an atomic/molecular model to explain phenomena. The study includes results from 359 distractor-driven, multiple-choice test items administered to over 20,000 students in grades 4 through 12 from across the U.S. Rasch analysis provided linear measures of student performance and item difficulty on the same scale. Results largely supported a model of students' growth of understanding that progresses from an understanding of forms and transformations of energy to energy transfer to conservation while also progressing along a separate dimension of cognitive complexity. An analysis of the current state of students' understanding with respect to the knowledge identified in the learning progression showed that elementary level students perform well in comparison to expectations but that middle and high school students' performance does not meet expectations. [This paper was published in the "Journal of Research in Science Teaching" v55 n1 p68-93 Jan 2018 (EJ1163319).]

Focusing on the science knowledge and pedagogical content knowledge that teachers need to realize the vision of the "Next Generation Science Standards" (NGSS) in their classrooms, this study presents findings from efforts to (1) adapt existing criteria and design heuristics for educative features of curriculum materials to the needs of NGSS and (2) develop an authentic measure of teachers' knowledge and practice based on an analysis of teachers' evaluations of their students' written explanations of phenomena. The study demonstrates that existing criteria and heuristics for designing educative features of curriculum materials can be used productively, with minor modifications, to design features that support teachers in their use of materials that support NGSS. It also provides quantitative and qualitative data to show that teachers' analyses of the explanation task produced useful information on their understanding of the science ideas targeted in the assessment, of the misconceptions their students held, and of their students' ability to reason from evidence, science ideas, and models in explaining phenomena. This study builds on and contributes to a body of work on the design and use of educative curriculum materials and the evaluation of teacher knowledge. It suggests a practical approach to the design of NGSS-aligned curriculum materials that support both student and teacher learning based on findings from analysis and empirical studies. [This paper was published in "Journal of Science Teacher Education" (EJ1187839).]

Students often have trouble understanding key biology ideas, in part because they lack an understanding of foundational chemistry ideas. AAAS [American Association for the Advancement of Science] is collaborating with BSCS [Biological Sciences Curriculum Study] in the development of a curriculum unit that connects core chemistry and biology ideas in order to help eighth grade students build the conceptual foundation needed for high school biology. The unit is designed to engage students in (a) observing phenomena that are explicitly aligned to the targeted ideas and address common student misconceptions and difficulties and (b) using models to help interpret the phenomena in light of the targeted ideas. An initial draft of the unit was pilot tested at two schools in 2011. The results of the pilot test were used to revise the unit. In the spring of 2012, the revised unit and teacher materials were field tested with 677 eighth grade students from four states across the U.S. Pretests and posttests were used to measure the change in students' understanding of chemical reactions, conservation of mass, and biological growth. The data were analyzed using Rasch modeling and the racking and stacking methods. The stacking method showed that, overall, the students made statistically significant gains, suggesting that their understanding of the targeted ideas improved. The racking method showed that the difficulty of most of the items decreased as a result of the intervention, suggesting that the unit successfully covered most of the ideas. An analysis of distractor selections and written explanations of their answer choices showed that fewer students held misconceptions after participating in the unit. These results were used to inform a second round of revisions to the unit. (Contains 10 tables and 3 figures.)

AAAS (American Association for the Advancement of Science) is collaborating with BSCS (Biological Sciences Curriculum Study) in the development of a curriculum unit for eighth grade students that connects fundamental chemistry and biology concepts to better prepare them for high school biology. Recognizing that teachers play an influential role in delivering the curriculum to students, we developed teacher support materials and professional development designed to help teachers use the unit effectively. In order to learn about the promise of the teacher support materials, we developed an assessment targeting aspects of participating teachers' (n = 8) science content knowledge and pedagogical content knowledge (PCK) for the specific learning goals of the unit. Specifically, the assessment targeted three areas of teachers' knowledge: 1) content knowledge 2) knowledge of student thinking, and 3) knowledge of strategies to move student thinking forward, across four item contexts: 1) chemical reactions, 2) conservation of mass, 3) flow of matter in living systems, and 4) plant growth. Teachers took the assessment three times: before PD, after PD, and after teaching the unit. The assessment items were mainly constructed response and were scored using indicators of success and difficulty. Teachers made gains over time in most of the knowledge areas and across most of the contexts. Areas where they did not make clear progress, or where their knowledge was particularly low, indicated that either the assessment instrument or the teacher support materials could be improved. Revisions based on these findings are reported. (Contains 4 tables and 1 figure.)

Students often have trouble understanding key biology ideas because they lack an understanding of foundational chemistry ideas. AAAS Project 2061 is collaborating with BSCS in the development a curriculum unit that connects core chemistry and biochemistry ideas in order to help eighth grade students build the conceptual foundation needed for high school biology. The unit is designed to engage students in (a) observing phenomena that are explicitly aligned to the targeted ideas and selected to address common learning difficulties and (b) using models to help them interpret the phenomena in light of the targeted ideas. The unit was pilot tested with students from an urban school and a suburban school in the mid-Atlantic region of the U.S. at the end of or just after eighth grade. Multiple choice pretests and posttests were used to measure the change in students' understanding. The pre/posttest data were analyzed using Rasch modeling and the racking and stacking methods. The stacking method showed that the students at both schools made statistically significant gains in their performance on the items testing understanding of the targeted chemistry and biochemistry ideas. The racking method showed that the difficulty of most of the items decreased as a result of the intervention, suggesting that the unit successfully targeted most of the chemistry and biochemistry ideas. A distractor analysis showed that the students appeared to hold fewer misconceptions after participating in the unit. These results were used to inform revisions to the curriculum unit. (Contains 6 tables and 3 figures.)