Search

Educators seek to harness knowledge from educational corpora to improve student performance outcomes. Although prior studies have compared the efficacy of data mining methods (DMMs) in pipelines for forecasting student success, less work has focused on identifying a set of relevant features prior to model development and quantifying the stability of feature selection techniques. Pinpointing a subset of pertinent features can (1) reduce the number of variables that need to be managed by stakeholders, (2) make “black-box” algorithms more interpretable, and (3) provide greater guidance for faculty to implement targeted interventions. To that end, we introduce a methodology integrating feature selection with cross-validation and rank each feature on subsets of the training corpus. This modified pipeline was applied to forecast the performance of 3225 students in a baccalaureate science course using a set of 57 features, four DMMs, and four filter feature selection techniques. Correlation Attribute Evaluation (CAE) and Fisher’s Scoring Algorithm (FSA) achieved significantly higher Area Under the Curve (AUC) values for logistic regression (LR) and elastic net regression (GLMNET), compared to when this pipeline step was omitted. Relief Attribute Evaluation (RAE) was highly unstable and produced models with the poorest prediction performance. Borda’s method identified grade point average, number of credits taken, and performance on concept inventory assessments as the primary factors impacting predictions of student performance. We discuss the benefits of this approach when developing data pipelines for predictive modeling in undergraduate settings that are more interpretable and actionable for faculty and stakeholders. Supplementary Information The online version contains supplementary material available at 10.1186/s41239-021-00279-6.

High levels of attrition characterize undergraduate science courses in the USA. Predictive analytics research seeks to build models that identify at-risk students and suggest interventions that enhance student success. This study examines whether incorporating a novel assessment type (concept inventories [CI]) and using machine learning (ML) methods (1) improves prediction quality, (2) reduces the time point of successful prediction, and (3) suggests more actionable course-level interventions. A corpus of university and course-level assessment and non-assessment variables (53 variables in total) from 3225 students (over six semesters) was gathered. Five ML methods were employed (two individuals, three ensembles) at three time points (pre-course, week 3, week 6) to quantify predictive efficacy. Inclusion of course-specific CI data along with university-specific corpora significantly improved prediction performance. Ensemble ML methods, in particular the generalized linear model with elastic net (GLMNET), yielded significantly higher area under the curve (AUC) values compared with non-ensemble techniques. Logistic regression achieved the poorest prediction performance and consistently underperformed. Surprisingly, increasing corpus size (i.e., amount of historical data) did not meaningfully impact prediction success. We discuss the roles that novel assessment types and ML techniques may play in advancing predictive learning analytics and addressing attrition in undergraduate science education.

Machine learning (ML) has been increasingly employed in science assessment to facilitate automatic scoring efforts, although with varying degrees of success (i.e., magnitudes of machine-human score agreements [MHAs]). Little work has empirically examined the factors that impact MHA disparities in this growing field, thus constraining the improvement of machine scoring capacity and its wide applications in science education. We performed a meta-analysis of 110 studies of MHAs in order to identify the factors most strongly contributing to scoring success (i.e., high Cohen's kappa [κ]). We empirically examined six factors proposed as contributors to MHA magnitudes: algorithm, subject domain, assessment format, construct, school level, and machine supervision type. Our analyses of 110 MHAs revealed substantial heterogeneity in $$\kappa{ (mean =} \, \text{.64; range = .09-.97}$$ , taking weights into consideration). Using three-level random-effects modeling, MHA score heterogeneity was explained by the variability both within publications (i.e., the assessment task level: 82.6%) and between publications (i.e., the individual study level: 16.7%). Our results also suggest that all six factors have significant moderator effects on scoring success magnitudes. Among these, algorithm and subject domain had significantly larger effects than the other factors, suggesting that technical features and assessment external features might be primary targets for improving MHAs and ML-based science assessments.

Belief in genetic determinism (BGD) has been associated with increased social stereotyping and prejudicial views and consequently is of significant concern to science educators. The Public Understanding and Attitudes towards Genetics and Genomics (PUGGS) instrument was developed to measure relationships among BGD, genetics knowledge, and demographic variables. PUGGS validity evidence has relied primarily on Classical Test Theory frameworks and Brazilian samples. Using a more advanced psychometric framework (Rasch analysis) and a large North American undergraduate sample (n > 800), we further evaluate validity claims by studying (1) dimensionality and function of PUGGS item sets; (2) magnitudes of item endorsement across human traits (social, biological) and taxonomic (animal, plant) categories; and (3) degree of trait-level genetic overattribution. Similar to Gericke et al. (Sci Educ 26:1223–1259, 2017), we identified a two-dimensional structure for the BGD scale (i.e., social, biological) and the genetics knowledge scale (i.e., gene-environment interactions [GEI], genetics and genomics knowledge [GGK]). However, there were several problems with the functioning of the item sets (e.g., low reliability for GEI, problematic rating scale for BGD biological). We report that the magnitudes of GEI and GGK did not differ by taxonomic context. Finally, genetic over- (and under-) attribution was identified for both biological and social traits, indicating that students harbored considerably diverse and frequently non-normative conceptions about genetic contributions to traits. Importantly, psychometric and theoretical concerns reported here raise questions about the operationalization of the PUGGS BGD construct. Recommendations for PUGGS revisions and educational implications are discussed.

Although recent studies have used the Classroom Observation Protocol for Undergraduate STEM (COPUS) to make claims about faculty reform, important questions remain: How should COPUS measures be situated within existing reform frameworks? Is there a universal sampling intensity that allows for valid inferences about the frequency of student-centered instruction within a semester or across semesters of a course? These questions were addressed using longitudinal COPUS observations (128 classes, three faculty, 4 years). COPUS behaviors were used to categorize classes into didactic, interactive lecture, or student-centered instructional styles. Sampling intensities (one to 11 classes) were simulated (1000 times) within a course and across semesters. The sampling intensities required for generating valid inferences about 1) the presence of student-centered instruction and 2) the proportion of instructional styles in a course and through time were calculated. Results indicated that the sampling intensity needed to characterize courses and instructors varied and was much higher than previously recommended for instructors with: 1) rare instances of student-centered classes, 2) variability in instructional style, and 3) longitudinal changes in instructional patterns. These conditions are common in early reform contexts. This study highlights the risks of broad, decontextualized sampling protocol recommendations and illustrates how reform frameworks, sampling intensities, and COPUS measures interact to impact inferences about faculty change.

The goal of our study was to examine a large (>400), cross-sectional sample of Chinese pre-service biology teachers (PBTs) in order to document their evolution acceptance levels, evolution knowledge, and evolutionary reasoning patterns. This approach was taken in order to better understand the degree to which particular evolutionary reasoning difficulties exist independent of religious worldviews. The sample included (1) 160 PBTs tasked with completing four items from the ACORNS instrument, (2) 320 PBTs who completed the CINS and MATE instruments, and (3) 32 teachers who completed semi-structured clinical interviews using four ACORNS items. Findings from these samples revealed that Chinese PBTs’ knowledge (CINS) and acceptance (MATE) were equivalent with teachers’ scores from other countries, whereas performance on explanation tasks was lower. Scores from the CINS, MATE, and ACORNS did not reveal any significant improvements through the four-year teacher education programme. Although a large body ...

BackgroundAlthough personal, familial, and community conflict with evolution have been documented in the literature, these scales require conceptualization as a construct and operationalization as a measure. The Scales of Conflict with Evolution Measure (SECM) instrument was developed in response to these needs. Using a construct validity framework, the content, internal structure, convergent, and substantive validity of the SECM were evaluated using Rasch analysis, Structural Equation Modeling (SEM), and follow up questioning. The conceptual utility of the instrument was explored by examining whether it added explanatory insights into evolution acceptance above and beyond religiosity, evolution knowledge, and background variables.ResultsA literature review and expert consultation indicated that construct of evolutionary conflict perception should (i) encompass the hierarchical nature of human social structures (personal, family, community) and (ii) probe conflict as it relates to human values, cultures, and beliefs. A three-dimensional construct was operationalized as a nine-item rating scale measure. Using Rasch analyses of SECM responses from a diverse sample of > 1000 students studying evolution, the instrument met criteria of robust measurement, including: fit to model expectations; three-dimensional structure; high reliability; good rating scale function; measurement invariance with time; and convergence with a similar construct. SEM showed that: (i) family and community conflict had unique causal contributions to personal conflict, with family showing a stronger and modest impact, and (ii) personal conflict had a significant and modest causal impact on evolution acceptance above and beyond the contributions of religiosity, evolution knowledge, and background variables.ConclusionThe SECM is an easy-to-administer instrument to measure conflict with evolution and is supported by several forms of validity evidence. The SECM has potential for facilitating measurement of evolutionary conflict in educational settings, thereby raising instructor awareness of conflict levels in students, promoting rigorous evaluations of educational interventions designed to reduce conflict, and fostering conceptual advances in the field of evolution education. Future work is needed to gather additional forms of validity evidence and to test current validity claims in additional participant samples. SECM measures should also be incorporated into more complex SEM models that treat evolution knowledge and religiosity as part of the structural paths to evolution acceptance. Such models could provide insights into the most worthwhile targets for the development of educational interventions to mitigate conflict at multiple scales.

College science instructors need continuous professional development (PD) to meet the call to evidence-based practice. New PD efforts need to focus on the nuanced blend of factors that influence instructors’ teaching practices. We used persona methodology to describe the diversity among instructors who were participating in a long-term PD initiative. Persona methodology originates from ethnography. It takes data from product users and compiles those data in the form of fictional characters. Personas facilitate user-centered design. We identified four personas among our participants: Emma the Expert views herself as the subject-matter expert in the classroom and values her hard-earned excellence in lecturing. Ray the Relater relates to students and focuses on their points of view about innovative pedagogies. Carmen the Coach coaches her students by setting goals for them and helping them develop skill in scientific practices. Beth the Burdened owns the responsibility for her students’ learning and feels overwhelmed that students still struggle despite her use of evidence-based practice. Each persona needs unique PD. We suggest ways that PD facilitators can use our personas as a reflection tool to determine how to approach the learners in their PD. We also suggest further avenues of research on learner-centered PD.

Helping faculty develop high-quality instruction that positively affects student learning can be complicated by time limitations, a lack of resources, and inexperience using student data to make iterative improvements. We describe a community of 16 faculty from five institutions who overcame these challenges and collaboratively designed, taught, iteratively revised, and published an instructional unit about the potential effect of mutations on DNA replication, transcription, and translation. The unit was taught to more than 2000 students in 18 courses, and student performance improved from preassessment to postassessment in every classroom. This increase occurred even though faculty varied in their instructional practices when they were teaching identical materials. We present information on how this faculty group was organized and facilitated, how members used student data to positively affect learning, and how they increased their use of active-learning instructional practices in the classroom as a result of participation. We also interviewed faculty to learn more about the most useful components of the process. We suggest that this professional development model can be used for geographically separated faculty who are interested in working together on a known conceptual difficulty to improve student learning and explore active-learning instructional practices.

Indonesia has received little attention in the evolution education research community despite being the world’s largest Muslim-majority nation and the third most populous democracy. As such, Indonesia has the potential to test generalizations and shed new light on the ways in which religion, culture, and formal education contribute to evolutionary understanding, reasoning, and acceptance levels. Here, we report on empirical studies of moderately large samples (n > 300) of Indonesian pre-service biology teachers’ understanding, reasoning, and acceptance of Evolution. In the first and second study, we compare American and Indonesian student’s evolutionary reasoning patterns across a range of tasks using written prompts and clinical interviews. In the third study we investigate Indonesian pre-service biology teachers’ acceptance of evolution. Our first and second studies found that Indonesian participants commonly displayed: lower levels of understanding compared to American samples, mixtures of naive and normative concepts in evolutionary explanations, weak cognitive coherence across tasks, and teleological reasoning in explanations of evolutionary change. In the third study, we found that Indonesian participants, like those in other cultures, have greater acceptance of microevolution, followed by lower acceptance of macroevolution and human evolution. Taken together, our studies suggest that cognitive difficulties inherent to thinking about evolution, to a greater extent than cultural and religious influences, are shaping evolutionary reasoning patterns and acceptance levels in Indonesian pre-service biology teachers. This finding is notable given the strong religious nature of Indonesian society and the prominent role of religion in Indonesian formal education.

Increasing the retention of STEM (science, technology, engineering, and mathematics) majors has recently emerged as a national priority in undergraduate education. Since poor performance in large introductory science and math courses is one significant factor in STEM dropout, early detection of struggling students is needed. Technology-supported “early warning systems” (EWSs) are being developed to meet these needs. Our study explores the utility of two commonly collected data sources—pre-course concept inventory scores and longitudinal clicker scores—for use in EWS, specifically, in determining the time points at which robust predictions of student success can first be established. The pre-course diagnostic assessments, administered to 287 students, included two concept inventories and one attitude assessment. Clicker question scores were also obtained for each of the 37 class sessions. Additionally, student characteristics (sex, ethnicity, and English facility) were gathered in a survey. Our analyses revealed that all variables were predictive of final grades. The correlation of the first 3 weeks of clicker scores with final grades was 0.53, suggesting that this set of variables could be used in an EWS starting at the third week. We also used group-based trajectory models to assess whether trajectory patterns were homogeneous in the class. The trajectory analysis identified three distinct clicker performance patterns that were also significant predictors of final grade. Trajectory analyses of clicker scores, student characteristics, and pre-course diagnostic assessment appear to be valuable data sources for EWS, although further studies in a diversity of instructional contexts are warranted.

A large body of work has been devoted to reducing assessment biases that distort inferences about students’ science understanding, particularly in multiple-choice instruments (MCI). Constructed-response instruments (CRI), however, have invited much less scrutiny, perhaps because of their reputation for avoiding many of the documented biases of MCIs. In this study we explored whether known biases of MCIs—specifically item sequencing and surface feature effects—were also apparent in a CRI designed to assess students’ understanding of evolutionary change using written explanation (Assessment of COntextual Reasoning about Natural Selection [ACORNS]). We used three versions of the ACORNS CRI to investigate different aspects of assessment structure and their corresponding effect on inferences about student understanding. Our results identified several sources of (and solutions to) assessment bias in this practice-focused CRI. First, along the instrument item sequence, items with similar surface features produced greater sequencing effects than sequences of items with dissimilar surface features. Second, a counterbalanced design (i.e., Latin Square) mitigated this bias at the population level of analysis. Third, ACORNS response scores were highly correlated with student verbosity, despite verbosity being an intrinsically trivial aspect of explanation quality. Our results suggest that as assessments in science education shift toward the measurement of scientific practices (e.g., explanation), it is critical that biases inherent in these types of assessments be investigated empirically.

An analysis of assessment quality in genomics and bioinformatics education literature found that a minority (, The growing importance of genomics and bioinformatics methods and paradigms in biology has been accompanied by an explosion of new curricula and pedagogies. An important question to ask about these educational innovations is whether they are having a meaningful impact on students’ knowledge, attitudes, or skills. Although assessments are necessary tools for answering this question, their outputs are dependent on their quality. Our study 1) reviews the central importance of reliability and construct validity evidence in the development and evaluation of science assessments and 2) examines the extent to which published assessments in genomics and bioinformatics education (GBE) have been developed using such evidence. We identified 95 GBE articles (out of 226) that contained claims of knowledge increases, affective changes, or skill acquisition. We found that 1) the purpose of most of these studies was to assess summative learning gains associated with curricular change at the undergraduate level, and 2) a minority (

Although historical changes in scientific ideas sometimes display striking similarities with students’ conceptual progressions, some scholars have cautioned that such similarities lack meaningful commonalities. In the history of evolution, while Darwin and his contemporaries often used natural selection to explain evolutionary trait gain or increase, they struggled to use it to convincingly account for cases of trait loss or decrease. This study examines Darwin’s evolutionary writings about trait gain and loss in the Origin of Species (On the origin of species by means of natural selection, or the preservation of favoured races in the struggle for life. D. Appleton, New York, 1859) and compares them to written evolutionary explanations for trait gain and loss in a large (n > 500), cross-cultural and cross-sectional sample (novices and experts from the USA and Korea). Findings indicate that significantly more students and experts applied natural selection to cases of trait gain, but like Darwin and his contemporaries, they more often applied ‘use and disuse’ and ‘inheritance of acquired characteristics’ to episodes of trait loss. Although the parallelism between Darwin’s difficulties and students’ struggles with trait loss are striking, significant differences also characterize explanatory model structure. Overall, however, students and scientists struggles to explain trait loss—which is a very common phenomenon in the history of life—appear to transcend time, place, and level of biological expertise. The significance of these findings for evolution education are discussed; in particular, the situated nature of biological reasoning, and the important role that the history of science can play in understanding cognitive constraints on science learning.

The landscape of science education is being transformed by the new Framework for Science Education (National Research Council, A framework for K-12 science education: practices, crosscutting concepts, and core ideas. The National Academies Press, Washington, DC, 2012), which emphasizes the centrality of scientific practices— such as explanation, argumentation, and communication—in science teaching, learning, and assessment. A major chal- lenge facing the field of science education is developing assessment tools that are capable of validly and efficiently evaluating these practices. Our study examined the efficacy of a free, open-source machine-learning tool for evaluating the quality of students' written explanations of the causes of evolutionary change relative to three other approaches: (1) human-scored written explanations, (2) a multiple- choice test, and (3) clinical oral interviews. A large sample of undergraduates (n = 104) exposed to varying amounts of evolution content completed all three assessments: a clinical oral interview, a written open-response assessment, and a multiple-choice test. Rasch analysis was used to compute linear person measures and linear item measures on a single logit scale. We found that the multiple-choice test displayed poor person and item fit (mean square outfit (1.3), while both oral interview measures and computer- generated written response measures exhibited acceptable fit (average mean square outfit for interview: person 0.97, item 0.97; computer: person 1.03, item 1.06). Multiple- choice test measures were more weakly associated with interview measures (r = 0.35) than the computer-scored explanation measures (r = 0.63). Overall, Rasch analysis indicated that computer-scored written explanation mea- sures (1) have the strongest correspondence to oral inter- view measures; (2) are capable of capturing students' normative scientific and naive ideas as accurately as human-scored explanations, and (3) more validly detect understanding than the multiple-choice assessment. These findings demonstrate the great potential of machine-learn- ing tools for assessing key scientific practices highlighted in the new Framework for Science Education.

Using item-response theory, the authors evaluated differences in performance by gender on a constructed-response assessment about natural selection. The results identify relationships between item features and performance by gender and highlight the importance of examining gender effects on performance in written assessment tasks in biology., Understanding sources of performance bias in science assessment provides important insights into whether science curricula and/or assessments are valid representations of student abilities. Research investigating assessment bias due to factors such as instrument structure, participant characteristics, and item types are well documented across a variety of disciplines. However, the relationships among these factors are unclear for tasks evaluating understanding through performance on scientific practices, such as explanation. Using item-response theory (Rasch analysis), we evaluated differences in performance by gender on a constructed-response (CR) assessment about natural selection (ACORNS). Three isomorphic item strands of the instrument were administered to a sample of undergraduate biology majors and nonmajors (Group 1: n = 662 [female = 51.6%]; G2: n = 184 [female = 55.9%]; G3: n = 642 [female = 55.1%]). Overall, our results identify relationships between item features and performance by gender; however, the effect is small in the majority of cases, suggesting that males and females tend to incorporate similar concepts into their CR explanations. These results highlight the importance of examining gender effects on performance in written assessment tasks in biology.

EvoGrader is a free, online, on-demand formative assessment service designed for use in undergraduate biology classrooms. EvoGrader’s web portal is powered by Amazon’s Elastic Cloud and run with LightSIDE Lab’s open-source machine-learning tools. The EvoGrader web portal allows biology instructors to upload a response file (.csv) containing unlimited numbers of evolutionary explanations written in response to 86 different ACORNS (Assessing COntextual Reasoning about Natural Selection) instrument items. The system automatically analyzes the responses and provides detailed information about the scientific and naive concepts contained within each student’s response, as well as overall student (and sample) reasoning model types. Graphs and visual models provided by EvoGrader summarize class-level responses; downloadable files of raw scores (in .csv format) are also provided for more detailed analyses. Although the computational machinery that EvoGrader employs is complex, using the system is easy. Users only need to know how to use spreadsheets to organize student responses, upload files to the web, and use a web browser. A series of experiments using new samples of 2,200 written evolutionary explanations demonstrate that EvoGrader scores are comparable to those of trained human raters, although EvoGrader scoring takes 99% less time and is free. EvoGrader will be of interest to biology instructors teaching large classes who seek to emphasize scientific practices such as generating scientific explanations, and to teach crosscutting ideas such as evolution and natural selection. The software architecture of EvoGrader is described as it may serve as a template for developing machine-learning portals for other core concepts within biology and across other disciplines.

To improve assessments of academic achievement, test developers have been urged to use an “assessment triangle” that starts with research-based models of cognition and learning [NRC (2001) Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press]. This approach has been successful in designing high-quality reading and math assessments, but less progress has been made for assessments in content-rich sciences such as biology. To rectify this situation, we applied the “assessment triangle” to design and evaluate new items for an instrument (ACORNS, Assessing Contextual Reasoning about Natural Selection) that had been proposed to assess students' use of natural selection to explain evolutionary change. Design and scoring of items was explicitly guided by a cognitive model that reflected four psychological principles: with development of expertise, (1) core concepts facilitate long-term recall, (2) causally-central features become weighted more strongly in explaining phenomena, (3) normative ideas co-exist but increasingly outcompete naive ideas in reasoning, and (4) knowledge becomes more abstract and less specific to the learning situation. We conducted an evaluation study with 320 students to examine whether scores from our new ACORNS items could detect gradations of expertise, provide insight into thinking about evolutionary change, and predict teachers' assessments of student achievement. Findings were consistent with our cognitive model, and ACORNS was revealing about undergraduates' thinking about evolutionary change. Results indicated that (1) causally-central concepts of evolution by natural selection typically co-existed and competed with the presence of naive ideas in all students' explanations, with naive ideas being especially prevalent in low-performers' explanations; (2) causally-central concepts were elicited most frequently when students were asked to explain evolution of animals and familiar plants, with influence of superficial features being strongest for low-performers; and (3) ACORNS scores accurately predicted students' later achievement in a college-level evolution course. Together, findings illustrate usefulness of cognitive models in designing instruments intended to capture students' developing expertise. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 744–777, 2012

Our study investigates the challenges introduced by students’ use of lexically ambiguous language in evolutionary explanations. Specifically, we examined students’ meaning of five key terms incorporated into their written evolutionary explanations: pressure, select, adapt, need, and must. We utilized a new technological tool known as the Assessment Cascade System (ACS) to investigate the frequency with which biology majors spontaneously used lexically ambiguous language in evolutionary explanations, as well as their definitions and explanations of what they meant when they used such terms. Three categories of language were identified and examined in this study: terms with Dual Ambiguity, Incompatible Ambiguity, and Unintended Ambiguity. In the sample of 1282 initial evolutionary explanations, 81 % of students spontaneously incorporated lexically ambiguous language at least once. Furthermore, the majority of these initial responses were judged to be inaccurate from a scientific point of view. While not significantly related to gender, age, or reading/writing ability, students’ use of contextually appropriate evolutionary language (pressure and adapt) was significantly associated with academic performance in biology. Comparisons of initial responses to follow-up responses demonstrated that the majority of student explanations were not reinterpreted after consideration of the follow-up response; nevertheless, a sizeable minority was interpreted differently. Most cases of interpretation change were a consequence of resolving initially ambiguous responses, rather than a change of accuracy, resulting in an increased understanding of students’ evolutionary explanations. We discuss a series of implications of lexical ambiguity for evolution education.

Studies of students' thinking about natural selection have revealed that the scenarios in which students reason evoke different types, magnitudes, and arrangements of knowledge elements and misconceptions. Diagnostic tests are needed that probe students' thinking across a representative array of evolutionary contexts. The ACORNS is a diagnostic test that treats different evolutionary contexts as unique scenarios worthy of focused assessment and targeted instruction. Our investigations revealed that ACORNS scores produce valid and reliable inferences about students' thinking about natural selection. We urge biology teachers at all educational levels to begin assessing and attending to their students' reasoning across a broader array of evolutionary contexts, as competency in one context is often not indicative of competency in another.

Our study explored the prospects and limitations of using machine-learning software to score introductory biology students’ written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors’ and nonmajors’ written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of “near-perfect” agreement (kappa > 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations.

Considerable research has focused on differences in expert and novice problem representation and performance within physics, chemistry, and genetics. Here, we examine whether models of problem solving based on this work are useful within the domain of evolutionary biology. We utilized card sort tasks, interviews, and paper-and-pencil tests to: (1) delineate problem categorization rules, (2) quantify problem solving success, and (3) measure the relationships between the composition, structure, and coherence of problem solutions. We found that experts and novices perceived different item features to be of significance in card sort tasks, and that sensitivity to item surface features was adversely associated with problem solving success. As in other science domains, evolutionary problem representation and problem solving performance were tightly coupled. Explanatory coherence and the absence of cognitive biases were distinguishing features of evolutionary expertise. We discuss the implications of these findings for biology teaching and learning.

We propose a new model of the factors influencing acceptance of evolutionary theory that highlights a novel variable unexplored in previous studies: the feeling of certainty (FOC). The model is grounded in an emerging understanding of brain function that acknowledges the contributions of intuitive cognitions in making decisions, such as whether or not to accept a particular theoretical explanation of events. Specifically, we examine the relationships among religious identity, level of edu- cation, level of knowledge, FOC, and level of evolutionary acceptance to test whether our proposed model accurately predicts hypothesized pathways. We employ widely used measures—the CINS, MATE, and ORI—in addition to new variables in multiple regression and path analyses in order to test the interrelationships among FOC and acceptance of evolutionary theory. We explore these relationships using a sample of 124 pre-service biology teachers found to display comparable knowledge and belief levels as reported in previous studies on this topic. All of our hypothesis tests corroborated the idea that FOC plays a moderating role in relationships among evolutionary knowledge and beliefs. Educational research into acceptance of evolutionary theory will likely benefit from increased attention to non-con- scious intuitive cognitions that give rise to feeling of knowing or certainty. 2011 Wiley Periodicals

Concept inventories, consisting of multiple-choice questions designed around common student misconceptions, are designed to reveal student thinking. However, students often have complex, heterogeneous ideas about scientific concepts. Constructed-response assessments, in which students must create their own answer, may better reveal students’ thinking, but are time- and resource-intensive to evaluate. This report describes the initial meeting of a National Science Foundation–funded cross-institutional collaboration of interdisciplinary science, technology, engineering, and mathematics (STEM) education researchers interested in exploring the use of automated text analysis to evaluate constructed-response assessments. Participants at the meeting shared existing work on lexical analysis and concept inventories, participated in technology demonstrations and workshops, and discussed research goals. We are seeking interested collaborators to join our research community.

This study explored the use of machine learning to automatically evaluate the accuracy of students’ written explanations of evolutionary change. Performance of the Summarization Integrated Development Environment (SIDE) program was compared to human expert scoring using a corpus of 2,260 evolutionary explanations written by 565 undergraduate students in response to two different evolution instruments (the EGALT-F and EGALT-P) that contained prompts that differed in various surface features (such as species and traits). We tested human-SIDE scoring correspondence under a series of different training and testing conditions, using Kappa inter-rater agreement values of greater than 0.80 as a performance benchmark. In addition, we examined the effects of response length on scoring success; that is, whether SIDE scoring models functioned with comparable success on short and long responses. We found that SIDE performance was most effective when scoring models were built and tested at the individual item level and that performance degraded when suites of items or entire instruments were used to build and test scoring models. Overall, SIDE was found to be a powerful and cost-effective tool for assessing student knowledge and performance in a complex science domain.

Despite concerted efforts by science educators to understand patterns of evolutionary reasoning in science students and teachers, the vast majority of evolution education studies have failed to carefully consider or control for item feature effects in knowledge measurement. Our study explores whether robust contextualization patterns emerge within particular evolutionary reasoning contexts, and the implications of these patterns for instruction, assessment, and models of cognition. We test four hypotheses regarding item feature effects on undergraduate biology majors' evolutionary reasoning using a sample of 1,200 open response explanations of evolutionary change across items differing in context and scale but standardized by taxon and trait. Evolutionary explanations were atomized into a series of scientific and naive biological elements and tallied among prompts and their features. We documented clear, significant, and predictable item feature effects on evolutionary explanations. Tasks involving evolutionary trait loss elicited a significantly greater number of naive biological elements than evolutionary trait gain tasks in all contexts, including: within species comparisons, between species comparisons, animal prompts, and plant prompts. Tasks involving between species evolutionary comparisons, regardless of gain or loss, animal or plant, always produced significantly more naive biological explanatory elements than within species comparisons. For items prompting explanation of trait gain, the use of the core concepts of natural selection were not influenced by the hierarchical level of the task (within or between species). Explanations of trait gain were also the least sensitive to scale and context. Core concepts of natural selection were always deployed less frequently in cases of evolutionary trait loss (within and between species, in animals and plants). We discuss a series of implications of these findings for curriculum, instruction, and assessment. © 2010 Wiley Periodicals, Inc., Inc. J Res Sci Teach 48: 237–256, 2011

The notion of “pressure” as an evolutionary “force” that “causes” evolution is a pervasive linguistic feature of biology textbooks, journal articles, and student explanatory discourse. We investigated the consequences of using a textbook and curriculum that incorporate so-called force-talk. We examined the frequency with which biology majors spontaneously used notions of evolutionary “pressures” in their explanations, students’ definitions and explanations of what they meant when they used pressures, and the structure of explanatory models that incorporated evolutionary pressures and forces. We found that 12–20 percent of undergraduates spontaneously used “pressures” and/or “forces” as explanatory factors but significantly more often in trait gain scenarios than in trait loss scenarios. The majority of explanations using “force-talk” were characterized by faulty evolutionary reasoning. We discuss the conceptual similarity between faulty notions of evolutionary pressures and linguists’ force-dynamic models of everyday reasoning and ultimately question the appropriateness of force-talk in evolution education.