Search

Although recent studies have used the Classroom Observation Protocol for Undergraduate STEM (COPUS) to make claims about faculty reform, important questions remain: How should COPUS measures be situated within existing reform frameworks? Is there a universal sampling intensity that allows for valid inferences about the frequency of student-centered instruction within a semester or across semesters of a course? These questions were addressed using longitudinal COPUS observations (128 classes, three faculty, 4 years). COPUS behaviors were used to categorize classes into didactic, interactive lecture, or student-centered instructional styles. Sampling intensities (one to 11 classes) were simulated (1000 times) within a course and across semesters. The sampling intensities required for generating valid inferences about: (1) the "presence" of student-centered instruction; and (2) the "proportion" of instructional styles in a course and through time were calculated. Results indicated that the sampling intensity needed to characterize courses and instructors varied and was much higher than previously recommended for instructors with: (1) rare instances of student-centered classes; (2) variability in instructional style; and (3) longitudinal changes in instructional patterns. These conditions are common in early reform contexts. This study highlights the risks of broad, decontextualized sampling protocol recommendations and illustrates how reform frameworks, sampling intensities, and COPUS measures interact to impact inferences about faculty change.

Machine learning (ML) has been increasingly employed in science assessment to facilitate automatic scoring efforts, although with varying degrees of success (i.e., magnitudes of machine-human score agreements [MHAs]). Little work has empirically examined the factors that impact MHA disparities in this growing field, thus constraining the improvement of machine scoring capacity and its wide applications in science education. We performed a meta-analysis of 110 studies of MHAs in order to identify the factors most strongly contributing to scoring success (i.e., high Cohen's kappa [kappa]). We empirically examined six factors proposed as contributors to MHA magnitudes: algorithm, subject domain, assessment format, construct, school level, and machine supervision type. Our analyses of 110 MHAs revealed substantial heterogeneity in [kappa](mean=0.64; range = 0.09-0.97, taking weights into consideration). Using three-level random-effects modeling, MHA score heterogeneity was explained by the variability both within publications (i.e., the assessment task level: 82.6%) and between publications (i.e., the individual study level: 16.7%). Our results also suggest that all six factors have significant moderator effects on scoring success magnitudes. Among these, algorithm and subject domain had significantly larger effects than the other factors, suggesting that technical features and assessment external features might be primary targets for improving MHAs and ML-based science assessments.

High levels of attrition characterize undergraduate science courses in the USA. Predictive analytics research seeks to build models that identify at-risk students and suggest interventions that enhance student success. This study examines whether incorporating a novel assessment type (concept inventories [CI]) and using machine learning (ML) methods (1) improves prediction quality, (2) reduces the time point of successful prediction, and (3) suggests more actionable course-level interventions. A corpus of university and course-level assessment and non-assessment variables (53 variables in total) from 3225 students (over six semesters) was gathered. Five ML methods were employed (two individuals, three ensembles) at three time points (pre-course, week 3, week 6) to quantify predictive efficacy. Inclusion of course-specific CI data along with university-specific corpora significantly improved prediction performance. Ensemble ML methods, in particular the generalized linear model with elastic net (GLMNET), yielded significantly higher area under the curve (AUC) values compared with non-ensemble techniques. Logistic regression achieved the poorest prediction performance and consistently underperformed. Surprisingly, increasing corpus size (i.e., amount of historical data) did not meaningfully impact prediction success. We discuss the roles that novel assessment types and ML techniques may play in advancing predictive learning analytics and addressing attrition in undergraduate science education.

Educators seek to harness knowledge from educational corpora to improve student performance outcomes. Although prior studies have compared the efficacy of data mining methods (DMMs) in pipelines for forecasting student success, less work has focused on identifying a set of relevant features prior to model development and quantifying the stability of feature selection techniques. Pinpointing a subset of pertinent features can (1) reduce the number of variables that need to be managed by stakeholders, (2) make "black-box" algorithms more interpretable, and (3) provide greater guidance for faculty to implement targeted interventions. To that end, we introduce a methodology integrating feature selection with cross-validation and rank each feature on subsets of the training corpus. This modified pipeline was applied to forecast the performance of 3225 students in a baccalaureate science course using a set of 57 features, four DMMs, and four filter feature selection techniques. Correlation Attribute Evaluation (CAE) and Fisher's Scoring Algorithm (FSA) achieved significantly higher Area Under the Curve (AUC) values for logistic regression (LR) and elastic net regression (GLMNET), compared to when this pipeline step was omitted. Relief Attribute Evaluation (RAE) was highly unstable and produced models with the poorest prediction performance. Borda's method identified grade point average, number of credits taken, and performance on concept inventory assessments as the primary factors impacting predictions of student performance. We discuss the benefits of this approach when developing data pipelines for predictive modeling in undergraduate settings that are more interpretable and actionable for faculty and stakeholders.

Belief in genetic determinism (BGD) has been associated with increased social stereotyping and prejudicial views and consequently is of significant concern to science educators. The Public Understanding and Attitudes towards Genetics and Genomics (PUGGS) instrument was developed to measure relationships among BGD, genetics knowledge, and demographic variables. PUGGS validity evidence has relied primarily on Classical Test Theory frameworks and Brazilian samples. Using a more advanced psychometric framework (Rasch analysis) and a large North American undergraduate sample (n > 800), we further evaluate validity claims by studying (1) dimensionality and function of PUGGS item sets; (2) magnitudes of item endorsement across human traits (social, biological) and taxonomic (animal, plant) categories; and (3) degree of trait-level genetic overattribution. Similar to Gericke et al. (Sci Educ 26:1223-1259, 2017), we identified a two-dimensional structure for the BGD scale (i.e., social, biological) and the genetics knowledge scale (i.e., gene-environment interactions [GEI], genetics and genomics knowledge [GGK]). However, there were several problems with the functioning of the item sets (e.g., low reliability for GEI, problematic rating scale for BGD biological). We report that the magnitudes of GEI and GGK did not differ by taxonomic context. Finally, genetic over- (and under-) attribution was identified for both biological and social traits, indicating that students harbored considerably diverse and frequently non-normative conceptions about genetic contributions to traits. Importantly, psychometric and theoretical concerns reported here raise questions about the operationalization of the PUGGS BGD construct. Recommendations for PUGGS revisions and educational implications are discussed.

This study develops a framework to conceptualize the use and evolution of machine learning (ML) in science assessment. We systematically reviewed 47 studies that applied ML in science assessment and classified them into five categories: (a) constructed response, (b) essay, (c) simulation, (d) educational game, and (e) inter-discipline. We compared the ML-based and conventional science assessments and extracted 12 critical characteristics to map three variables in a three-dimensional framework: "construct," "functionality," and "automaticity." The 12 characteristics used to construct a profile for ML-based science assessments for each article were further analyzed by a two-step cluster analysis. The clusters identified for each variable were summarized into four levels to illustrate the evolution of each. We further conducted cluster analysis to identify four classes of assessment across the three variables. Based on the analysis, we conclude that ML has transformed--but not "yet" redefined--conventional science assessment practice in terms of fundamental purpose, the nature of the science assessment, and the relevant assessment challenges. Along with the three-dimensional framework, we propose five anticipated trends for incorporating ML in science assessment practice for future studies: addressing developmental cognition, changing the process of educational decision making, personalized science learning, borrowing 'good' to advance 'good', and integrating knowledge from other disciplines into science assessment.

College science instructors need continuous professional development (PD) to meet the call to evidence-based practice. New PD efforts need to focus on the nuanced blend of factors that influence instructors' teaching practices. We used persona methodology to describe the diversity among instructors who were participating in a long-term PD initiative. Persona methodology originates from ethnography. It takes data from product users and compiles those data in the form of fictional characters. Personas facilitate user-centered design. We identified four personas among our participants: Emma the Expert views herself as the subject-matter expert in the classroom and values her hard-earned excellence in lecturing. Ray the Relater relates to students and focuses on their points of view about innovative pedagogies. Carmen the Coach coaches her students by setting goals for them and helping them develop skill in scientific practices. Beth the Burdened owns the responsibility for her students' learning and feels overwhelmed that students still struggle despite her use of evidence-based practice. Each persona needs unique PD. We suggest ways that PD facilitators can use our personas as a reflection tool to determine how to approach the learners in their PD. We also suggest further avenues of research on learner-centered PD.

A large body of research has examined students' conceptions of evolution and their relationships to acceptance of evolution. Proficiency in statistical and probabilistic reasoning has long been considered to be an essential feature of evolutionary reasoning, yet almost no empirical work has explored these putative connections. The RaPro instruments have recently been developed to measure statistical reasoning in the contexts of mathematics (RaProMath) and evolution (RaProEvo). Our study provides additional validation of these instruments using Rasch analysis and quantifies the contribution of statistical reasoning to both understanding and accepting evolution. We recruited a large sample (N = 564) of undergraduate students enrolled in an introductory biology course at a large public research university in the United States. Students completed a suite of published instruments that assessed statistical reasoning, evolutionary understanding, and evolutionary acceptance. Our findings indicate that validity inferences derived from RaPro scores generalized to the new sample, and that proficiency in statistical reasoning explained 28% of the variance in evolutionary knowledge and 19% of the variation in evolutionary acceptance. The inclusion of demographic variables into the model significantly increased the explained variance in acceptance. Notably, the variance in evolution acceptance explained by statistical reasoning was comparable to that of thinking dispositions or evolutionary knowledge reported in the literature. This work provides the first large-scale evidence of the role of statistical reasoning in evolutionary knowledge and acceptance and motivates future work to explore how statistical literacy should be integrated into evolution education efforts.

Theoretical perspectives and empirical research suggest that acceptance of evolution may be contingent upon what is evolving (e.g., plants vs. humans) and the scale of change (microevolution vs. macroevolution). The Inventory of Student Evolution Acceptance (I-SEA) is the only instrument designed to measure acceptance at different evolutionary scales and in different evolutionary contexts. Nevertheless, current validity testing for this instrument remains limited and grounded in Classical Test Theory. In this study, we examine patterns of evolution acceptance using the I-SEA instrument in a large sample of undergraduates (n > 2,000 participants) from six semesters of an evolution-focused biology class. We examine three research questions: (RQ1) Does Rasch analysis support the psychometric properties of the I-SEA discussed by Nadelson and Southerland? (RQ2) How variable are I-SEA measures across semesters, and are they sensitive to evolution instruction? (RQ3) Are I-SEA measures comparable to evolution acceptance measures produced by the Generalized Acceptance of EvolutioN Evaluation? Our empirical results support theoretical claims that evolution acceptance does not appear to be a unidimensional construct among novice learners. We identify limitations of the I-SEA and recommend multiple modifications to improve instrument quality.

The goal of our study was to examine a large (>400), cross-sectional sample of Chinese pre-service biology teachers (PBTs) in order to document their evolution acceptance levels, evolution knowledge, and evolutionary reasoning patterns. This approach was taken in order to better understand the degree to which particular evolutionary reasoning difficulties exist independent of religious worldviews. The sample included (1) 160 PBTs tasked with completing four items from the ACORNS instrument, (2) 320 PBTs who completed the CINS and MATE instruments, and (3) 32 teachers who completed semi-structured clinical interviews using four ACORNS items. Findings from these samples revealed that Chinese PBTs' knowledge (CINS) and acceptance (MATE) were equivalent with teachers' scores from other countries, whereas performance on explanation tasks was lower. Scores from the CINS, MATE, and ACORNS did not reveal any significant improvements through the four-year teacher education programme. Although a large body of work has shown the important roles that religious affiliation and religiosity can play in evolutionary understanding and acceptance, our findings demonstrate that many evolutionary reasoning difficulties extend beyond religious factors, and add to a growing body of work showing that religiosity does not adequately account for PBTs' moderate evolution acceptance.

Helping faculty develop high-quality instruction that positively affects student learning can be complicated by time limitations, a lack of resources, and inexperience using student data to make iterative improvements. We describe a community of 16 faculty from five institutions who overcame these challenges and collaboratively designed, taught, iteratively revised, and published an instructional unit about the potential effect of mutations on DNA replication, transcription, and translation. The unit was taught to more than 2000 students in 18 courses, and student performance improved from preassessment to postassessment in every classroom. This increase occurred even though faculty varied in their instructional practices when they were teaching identical materials. We present information on how this faculty group was organized and facilitated, how members used student data to positively affect learning, and how they increased their use of active-learning instructional practices in the classroom as a result of participation. We also interviewed faculty to learn more about the most useful components of the process. We suggest that this professional development model can be used for geographically separated faculty who are interested in working together on a known conceptual difficulty to improve student learning and explore active-learning instructional practices.

Understanding how situational features of assessment tasks impact reasoning is important for many educational pursuits, notably the selection of curricular examples to illustrate phenomena, the design of formative and summative assessment items, and determination of whether instruction has fostered the development of abstract schemas divorced from particular instances. The goal of our study was to employ an experimental research design to quantify the degree to which situational features impact inferences about participants' understanding of Mendelian genetics. Two participant samples from different educational levels and cultural backgrounds (high school, n = 480; university, n = 444; Germany and USA) were used to test for context effects. A multi-matrix test design was employed, and item packets differing in situational features (e.g., plant, animal, human, fictitious) were randomly distributed to participants in the two samples. Rasch analyses of participant scores from both samples produced good item fit, person reliability, and item reliability and indicated that the university sample displayed stronger performance on the items compared to the high school sample. We found, surprisingly, that in both samples, no significant differences in performance occurred among the animal, plant, and human item contexts, or between the fictitious and "real" item contexts. In the university sample, we were also able to test for differences in performance between genders, among ethnic groups, and by prior biology coursework. None of these factors had a meaningful impact upon performance or context effects. Thus some, but not all, types of genetics problem solving or item formats are impacted by situational features.

We conducted a study of 19 biology instructors participating in small, local groups at six research-intensive universities connected to the Automated Analysis of Constructed Response (AACR) project (www.msu.edu/~aacr). Our aim was to uncover participants' motivation to persist in a long-term teaching professional development effort, a topic that is understudied in discipline-based educational research. We interviewed each participant twice over a 2-year period and conducted qualitative analyses on the data, using expectancy-value theory as a framework for considering motivation. Our analyses revealed that motivation among instructors was high due to their enjoyment of the AACR groups. The high level of motivation is further explained by the fact that AACR groups facilitated instructor involvement with the larger AACR project. We also found that group dynamics encouraged persistence; instructors thought they might never talk with colleagues about teaching in the absence of AACR groups; and groups were perceived to have a low-enough time requirement to warrant sustained involvement. We conclude that instructors have persisted in AACR groups because the groups provided great value with limited cost. The characterization of instructor experiences described here can contribute to a better understanding of faculty needs in teaching professional development.

Automated computerized scoring systems (ACSSs) are being increasingly used to analyze text in many educational settings. Nevertheless, the impact of misspelled words (MSW) on scoring accuracy remains to be investigated in many domains, particularly jargon-rich disciplines such as the life sciences. Empirical studies confirm that MSW are a pervasive feature of human-generated text and that despite improvements, spell-check and auto-replace programs continue to be characterized by significant errors. Our study explored four research questions relating to MSW and text-based computer assessments: (1) Do English language learners (ELLs) produce equivalent magnitudes and types of spelling errors as non-ELLs? (2) To what degree do MSW impact concept-specific computer scoring rules? (3) What impact do MSW have on computer scoring accuracy? and (4) Are MSW more likely to impact false-positive or false-negative feedback to students? We found that although ELLs produced twice as many MSW as non-ELLs, MSW were relatively uncommon in our corpora. The MSW in the corpora were found to be important features of the computer scoring models. Although MSW did not significantly or meaningfully impact computer scoring efficacy across nine different computer scoring models, MSW had a greater impact on the scoring algorithms for naïve ideas than key concepts. Linguistic and concept redundancy in student responses explains the weak connection between MSW and scoring accuracy. Lastly, we found that MSW tend to have a greater impact on false-positive feedback. We discuss the implications of these findings for the development of next-generation science assessments.

Understanding sources of performance bias in science assessment provides important insights into whether science curricula and/or assessments are valid representations of student abilities. Research investigating assessment bias due to factors such as instrument structure, participant characteristics, and item types are well documented across a variety of disciplines. However, the relationships among these factors are unclear for tasks evaluating understanding through performance on scientific practices, such as explanation. Using item-response theory (Rasch analysis), we evaluated differences in performance by gender on a constructed-response (CR) assessment about natural selection (ACORNS). Three isomorphic item strands of the instrument were administered to a sample of undergraduate biology majors and nonmajors (Group 1: n = 662 [female = 51.6%]; G2: n = 184 [female = 55.9%]; G3: n = 642 [female = 55.1%]). Overall, our results identify relationships between item features and performance by gender; however, the effect is small in the majority of cases, suggesting that males and females tend to incorporate similar concepts into their CR explanations. These results highlight the importance of examining gender effects on performance in written assessment tasks in biology.

A large body of work has been devoted to reducing assessment biases that distort inferences about students' science understanding, particularly in multiple-choice instruments (MCI). Constructed-response instruments (CRI), however, have invited much less scrutiny, perhaps because of their reputation for avoiding many of the documented biases of MCIs. In this study we explored whether known biases of MCIs--specifically item sequencing and surface feature effects--were also apparent in a CRI designed to assess students' understanding of evolutionary change using written explanation (Assessment of COntextual Reasoning about Natural Selection [ACORNS]). We used three versions of the ACORNS CRI to investigate different aspects of assessment structure and their corresponding effect on inferences about student understanding. Our results identified several sources of (and solutions to) assessment bias in this practice-focused CRI. First, along the instrument item sequence, items with similar surface features produced greater sequencing effects than sequences of items with dissimilar surface features. Second, a counterbalanced design (i.e., Latin Square) mitigated this bias at the population level of analysis. Third, ACORNS response scores were highly correlated with student verbosity, despite verbosity being an intrinsically trivial aspect of explanation quality. Our results suggest that as assessments in science education shift toward the measurement of scientific practices (e.g., explanation), it is critical that biases inherent in these types of assessments be investigated empirically.

Although historical changes in scientific ideas sometimes display striking similarities with students' conceptual progressions, some scholars have cautioned that such similarities lack meaningful commonalities. In the history of evolution, while Darwin and his contemporaries often used natural selection to explain evolutionary trait gain or increase, they struggled to use it to convincingly account for cases of trait loss or decrease. This study examines Darwin's evolutionary writings about trait gain and loss in the "Origin of Species" ("On the origin of species by means of natural selection, or the preservation of favoured races in the struggle for life." D. Appleton, New York, 1859) and compares them to written evolutionary explanations for trait gain and loss in a large (n > 500), cross-cultural and cross-sectional sample (novices and experts from the USA and Korea). Findings indicate that significantly more students and experts applied natural selection to cases of trait gain, but like Darwin and his contemporaries, they more often applied "use and disuse" and "inheritance of acquired characteristics" to episodes of trait loss. Although the parallelism between Darwin's difficulties and students' struggles with trait loss are striking, significant differences also characterize explanatory model structure. Overall, however, students and scientists struggles to explain trait loss--which is a very common phenomenon in the history of life--appear to transcend time, place, and level of biological expertise. The significance of these findings for evolution education are discussed; in particular, the situated nature of biological reasoning, and the important role that the history of science can play in understanding cognitive constraints on science learning.

The landscape of science education is being transformed by the new "Framework for Science Education" (National Research Council, "A framework for K-12 science education: practices, crosscutting concepts, and core ideas." The National Academies Press, Washington, DC, 2012), which emphasizes the centrality of scientific practices--such as explanation, argumentation, and communication--in science teaching, learning, and assessment. A major challenge facing the field of science education is developing assessment tools that are capable of validly and efficiently evaluating these practices. Our study examined the efficacy of a free, open-source machine-learning tool for evaluating the quality of students' written explanations of the causes of evolutionary change relative to three other approaches: (1) human-scored written explanations, (2) a multiple-choice test, and (3) clinical oral interviews. A large sample of undergraduates (n = 104) exposed to varying amounts of evolution content completed all three assessments: a clinical oral interview, a written open-response assessment, and a multiple-choice test. Rasch analysis was used to compute linear person measures and linear item measures on a single logit scale. We found that the multiple-choice test displayed poor person and item fit (mean square outfit >1.3), while both oral interview measures and computer-generated written response measures exhibited acceptable fit (average mean square outfit for interview: person 0.97, item 0.97; computer: person 1.03, item 1.06). Multiple-choice test measures were more weakly associated with interview measures (r = 0.35) than the computer-scored explanation measures (r = 0.63). Overall, Rasch analysis indicated that computer-scored written explanation measures (1) have the strongest correspondence to oral interview measures; (2) are capable of capturing students' normative scientific and naive ideas as accurately as human-scored explanations, and (3) more validly detect understanding than the multiple-choice assessment. These findings demonstrate the great potential of machine-learning tools for assessing key scientific practices highlighted in the new "Framework for Science Education."

The growing importance of genomics and bioinformatics methods and paradigms in biology has been accompanied by an explosion of new curricula and pedagogies. An important question to ask about these educational innovations is whether they are having a meaningful impact on students' knowledge, attitudes, or skills. Although assessments are necessary tools for answering this question, their outputs are dependent on their quality. Our study 1) reviews the central importance of reliability and construct validity evidence in the development and evaluation of science assessments and 2) examines the extent to which published assessments in genomics and bioinformatics education (GBE) have been developed using such evidence. We identified 95 GBE articles (out of 226) that contained claims of knowledge increases, affective changes, or skill acquisition. We found that 1) the purpose of most of these studies was to assess summative learning gains associated with curricular change at the undergraduate level, and 2) a minority (less than 10%) of studies provided any reliability or validity evidence, and only one study out of the 95 sampled mentioned both validity and reliability. Our findings raise concerns about the quality of evidence derived from these instruments. We end with recommendations for improving assessment quality in GBE.

Our study investigates the challenges introduced by students' use of lexically ambiguous language in evolutionary explanations. Specifically, we examined students' meaning of five key terms incorporated into their written evolutionary explanations: "pressure", "select", "adapt", "need", and "must". We utilized a new technological tool known as the Assessment Cascade System (ACS) to investigate the frequency with which biology majors spontaneously used lexically ambiguous language in evolutionary explanations, as well as their definitions and explanations of what they meant when they used such terms. Three categories of language were identified and examined in this study: terms with "Dual Ambiguity", "Incompatible Ambiguity", and "Unintended Ambiguity". In the sample of 1282 initial evolutionary explanations, 81% of students spontaneously incorporated lexically ambiguous language at least once. Furthermore, the majority of these initial responses were judged to be inaccurate from a scientific point of view. While not significantly related to gender, age, or reading/writing ability, students' use of contextually appropriate evolutionary language (pressure and adapt) was significantly associated with academic performance in biology. Comparisons of initial responses to follow-up responses demonstrated that the majority of student explanations were not reinterpreted after consideration of the follow-up response; nevertheless, a sizeable minority was interpreted differently. Most cases of interpretation change were a consequence of resolving initially ambiguous responses, rather than a change of accuracy, resulting in an increased understanding of students' evolutionary explanations. We discuss a series of implications of lexical ambiguity for evolution education.