Search

An explicit account of validity considerations within a published paper allows readers to evaluate the evidence that supports the interpretation and use of the data collected within a project. This editorial is meant to provide considerations on how validity has been presented and reviewed among papers submitted to Chemistry Education Research and Practice (CERP) that analyze quantitative data. Authors submitting to CERP are encouraged to make an explicit case for validity and this editorial describes the varying sources of evidence that can be used to organize the evidence presented for validity.

Prior research has demonstrated the important role of chemistry students’ affect in academic performance. Likert-scale surveys are the most prevalent tools to measure students’ affect within chemistry education research, however, data collected through a Likert-scale survey may exhibit response style bias which can hinder accurately measuring students’ affect. This study investigates the utility of a novel survey methodology, termed rank-sort survey, in understanding students’ academic motivation in a general chemistry course. Informed by Q methodology, in a rank-sort survey participants rank a set of statements in terms of level of agreement with limits in place on how many items can be assigned a particular rank. In this investigation, a rank-sort survey was developed by using statements from an existing Likert-scale instrument, the Academic Motivation Survey in Chemistry. Data collected from the rank-sort surveys, compared to Likert-scale surveys, showed a better alignment with self-determination theory, the underlying theoretical framework, and a better ability to predict students’ academic performance in chemistry. The study also discusses which surveys in chemistry education research are likely to benefit from adopting a rank-sort approach.

Representations are ubiquitous in chemistry. They are part of the chemistry language instructors use to communicate chemistry phenomenon to students. Literature calls in support of learning with multiple representations, but there is a pre-requisite for students to learn from a single representation. In this exploratory study, 1086 students in second semester general chemistry were randomly assigned to one of four representations showing bonding of sulfur dioxide: chemical formula, Lewis dot structure, an image of a ball and stick model, or an image of a space filling model. Students were asked to predict chemical properties of sulfur dioxide: relative bond length, molecular polarity, and the strongest intermolecular force with a water molecule. Using the lens of Multimedia Learning Theory on Learning with Text and Visual Representations, analyses of students’ prediction of chemical properties and the features cited when making predictions was conducted. Effect sizes were used to describe variations among representations in terms of how students predicted bond length, polarity and intermolecular forces. Meaningful differences were found across representations in students’ ability to correctly predict relative bond length and molecular polarity. These explorations generated the following hypotheses: (1) chemical formula leads students to depend on chemical conventions, (2) Lewis dot structure hinders predicting polarity when it shows an inaccurate shape, and (3) visual representations of ball and stick and space filling cue students to rely on visual estimations more than the other representations. Upon further testing, these hypotheses can inform instructors how to introduce representations and in the decision-making process of which representations to use to convey or assess a specific chemical property.

The methodological limitations education researchers face in the evaluation of reformed instruction have led to debates as to the evidence advancing evidence-based practices. To conduct more effective research, methodological pluralism in the evaluation of educational reforms can be used to complement the strengths and limitations of a corpus of literature informing the impact of an evidence-based practice. This study seeks to introduce randomization tests, a nonparametric statistical analysis incorporating a random-assignment component that can be applied to a single-subject (N = 1) research design, as a methodology to be counted amongst evaluations of instructional reforms. To demonstrate the utility of this approach, an evaluation of peer-led team learning (PLTL) for classes of second-semester general chemistry spanning 7 semesters was conducted using randomization tests. The design contributes novel understandings of PLTL including differences in effectiveness across instructors, trends in effectiveness over time, and a perspective as to the appropriateness of assumptions concerning statistical independence when applied to educational settings. At the research setting, four instructors (each constituting an individual case) alternated implementing lecture-based instruction and PLTL by term. Across these four instructors, the treatment effects of peer-led team learning when compared to lecture-based instruction ranged in impact (from d = 0.233 to 2.09). For two instructors, PLTL provided a means by which to significantly reduce the differential performances observed of students with variable preparations in mathematics, thereby advancing the equitability of their courses. Implications of this work include the incorporation of single-subject research designs in establishing evidence-based instructional practices, the effectiveness of PLTL as interpreted in a methodologically pluralistic context of the research literature, and enacting measurements of equity when gauging the success of instructional reforms in science. Further, this introduction to randomization tests offers another methodology for the evaluation of instructional reforms more widely applicable in educational settings with smaller sample sizes (e.g., reforms conducted within a single classroom or upper-level courses with small class sizes).

Conceptually understanding chemistry requires the ability to transition among representational levels to use an understanding of submicroscopic entities and properties to explain macroscopic phenomena. Past literature describes student struggles with these transitions but provides limited information about upper-level post-secondary chemistry students’ abilities to transition among levels. This group is of particular interest as they are engaging in potentially their final training before entering a career as professional chemists, thus if students are likely to develop this skill during their formal education it should be manifest among this group. This study characterized analytical chemistry students’ responses to open-ended assessments on acid–base titrations and thin-layer chromatography for the use of sub-microscopic entities or properties to explain these macroscopic phenomena. Further, to understand whether explanatory statements were an expectation inherent in the instructional context of the setting, the analytical chemistry instructor's lectures on acid–base titrations and thin-layer chromatography were analyzed with the same framework. The analysis found that students seldom invoked explanatory statements within their responses and that congruence between lectures and responses to assessment was primarily limited to the use of macroscopic, descriptive terms. Despite the fact that the lecture in class regularly invoked explanatory statements in one context, this did not translate to student use of explanatory statements. To further test the hypothesis that analytical chemistry students struggle with explanatory statements, a follow-on study was also conducted among a second cohort of students reviewing their responses when specifically prompted to use sub-microscopic entities to explain a macroscopic phenomenon. The results suggest that fewer than half of the students showed proficiency on generating explanatory statements when explicitly prompted to do so. Instructional implications to promote explanatory statements are proposed in the discussion.

Students who score within the bottom quartile on cognitive measures of math aptitude have been identified as at-risk for low performance in chemistry courses, with less attention as to why such differential performance persists. At-risk students struggle most differentially on assessment items related to the mole concept and stoichiometry. An exploration as to the nature of the differential performance observed became of great interest as the assessment of these topics rarely progresses beyond multiplication or division, and at-risk students who achieved proficiency with the mole concept and stoichiometry had no noticeable gaps in academic chemistry performance when compared to students scoring in the top three quartiles of math aptitude. Thus, students in first-semester general chemistry were surveyed to describe their solution processes toward assessment items involving the mole concept and stoichiometry. Three hundred and forty-eight students responded to all survey prompts with 101 identified as at-risk. Findings suggest that while all students were observed to struggle in the conceptualization of the algorithms by which they execute solution processes, not-at-risk chemistry students were more likely to achieve correct answers via chemically implausible solution pathways. Rather than suggest the removal of assessment practices involving algorithmic, multiple-choice assessment on these topics, the implications include practical suggestions and opportunities for further research toward improving the equitability of measures used to assess proficiency with stoichiometry.

The identification of students at risk for academic failure in undergraduate chemistry courses has been heavily addressed in the literature. Arguably one of the strongest and most well-supported predictors of undergraduate success in chemistry is the mathematics portion of the SAT (SAT-M), a college-entrance, standardized test administered by the College Board. While students scoring in the bottom quartile of the SAT-M (herein referred to as at-risk) perform significantly worse on first-semester chemistry assessments, little is known of the topics on which these students differentially struggle. The purpose of this study is to provide insight as to which first-semester chemistry topics present an incommensurate challenge to at-risk students. Students were identified as either at-risk or not at-riskviaSAT-M scores. Students’ assessment responses were collected across four semesters of first-semester chemistry courses at a large, public university (N= 5636). At-risk students struggled consistently across all topics but disproportionately with mole concept and stoichiometry. Analyzing the trend in topics suggests that the struggles of at-risk students are not entirely attributable to topics that rely heavily on algorithms or algebraic math. Moreso, at-risk students found to have performed well on mole concept and stoichiometry went on to perform similarly as their not at-risk peers. The results support an instructional emphasis on these topics with reviewed literature offering promising, practical options to better serve at-risk students and broaden representation in the sciences.

The Achievement Goal Framework describes students’ goal orientations as: task-based, focusing on the successful completion of the task; self-based, evaluating performance relative to one's own past performance; or other-based, evaluating performance relative to the performance of others. Goal orientations have been used to explain student success in a range of educational settings, but have not been used in post-secondary chemistry. This study describes the goal orientations of General Chemistry students and explores the relationship of goal orientations to success in the course. On average, students report higher task and self orientations than other orientation. Task orientation had a positive relationship with exam performance and self orientation had a negative relationship with exam performance. Clustering students showed that for the majority of students task and self orientations moved concurrently and students with low preference across the three orientations also performed lowest on exams. Finally, students in classes using Flipped-Peer Led Team Learning, a pedagogy designed to bring active learning to a large lecture class, showed higher task orientation than those in classes with lecture-based instruction.

A well-established literature base identifies a portion of students enrolled in post-secondary General Chemistry as at-risk of failing the course based on incoming metrics. Learning about the experiences and factors that lead to this higher failure rate is essential toward improving retention in this course. This study examines the relationship between study habits and academic performance for at-risk students in General Chemistry. Students who were in the bottom quartile of SAT math scores were identified as at-risk students. The study habits of General Chemistry students, both those identified as at-risk and those not identified were measured by text message inquiries. The text message asked ‘‘Have you studied for General Chemistry I in the past 48 hours? If so, how did you study?” twice a week throughout a semester. Student responses to the messages were used to calculate the frequency of studying throughout the term. The results from a multiple regression analysis showed that high frequency of studying could mitigate the difference between at-risk and non-at-risk students on final exam scores. Additionally, the quality of studying for six at-risk students was analyzed by student interviews in concert with their text message responses. The results indicated that the quality of studying is not necessarily linked to frequency of studying and both quality and frequency can play a role in at-risk students' academic performance. The results presented offer a path for at-risk students to succeed in General Chemistry and the methodology presented offers a potential avenue for evaluating future efforts to improve student success.

This study used a series of text message inquiries sent to General Chemistry students asking: “Have you studied for General Chemistry I in the past 48 hours? If so, how did you study?” This method for collecting data is novel to chemistry education research so the first research goals were to investigate the feasibility of the technique and the evidence for validity of the data collected. The results showed that text messages provide ample data on students' study habits though initial participant recruitment may pose a challenge. This study also explored evidence for validity and found that the percent of students reporting studying peaked with each exam date matching the expected trend (content validity) and participants in the study had only small departures from the population of students at the setting (generalizable validity). Second, students' study habits were characterized using cluster analysis finding three clusters: students that knowingly do not study, students who describe mandatory course components as studying and students who study in addition to the mandatory course components. These student groups were compared on a common exam in the course with the last group out-performing those who knowingly do not study. Finally, student study habits were charted across the semester and show signs of adapting, possibly as a result of course expectations or course content.

Assumptive Learning Theory values the active process of linking concepts to promote meaningful over rote learning. To promote meaningful learning, assessment practices that encourage the linking of concepts need to be developed and utilized. Creative Exercises (CEs) have the potential to encourage such links. CEs are an open-ended assessment technique where students are given a single prompt and are asked to describe as many statements as they can that are distinct, correct, and relevant to the prompt. This study describes a qualitative investigation into student responses to CEs for evidence of students linking concepts throughout the course and the nature of the linked concepts. The findings indicate considerable interconnections of content in student responses. Further, students’ efforts toward making connections revealed several misconceptions regarding their understanding of the limits of models. CEs are therefore proposed as a means to encourage students to link concepts and to inform instructors about the links made, both correctly and incorrectly. Finally, to determine the prevalence of the incorrect links, a novel assessment technique is proposed based on students' responses to CEs.

Peer-Led Team Learning (PLTL) is a widely disseminated pedagogical reform that employs previously successful undergraduate students, peer leaders, to lead sessions of structured group work in the target class. Numerous studies have evaluated the impact of this reform in various post-secondary chemistry classes. Results from these studies suggest that PLTL may be effective at improving student success in these classes, either through improved performance on common exams or reduced student attrition in the classes. This study seeks to take a broader picture at measuring the impact, by examining the role PLTL plays across a two semester General Chemistry sequence. This includes an analysis of PLTL on students' decision to progress through the two semester sequence, and on PLTL impact on the algorithm-heavy second-semester General Chemistry. The findings suggest that the PLTL implementation is robust in improving student success directly in terms of the target class. However, PLTL had little to no effect on students' decision to continue in the General Chemistry sequence. Additionally, PLTL had little effect on student performance in subsequent courses where the pedagogy returned to lecture-only instruction. The results suggest that PLTL implementation on one course within a sequence would have limited impact, and in order to improve student progress toward graduation, PLTL implementation may have to be curricular wide.

Open-ended assessments, defined as assessments with a large set of possible correct answers, by nature lend themselves to concerns regarding accurate and consistent grading. This article describes one particular open-ended assessment, named Creative Exercises (CE), designed for promoting students' interconnection of concepts in a college general chemistry setting. The article presents evidence concerning several aspects of validity, including the extent scores represent chemistry knowledge and the extent scoring is consistent across three graders. The evidence is also presented in the context of what is known about concept maps, a commonly employed open-ended assessment in chemistry. Implications for the administration of CEs and the appropriateness of measuring students' hierarchical organization of knowledge are also discussed as a result of this comparison.

This study describes the conceptual understanding for a series of fundamental organic concepts by fourth year chemistry students from a midsize, southeastern, state university. Student volunteers (n =19) participated in semi-structured interviews using a think aloud protocol. The interview questions were eleven multiple choice questions selected from the Organic Chemistry examination developed by the Examinations Institute of the Division of Chemical Education, American Chemical Society. The questions were chosen to represent a range of concepts that are fundamental to the organic chemistry course sequence. Analysis of the interview transcripts revealed trends in the students’ knowledge and alternative conceptions. Of particular interest, it was found that some alternative conceptions mirrored those previously seen with students in introductory chemistry, indicating the persistence of such alternative conceptions. Implications for instructional approaches particular to organic chemistry are discussed.