Search

An explicit account of validity considerations within a published paper allows readers to evaluate the evidence that supports the interpretation and use of the data collected within a project. This editorial is meant to provide considerations on how validity has been presented and reviewed among papers submitted to Chemistry Education Research and Practice (CERP) that analyze quantitative data. Authors submitting to CERP are encouraged to make an explicit case for validity and this editorial describes the varying sources of evidence that can be used to organize the evidence presented for validity.

Prior research has demonstrated the important role of chemistry students’ affect in academic performance. Likert-scale surveys are the most prevalent tools to measure students’ affect within chemistry education research, however, data collected through a Likert-scale survey may exhibit response style bias which can hinder accurately measuring students’ affect. This study investigates the utility of a novel survey methodology, termed rank-sort survey, in understanding students’ academic motivation in a general chemistry course. Informed by Q methodology, in a rank-sort survey participants rank a set of statements in terms of level of agreement with limits in place on how many items can be assigned a particular rank. In this investigation, a rank-sort survey was developed by using statements from an existing Likert-scale instrument, the Academic Motivation Survey in Chemistry. Data collected from the rank-sort surveys, compared to Likert-scale surveys, showed a better alignment with self-determination theory, the underlying theoretical framework, and a better ability to predict students’ academic performance in chemistry. The study also discusses which surveys in chemistry education research are likely to benefit from adopting a rank-sort approach.

Representations are ubiquitous in chemistry. They are part of the chemistry language instructors use to communicate chemistry phenomenon to students. Literature calls in support of learning with multiple representations, but there is a pre-requisite for students to learn from a single representation. In this exploratory study, 1086 students in second semester general chemistry were randomly assigned to one of four representations showing bonding of sulfur dioxide: chemical formula, Lewis dot structure, an image of a ball and stick model, or an image of a space filling model. Students were asked to predict chemical properties of sulfur dioxide: relative bond length, molecular polarity, and the strongest intermolecular force with a water molecule. Using the lens of Multimedia Learning Theory on Learning with Text and Visual Representations, analyses of students’ prediction of chemical properties and the features cited when making predictions was conducted. Effect sizes were used to describe variations among representations in terms of how students predicted bond length, polarity and intermolecular forces. Meaningful differences were found across representations in students’ ability to correctly predict relative bond length and molecular polarity. These explorations generated the following hypotheses: (1) chemical formula leads students to depend on chemical conventions, (2) Lewis dot structure hinders predicting polarity when it shows an inaccurate shape, and (3) visual representations of ball and stick and space filling cue students to rely on visual estimations more than the other representations. Upon further testing, these hypotheses can inform instructors how to introduce representations and in the decision-making process of which representations to use to convey or assess a specific chemical property.

This paper is designed to synopsize the efforts of a team of general chemistry teachers to enact assessments during the abrupt transition to online-only instruction and reflect on what was done successfully and what could be improved. The focus is on the extent remote, online assessments accurately measured student knowledge described within the context of the decisions made for administration of the assessments. To limit the extent of unintended student collaboration, exams were given at a set day and time; question banks were developed so that students received variants of similar questions, and a remote proctoring software was used. Correlations showed consistency between online exams and paper exams, alleviating concerns that widespread cheating may take place. A concern was found with reusing the same exam at a later date for students who missed the original exam despite efforts to keep the online exam secure. The generation of alternative exams for later dates is recommended, potentially supported by creating and curating an online repository of assessment items to utilize.

Calls for assessments incorporating representations beyond the symbolic level (e.g., chemical reactions and formulas) have encouraged assessment designers to choose from a variety of representations in the design of chemistry assessments. This work expands on prior work in considering how representations are incorporated within assessments. First, students were given assessment items similar in task with common rationales for the design of answer choices with different representations substituted across the items. Second, students were randomly assigned to items identical in prompt, answer choices, and symbolic representation with experimental items supplementing symbolic representation with particulate or mathematical representation. The impact was operationalized as student performance, the performance of at-risk chemistry students scoring in the bottom quartile of the cohort’s standardized math test scores, and on the answer choices to which students attend. Substituting symbolic for particulate representation had a significant, negative impact on overall and at-risk student performance and a considerable impact on students’ selection of distractors. However, supplementing symbolic with a particulate representation had no discernible impact. Supplementing symbolic representations with tables including a column for “moles” had a significant, positive impact on overall student performance with considerable improvement observed among at-risk students. Students’ selection of a distractor representative of a failure to apply a mole-to-mole ratio was reduced when symbolic representations were supplemented with tables. Practical implications for the use of representations in a more equitable assessment design, and as a learning experience for students overall, are described.

The methodological limitations education researchers face in the evaluation of reformed instruction have led to debates as to the evidence advancing evidence-based practices. To conduct more effective research, methodological pluralism in the evaluation of educational reforms can be used to complement the strengths and limitations of a corpus of literature informing the impact of an evidence-based practice. This study seeks to introduce randomization tests, a nonparametric statistical analysis incorporating a random-assignment component that can be applied to a single-subject (N = 1) research design, as a methodology to be counted amongst evaluations of instructional reforms. To demonstrate the utility of this approach, an evaluation of peer-led team learning (PLTL) for classes of second-semester general chemistry spanning 7 semesters was conducted using randomization tests. The design contributes novel understandings of PLTL including differences in effectiveness across instructors, trends in effectiveness over time, and a perspective as to the appropriateness of assumptions concerning statistical independence when applied to educational settings. At the research setting, four instructors (each constituting an individual case) alternated implementing lecture-based instruction and PLTL by term. Across these four instructors, the treatment effects of peer-led team learning when compared to lecture-based instruction ranged in impact (from d = 0.233 to 2.09). For two instructors, PLTL provided a means by which to significantly reduce the differential performances observed of students with variable preparations in mathematics, thereby advancing the equitability of their courses. Implications of this work include the incorporation of single-subject research designs in establishing evidence-based instructional practices, the effectiveness of PLTL as interpreted in a methodologically pluralistic context of the research literature, and enacting measurements of equity when gauging the success of instructional reforms in science. Further, this introduction to randomization tests offers another methodology for the evaluation of instructional reforms more widely applicable in educational settings with smaller sample sizes (e.g., reforms conducted within a single classroom or upper-level courses with small class sizes).

Conceptually understanding chemistry requires the ability to transition among representational levels to use an understanding of submicroscopic entities and properties to explain macroscopic phenomena. Past literature describes student struggles with these transitions but provides limited information about upper-level post-secondary chemistry students’ abilities to transition among levels. This group is of particular interest as they are engaging in potentially their final training before entering a career as professional chemists, thus if students are likely to develop this skill during their formal education it should be manifest among this group. This study characterized analytical chemistry students’ responses to open-ended assessments on acid–base titrations and thin-layer chromatography for the use of sub-microscopic entities or properties to explain these macroscopic phenomena. Further, to understand whether explanatory statements were an expectation inherent in the instructional context of the setting, the analytical chemistry instructor's lectures on acid–base titrations and thin-layer chromatography were analyzed with the same framework. The analysis found that students seldom invoked explanatory statements within their responses and that congruence between lectures and responses to assessment was primarily limited to the use of macroscopic, descriptive terms. Despite the fact that the lecture in class regularly invoked explanatory statements in one context, this did not translate to student use of explanatory statements. To further test the hypothesis that analytical chemistry students struggle with explanatory statements, a follow-on study was also conducted among a second cohort of students reviewing their responses when specifically prompted to use sub-microscopic entities to explain a macroscopic phenomenon. The results suggest that fewer than half of the students showed proficiency on generating explanatory statements when explicitly prompted to do so. Instructional implications to promote explanatory statements are proposed in the discussion.

Chemistry-education research (CER) has progressed considerably in the United States since emerging as a discipline in the 1970s. Although CER graduate programs have become well established at a few universities, their success and growth may not be assured. Even with an increasing number of chemistry departments across the United States employing one or more CER faculty, CER can still be a novel entity to many traditional chemists. CER faculty continue to face the challenge of educating colleagues and students about CER scholarship. To start conversations about how we as a community can begin to address some of these challenges, a group of CER faculty representing a variety of backgrounds and experiences were brought together for a symposium at the 2018 Biennial Conference on Chemical Education. Most talks at the symposium were given by copresenters who had not previously worked together but had experience within the given topic. This commentary is an extension of that symposium in which the presenters use...

Students who score within the bottom quartile on cognitive measures of math aptitude have been identified as at-risk for low performance in chemistry courses, with less attention as to why such differential performance persists. At-risk students struggle most differentially on assessment items related to the mole concept and stoichiometry. An exploration as to the nature of the differential performance observed became of great interest as the assessment of these topics rarely progresses beyond multiplication or division, and at-risk students who achieved proficiency with the mole concept and stoichiometry had no noticeable gaps in academic chemistry performance when compared to students scoring in the top three quartiles of math aptitude. Thus, students in first-semester general chemistry were surveyed to describe their solution processes toward assessment items involving the mole concept and stoichiometry. Three hundred and forty-eight students responded to all survey prompts with 101 identified as at-risk. Findings suggest that while all students were observed to struggle in the conceptualization of the algorithms by which they execute solution processes, not-at-risk chemistry students were more likely to achieve correct answers via chemically implausible solution pathways. Rather than suggest the removal of assessment practices involving algorithmic, multiple-choice assessment on these topics, the implications include practical suggestions and opportunities for further research toward improving the equitability of measures used to assess proficiency with stoichiometry.

The identification of students at risk for academic failure in undergraduate chemistry courses has been heavily addressed in the literature. Arguably one of the strongest and most well-supported predictors of undergraduate success in chemistry is the mathematics portion of the SAT (SAT-M), a college-entrance, standardized test administered by the College Board. While students scoring in the bottom quartile of the SAT-M (herein referred to as at-risk) perform significantly worse on first-semester chemistry assessments, little is known of the topics on which these students differentially struggle. The purpose of this study is to provide insight as to which first-semester chemistry topics present an incommensurate challenge to at-risk students. Students were identified as either at-risk or not at-riskviaSAT-M scores. Students’ assessment responses were collected across four semesters of first-semester chemistry courses at a large, public university (N= 5636). At-risk students struggled consistently across all topics but disproportionately with mole concept and stoichiometry. Analyzing the trend in topics suggests that the struggles of at-risk students are not entirely attributable to topics that rely heavily on algorithms or algebraic math. Moreso, at-risk students found to have performed well on mole concept and stoichiometry went on to perform similarly as their not at-risk peers. The results support an instructional emphasis on these topics with reviewed literature offering promising, practical options to better serve at-risk students and broaden representation in the sciences.

The Achievement Goal Framework describes students’ goal orientations as: task-based, focusing on the successful completion of the task; self-based, evaluating performance relative to one's own past performance; or other-based, evaluating performance relative to the performance of others. Goal orientations have been used to explain student success in a range of educational settings, but have not been used in post-secondary chemistry. This study describes the goal orientations of General Chemistry students and explores the relationship of goal orientations to success in the course. On average, students report higher task and self orientations than other orientation. Task orientation had a positive relationship with exam performance and self orientation had a negative relationship with exam performance. Clustering students showed that for the majority of students task and self orientations moved concurrently and students with low preference across the three orientations also performed lowest on exams. Finally, students in classes using Flipped-Peer Led Team Learning, a pedagogy designed to bring active learning to a large lecture class, showed higher task orientation than those in classes with lecture-based instruction.

A well-established literature base identifies a portion of students enrolled in post-secondary General Chemistry as at-risk of failing the course based on incoming metrics. Learning about the experiences and factors that lead to this higher failure rate is essential toward improving retention in this course. This study examines the relationship between study habits and academic performance for at-risk students in General Chemistry. Students who were in the bottom quartile of SAT math scores were identified as at-risk students. The study habits of General Chemistry students, both those identified as at-risk and those not identified were measured by text message inquiries. The text message asked ‘‘Have you studied for General Chemistry I in the past 48 hours? If so, how did you study?” twice a week throughout a semester. Student responses to the messages were used to calculate the frequency of studying throughout the term. The results from a multiple regression analysis showed that high frequency of studying could mitigate the difference between at-risk and non-at-risk students on final exam scores. Additionally, the quality of studying for six at-risk students was analyzed by student interviews in concert with their text message responses. The results indicated that the quality of studying is not necessarily linked to frequency of studying and both quality and frequency can play a role in at-risk students' academic performance. The results presented offer a path for at-risk students to succeed in General Chemistry and the methodology presented offers a potential avenue for evaluating future efforts to improve student success.

The links students make among chemistry content is considered essential for a robust, enduring understanding in multiple learning theories. This article describes the development and implementation of an assessment technique, termed a Measure of Linked Concepts, designed to inform instructors on students’ understanding of linking content throughout General Chemistry. Student performance on the assessment technique has provided unique insights relevant for instruction. For example, a substantial proportion of students could not identify when a model was used beyond its intended limit or show proficiency in tasks that the course assumed was prior knowledge. The use of these assessments also provides a means for instruction to show the relevance for topics such as periodic trends or ionic/molecular classification in a variety of subsequent topics throughout the course.

Evaluating the impact of Chemistry Education Research articles has historically centered on the impact factor of the publishing journal. With the advent of electronic journal indices, it is possible to determine the impact of individual research articles by the number of citations it has received. However, in a relatively new discipline, such as Chemistry Education Research, it is necessary to provide context for the citation counts, particularly because Chemistry Education Research faculty are likely evaluated by chemistry faculty in more established subdisciplines. This study seeks to provide context by reviewing the citation counts for a sample of 749 Chemistry Education Research articles published in chemistry education or science education journals from 2007 through 2013. The number of citations was found to follow a non-normal distribution and, thus, results are presented using quartiles to describe the range of citations. The results are delineated by metric of citations (Web of Science and Google ...

This study used a series of text message inquiries sent to General Chemistry students asking: “Have you studied for General Chemistry I in the past 48 hours? If so, how did you study?” This method for collecting data is novel to chemistry education research so the first research goals were to investigate the feasibility of the technique and the evidence for validity of the data collected. The results showed that text messages provide ample data on students' study habits though initial participant recruitment may pose a challenge. This study also explored evidence for validity and found that the percent of students reporting studying peaked with each exam date matching the expected trend (content validity) and participants in the study had only small departures from the population of students at the setting (generalizable validity). Second, students' study habits were characterized using cluster analysis finding three clusters: students that knowingly do not study, students who describe mandatory course components as studying and students who study in addition to the mandatory course components. These student groups were compared on a common exam in the course with the last group out-performing those who knowingly do not study. Finally, student study habits were charted across the semester and show signs of adapting, possibly as a result of course expectations or course content.

Considerable effort in chemistry education research has been dedicated to developing and evaluating reform pedagogies designed to improve student success in general chemistry. Policy recommendations propose adoption of these techniques as a means to increase the number of science graduates, however there is the potential that the impact of these reforms is mitigated in subsequent classes. This study examines the longitudinal impact of a successful peer-led team learning reform that targets general chemistry. The study uses a quasi-experimental methodology comparing students who took general chemistry with peer-led team learning to students who experienced traditional instruction. Student enrollment and academic performance in subsequent classes were the outcome metrics evaluated. The results found that the reform impacts enrollment in the class that directly follows the target class but enrollment in subsequent classes is mitigated through student attrition within the curriculum. Additionally, no evidence...

Validity of educational research instruments and student assessments has appropriately become a growing interest in the chemistry education research community. Of particular concern is an attention to the consequences to students that result from the interpretation of assessment scores and whether those consequences are swayed by invalidity within an assessment. This study examines external and consequential validity of a first-term general chemistry exam from the ACS Examinations Institute. The measure was used as a final exam in the intended course at the research setting. As a result, student performance on the exam contributed to the grading decision in the course. Owing to the prerequisite relationship, this grading decision then informed the decision to permit student enrollment in second-semester general chemistry. This study evaluates the appropriateness of the first-term general chemistry exam to inform this decision. Results indicate that the exam offers meaningful information into students' performance in the follow-on course with a consistent decision process across student subgroups.

Assumptive Learning Theory values the active process of linking concepts to promote meaningful over rote learning. To promote meaningful learning, assessment practices that encourage the linking of concepts need to be developed and utilized. Creative Exercises (CEs) have the potential to encourage such links. CEs are an open-ended assessment technique where students are given a single prompt and are asked to describe as many statements as they can that are distinct, correct, and relevant to the prompt. This study describes a qualitative investigation into student responses to CEs for evidence of students linking concepts throughout the course and the nature of the linked concepts. The findings indicate considerable interconnections of content in student responses. Further, students’ efforts toward making connections revealed several misconceptions regarding their understanding of the limits of models. CEs are therefore proposed as a means to encourage students to link concepts and to inform instructors about the links made, both correctly and incorrectly. Finally, to determine the prevalence of the incorrect links, a novel assessment technique is proposed based on students' responses to CEs.

Peer-Led Team Learning (PLTL) is a widely disseminated pedagogical reform that employs previously successful undergraduate students, peer leaders, to lead sessions of structured group work in the target class. Numerous studies have evaluated the impact of this reform in various post-secondary chemistry classes. Results from these studies suggest that PLTL may be effective at improving student success in these classes, either through improved performance on common exams or reduced student attrition in the classes. This study seeks to take a broader picture at measuring the impact, by examining the role PLTL plays across a two semester General Chemistry sequence. This includes an analysis of PLTL on students' decision to progress through the two semester sequence, and on PLTL impact on the algorithm-heavy second-semester General Chemistry. The findings suggest that the PLTL implementation is robust in improving student success directly in terms of the target class. However, PLTL had little to no effect on students' decision to continue in the General Chemistry sequence. Additionally, PLTL had little effect on student performance in subsequent courses where the pedagogy returned to lecture-only instruction. The results suggest that PLTL implementation on one course within a sequence would have limited impact, and in order to improve student progress toward graduation, PLTL implementation may have to be curricular wide.

This study describes an evaluation of the Peer-Led Team Learning (PLTL) reform model in first-semester general chemistry. PLTL was implemented in place of one-third of the available lecture time, maintaining the same amount of structured class time under the reform. The evaluation demonstrates that classes implementing the PLTL reform at the setting featured a statistically significant improvement of 15% in the pass rate for the classes, compared to conventional, lecture-only classes at the same setting. Additionally, the PLTL classes maintained a comparable score on a comprehensive American Chemical Society final exam to that of the conventional classes. Combined, these findings suggest that the improvement in pass rates occurs while maintaining the same level of rigor in the PLTL classes. Examining student groups shows that students in underrepresented minority groups demonstrated the largest improvement in pass rates with the PLTL reform.

Open-ended assessments, defined as assessments with a large set of possible correct answers, by nature lend themselves to concerns regarding accurate and consistent grading. This article describes one particular open-ended assessment, named Creative Exercises (CE), designed for promoting students' interconnection of concepts in a college general chemistry setting. The article presents evidence concerning several aspects of validity, including the extent scores represent chemistry knowledge and the extent scoring is consistent across three graders. The evidence is also presented in the context of what is known about concept maps, a commonly employed open-ended assessment in chemistry. Implications for the administration of CEs and the appropriateness of measuring students' hierarchical organization of knowledge are also discussed as a result of this comparison.

This study employed hierarchical linear models (HLM) to investigate Peer-Led Guided Inquiry (PLGI), a teaching practice combining cooperative learning and inquiry and tailored for a large class. Ultimately, the study provided an example of the effective introduction of a reform pedagogical approach in a large class setting. In the narrative, the strengths of HLM as a statistical technique are discussed in the context of pedagogical reforms, and the case is made for the use of HLM in rigorous determinations of both effectiveness and equity associated with reform practices. The implementation of PLGI in a college-level general chemistry course is also described. For the study, two general HLM models suited to investigating reform were developed and utilized to examine the effectiveness and equity of PLGI as implemented in this setting. The models showed that the reform was associated with statistically significant improvement over traditional pedagogy in terms of academic performance on multiple measures, but achieving equity remained elusive: the consequences of pre-existing achievement gaps did not lessen as a result of reform implementation. The analysis led to several implications for educational research, particularly in terms of evaluating pedagogical reforms, as well as for teaching. Choosing effective and equitable pedagogies for classroom implementation remains a critical goal for classes of any size. © 2008 Wiley Periodicals, Inc. J Res Sci Teach 45: 794–811, 2008

This study describes the conceptual understanding for a series of fundamental organic concepts by fourth year chemistry students from a midsize, southeastern, state university. Student volunteers (n =19) participated in semi-structured interviews using a think aloud protocol. The interview questions were eleven multiple choice questions selected from the Organic Chemistry examination developed by the Examinations Institute of the Division of Chemical Education, American Chemical Society. The questions were chosen to represent a range of concepts that are fundamental to the organic chemistry course sequence. Analysis of the interview transcripts revealed trends in the students’ knowledge and alternative conceptions. Of particular interest, it was found that some alternative conceptions mirrored those previously seen with students in introductory chemistry, indicating the persistence of such alternative conceptions. Implications for instructional approaches particular to organic chemistry are discussed.

This article describes a guided-inquiry laboratory for upper-division inorganic chemistry. The experiment is designed to teach students about the trans effect through synthesis of the cis and trans geometric isomers of diamminedichloroplatinum(II). Students discover which factors lead to trans-substitution versus cis-substitution in square-planar transition-metal complexes. Student pairs are assigned a synthetic protocol, and product geometry is determined experimentally using a colorimetric thiourea test. Focus questions guide students to generate a trans-directing ligand series and to construct the trans effect concept. This experiment gives students technical experience in microscale inorganic synthesis and stereochemical analysis while teaching them about coordination chemistry and the importance of metals in medicine.

General chemistry is a required first step for students who wish to pursue a career in science or health professions. The course often has low rates of student success and as a result serves as a gateway limiting access to science fields. This study seeks to better understand factors that are related to student success in general chemistry by focusing on the affective domain, in this case students' self-concept, or self-evaluation of ability as it pertains to a specific field of study. First, a profile of students' self-concept in the general chemistry setting is created. Next, the relationship between self-concept and success in the course is investigated, including examining the role of self-concept after taking into account a cognitive measure (SAT scores). This study is unique in that evidence is found for the impact of self-concept after taking into account a cognitive measure. Finally, the effect of a semester-long, inquiry-oriented learning environment on students' self-concept is described. Sugges...

In educational research, particularly in the sciences, a common research design calls for the establishment of a control and experimental group to determine the effectiveness of an intervention. As part of this design, it is often desirable to illustrate that the two groups were equivalent at the start of the intervention, based on measures such as standardized cognitive tests or student grades in prior courses. In this article we use SAT and ACT scores to illustrate a more robust way of testing equivalence. The method incorporates two one-sided t tests evaluating two null hypotheses, providing a stronger claim for equivalence than the standard method, which often does not address the possible problem of low statistical power. The two null hypotheses are based on the construction of an equivalence interval particular to the data, so the article also provides a rationale for and illustration of a procedure for constructing equivalence intervals. Our consideration of equivalence using this method also underscores the need to include sample sizes, standard deviations, and group means in published quantitative studies.

To improve a large-enrollment general chemistry course based on conventional lectures, we instituted a reform combining peer-led team learning with a guided inquiry approach, together called peer-led guided inquiry (PLGI). For one group of first-semester general chemistry students, a PLGI session was combined with two lectures per week, and this group was compared to a control group that had the usual three lectures per week. Students were compared based on performance on identical course exams and on a final exam from the ACS Examinations Institute given at the end of the semester. The experimental group was found to perform better than the control group overall, in spite of experiencing one fewer lecture each week. Also, attendance at the PLGI sessions was found to have a significant positive impact on student performance, even when controlling for students’ SAT mathematics and verbal scores. This method of evaluating reform effects for institutions with several large sections of introductory chemistry ...