Search

Using multiple versions of an assessment has the potential to introduce item environment effects. These types of effects result in version dependent item characteristics (i.e., difficulty and discrimination). Methods to detect such effects and resulting implications are important for all levels of assessment where multiple forms of an assessment are created. This report describes a novel method for identifying items that do and do not display form dependence. The first two steps identify form dependent items using a differential item functioning (DIF) analysis of item parameters estimated by Item Response Theory. The method is illustrated using items that appeared in four forms (two trial and two released versions) of a first semester general chemistry examination. Eighteen of fifty-six items were identified as having item parameters that were form dependent. Thirteen of those items displayed a form dependence consistent with reasons previously identified in the literature: preceding item difficulty, content priming, and a combination of preceding item difficulty and content priming. The remaining five items had form dependence that did not align reasons reported in the literature. An analysis was done to determine if all possible instances of predicted form dependence could be found. Several instances where form dependence could have been found, based on the preceding item difficulty or content priming, were identified, and those items did not display form dependence. We identify and rationalize form dependence for thirteen of the eighteen items flagged; however, we are unable to predict form dependence for items.

Practice examinations developed at the ACS Examinations Institute ask students to self-report mental effort when answering items. This self-reported mental effort together with performance can be represented in the form of a cognitive efficiency graph for each student giving information on the utilization of cognitive resources and content knowledge. When performance and mental effort used in the cognitive efficiency graph for a student is based on dichotomously scored data and raw reported mental effort, limitations arise. For example, it might be difficult to see a greater differentiation among students, and instructors might be interpreting student cognitive efficiency in certain content areas incorrectly. This study utilizes organic chemistry practice exam data to report an internal calibration method where a student's mental effort is calibrated using their average reported mental effort across all items. Additionally, this study describes the benefits of using partial-credit assignment to provide greater differentiation in student performance. Results suggest that a more accurate cognitive efficiency graph can be obtained leading to a better assessment of student utilization of cognitive resources for different content areas. These results can have implications for how similar data on other ACS exams or similar data being used by researchers and practitioners can be analyzed to better represent cognitive efficiency.

Pedagogies of engagement (i.e., Peer-Led Teaming Learning, Problem-Based Learning, and Process-Oriented Guided Inquiry Learning) are active learning approaches used in postsecondary chemistry courses. In this study, we use data from a national survey of postsecondary chemistry instructors in the United States to estimate use of three pedagogies in the course for which the instructor feels they have the most control. We found that 16.6% of these faculties report they are implementing Peer-Led Team Learning (PLTL), 10.6% report implementing Problem-Based Learning (PBL), and 10.7% report implementing Process-Oriented Guided-Inquiry Learning (POGIL). We compare use of select teaching practices and other active learning strategies with implementation of PLTL, PBL, and POGIL. Additionally, we use items from the survey to understand course, institution, and instructor characteristics associated with use. Key findings include that lower-level courses and courses with large enrollments are the most likely places for PLTL to be implemented and that instructors who are not on the tenure track are more likely to implement PLTL and POGIL than tenured/tenure-track instructors. Instructors who report implementing PLTL and PBL have more student-centered beliefs about teaching and learning, while instructors who report implementing POGIL have more teacher-centered beliefs about learning, albeit all with small effect sizes. Implications are offered for how instructors, researchers, developers and disseminators of these pedagogies can use our results to inform their practices and efforts.

Student interviews have found an increasingly important place in the educational research landscape. They are used to create and validate assessments and concept inventories, as well as to more clearly elucidate student thinking. One-on-one interviews are most often completed in person at the researcher's or student's institution. There are significant advantages, however, of conducting interviews remotely including access to more varied populations, travel-related time and cost savings, and the ability of a single researcher to efficiently collect data at multiple institutions. This paper presents a description of a remote interview process along with details of ways to maintain commonality with its in-person analog. Our research provides evidence that the efficacy of results is maintained and that remote interviews provide a viable alternative to more costly and time-intensive on-site interviews. These results are particularly timely as ways to enable graduate education and research to continue during the COVID-19 pandemic response.

The American Chemical Society, Division of Chemical Education, Examinations Institute has been developing content maps for the undergraduate program based on subdiscipline specifications since 2008. The Anchoring Concepts Content Maps (or ACCM) have been published in four subdisciplines (general, organic, physical, and inorganic chemistry) with the latest work presented here in a fifth subdiscipline: analytical chemistry. As with the development of the other maps, an alignment process of identifying key components of exam items including content, complexity, and other item characteristics was conducted with both analytical chemistry and instrumental analysis items. The results of the alignments as well as an evaluation of the newest ACCM in relation to the other published maps are presented.

The American Chemical Society (ACS), Division of Chemical Education, Examinations Institute, has been developing content maps to describe comprehensively the undergraduate curriculum aligned to traditional subdisciplines. These content maps have been developed through the combined efforts of many faculty members who teach the targeted courses in the subdiscipline. A recent collaboration between the Examinations Institute and the ACS Green Chemistry Institute has resulted in the consideration of green chemistry content in the context of the content maps. The inclusion of green chemistry concepts in the context of organic chemistry has been the initial focus of the work of this collaboration. Through working with faculty who teach green chemistry courses or organic chemistry courses with the theme of green chemistry, the organic chemistry content map has been revised and reconceptualized with the theme of green chemistry included. The process, excerpts of the content map, and alignments of traditional organic chemistry exam items to both the organic chemistry content map and the organic chemistry content map with the inclusion of green chemistry are reported.

As more instructors articulate learning objectives for their students within one course, or academic staff collaborate to articulate learning outcomes for programs, a robust means to assess student performance within these becomes increasingly important. The Examinations Institute of the American Chemical Society (ACS), Division of Chemical Education, has recently published content maps that utilise a structure of subdiscipline-independent fundamental concepts narrowing down to content details that are specific to subdisciplines. This structure has then been utilised to align items and can be used to assess student performance throughout a program. Learning objectives that are designed for a course can then be aligned to the framework and used to gauge student learning within a course or across a program. One key to making well-informed instructional decisions is to obtain as much valid information from such assessment work as possible. This paper describes the combination of using a rubric for assigning complexity with student performance to gauge achieving learning objectives that are aligned to fundamental concepts in the content maps in general chemistry and organic chemistry. It can be argued that information in these forms can provide useful guidance for designing improved instruction.

One theme that both the American Association for the Advancement of Science (AAAS) and the National Research Council (NRC) agree pervades any science course (and specifically chemistry) is scale. Despite the fact that scale concepts and skills are at the core of a student's scientific literacy, scale or the skills related to understanding quantity within chemistry concepts is often neglected in instruction. Beginning with a supplemental instruction module and moving into both laboratory and lecture, we have systematically developed and integrated increasingly more effective methods of delivery of scale-themed content across the general chemistry curriculum at one institution. Using regression models to examine for learning gains on chemistry content final exams, we found that the greatest learning gains for General Chemistry I resulted from combining all components of scale-themed instruction; for General Chemistry II the greatest impact was found with incorporating scale-themed laboratory experiments.

The ability to assess students' content knowledge and make meaningful comparisons of student performance is an important component of instruction. ACS exams have long served as tools for standardized assessment of students' chemistry knowledge. Because these exams are designed by committees of practitioners to cover a breadth of topics in the curriculum, they may contain items that an individual instructor may not cover during classroom instruction and therefore chooses not to assess. For the instructor to make meaningful comparisons between his or her students and the national norm sample, the instructor needs norms that are generated upon the basis of the subset of items he or she used. The goal of this project was to investigate the effects of norm stability when items were removed from ACS General Chemistry Exams. This was achieved by monitoring the average change in percentile for students as items were removed from the exam and noting when average change crossed a specified threshold. An exploration of subset norm stability for three commonly used ACS General Chemistry Exams is presented along with implications for research and instruction.

Assessment and evaluation tools and instruments are developed to measure many things from content knowledge to misconceptions to student affect. The standard validation processes for these are regularly conducted and provide strong evidence for the validity of the measurements that are made. As part of the suite of validation tools available to researchers and practitioners, response process validity studies are an important component for the development and study of multiple-choice or forced-response items. An assessment was developed to measure the scale literacy skills of students in General Chemistry. The initial validation of this instrument was reported earlier, with the response process validity work now presented here to complement the previous study. In addition to the validity work with students in General Chemistry, an additional study was conducted with students in Anatomy and Physiology, as part of the validation studies as the assessment was moved into this new discipline. The process used, threats identified, and implications of these identified threats are reported.

The American Association for the Advancement of Science (AAAS) has outlined four themes that define science literacy; these are systems, models, constancy and change, and scale. More recently, the National Research Council has released the framework for K-12 science education that includes "Scale, Proportion, and Quantity". Scale literacy, the collection of concepts and skills that contribute to the ability of a student to move between different scale worlds, has long been of interest in education research. In chemistry, students are confronted with issues of scale in almost every aspect of their learning as they navigate the macroscopic world we live in and the submicroscopic world chemistry happens on. Unsurprisingly, previous research has shown that students have a limited understanding of scale and demonstrate a narrow range of size in which they are comfortable operating in. Before meaningful instruction can be developed to help students increase their ability in scale, research must first focus on understanding the current conceptions of scale held by students. The research presented here replicates previous work that investigated undergraduate chemistry students' absolute and relative scaling abilities in an interview setting, on a larger classwide scale.

Missing data is a regular issue that researchers and practitioners must consider for treatment. Commonly, cases for which data is missing are excluded from inclusion in larger data sets. However, this is not the only option and could artificially alter the sample. Other options are available for imputing missing data. Expanding on work previously reported, a method is presented here that not only preserves all observed data but also is shown to function for smaller data sets. As an example of the process, four ACS Exams are used as prototypes with a discussion on an expected noise level of any imputed sample.

The aim of this study is to offer a current snapshot of flipped classroom use in postsecondary chemistry education. Data from a national survey of chemistry faculty members in the United States formed the basis of an investigation into the instructional contexts in which flipped classroom pedagogies are employed in postsecondary chemistry education. Our results reveal an association between flipped classroom use and the level at which a course is taught; in addition, our results provide support for the utility of flipped classrooms as a means for incorporation of pedagogical practices focused on active and collaborative learning.

While research on and development of evidence-based instructional practices (EBIPs) in STEM education has flourished, implementation of these practices classrooms has not been as prolific. Using the teacher-centered systemic reform model as a framework, we explore the connection between chemistry instructors' beliefs about teaching and learning and self-efficacy beliefs, and their enacted classroom practices. Postsecondary chemistry faculty present a unique population for the study because of their role in teaching prerequisite courses, such as general and organic chemistry, which are key to many science major fields. A measure of teacher beliefs and self-efficacy was administered to a national survey of postsecondary chemistry faculty members. Instructional practices used in a chemistry course were also collected via self-report. While instructional practices were not directly observed, a cluster analysis of our data mirrors patterns of instructional practices found in observation-based studies of chemistry faculty. Significant differences are found on teacher thinking and self-efficacy measures based on enacted instructional practices. Results support the hypothesized connection between beliefs and instructional practice on a larger scale than in previous studies of this relationship, bolstering the evidence for the importance of this relationship over previously criticized results. These results present a call for reform efforts on fostering change from its core, that is, the beliefs of those who ultimately adopt EBIPs. Dissemination and design should incorporate training and materials that highlight the process by which faculty members interpret reformed practices within their belief system, and explore belief change in the complex context of education reform.

The American Chemical Society-Examinations Institute (ACS-EI) released the first ACS Foundations of Inorganic Chemistry Examination in 2016 to better address the two generalized types of inorganic chemistry courses (i.e., at the foundation or second/third-year level and the indepth or third/fourth-year level). The ACS Inorganic Chemistry Exam that has traditionally been offered addresses the content and skills most associated with a third/fourth-year level course. Topics selected for the foundations level exam were informed by a national survey of inorganic chemistry faculty; however, the topical framework for the survey study was notably different from the test architecture employed by ACS Exams (i.e., the Anchoring Concepts Content Map). We should note that when the foundations level exam was being developed, the Inorganic Chemistry Anchoring Concepts Content Map was not completed; however, now that the ACCM framework has been finalized, there is an opportunity to consider how the foundations and in-depth level examinations differ based on the content map. This study, therefore, compares content coverage of the 2016 ACS Foundations of Inorganic Chemistry Examination and the 2014 ACS Inorganic Chemistry Examination. Exam items were aligned using the Big Ideas and Enduring Understanding levels of the Anchoring Concepts Content Map (ACCM) and were also evaluated based on complexity. This work demonstrates the notable differences in the two examinations by content; this work also provides a structure for faculty or instructors to use when aligning test items or learning objectives from different courses and a discussion on how to utilize the results of those alignments.

ACS Examinations provide a lens through which to examine historical changes in topic coverage via analyses of course-specific examinations. This study is an extension of work completed previously by the ACS Exams Research Staff and collaborators in general chemistry, organic chemistry, and physical chemistry to explore content changes in the principal courses of the postsecondary chemistry curriculum. In this study, we consider how inorganic chemistry content coverage has varied over a 55-year period since the first inorganic chemistry ACS Examination was released in 1961. A total of 860 items was evaluated on the basis of problem type (i.e., algorithmic, conceptual, or recall), use of visual-spatial or reference components, and content coverage. Our analyses identify core content areas in the inorganic chemistry curriculum, consistent with those reported in faculty surveys. Each examination also contained questions addressing a variety of specialty areas that vary widely within the discipline between 1961 and 2016. Unlike the results from historical reviews of general chemistry and organic chemistry ACS Examinations, we observe great variability across the 13 inorganic chemistry examinations with an absence of strong trends in inclusion or exclusion of problem types, visual-spatial or reference components, or content across the 13 exams analyzed. Our results offer a framework for using historical ACS Examinations as a tool to make decisions about the future of content coverage in postsecondary inorganic chemistry education.

For the past eight years, the ACS Examinations Institute has been developing Anchoring Concepts Content Maps for the different subdisciplines taught throughout the undergraduate curriculum. The structure of the map consists of two top levels that are shared throughout the entire curriculum and two subdiscipline specific levels that contain finer-grained content details. This paper presents the four levels of the content map for inorganic chemistry.

General chemistry courses are often the foundation for the study of other science disciplines and upper-level chemistry concepts. Students who take introductory chemistry courses are more often from health and science-related fields than chemistry. As such, the content taught and assessed in general chemistry courses is envisioned as building blocks for concepts in other science courses and across many science disciplines. American Chemical Society (ACS) exams are developed by committees of expert chemists and serve as representative artifacts of content valued by the chemistry community. Before an exam is released, items developed by an examination committee undergo trial testing, and items deemed too easy, too hard, or not discriminating based on item statistics are removed. Analysis of content coverage from items on multiple released general chemistry ACS exams from the past decade and their associated unreleased trial tests revealed content areas where few exam items were written. Content coverage was analyzed by aligning exam items to the ACS Anchoring Concepts Content Map for General Chemistry. By comparing released and unreleased trial items, we found that many of the content coverage gaps on released exams are the result of a lack of items developed about these concepts. These conceptual holes warrant a discussion of whether what is being assessed is what is desired to be assessed and how curricular improvement and assessment reform efforts address content coverage.

The concept of assigning partial credit on multiple-choice test items is considered for items from ACS Exams. Because the items on these exams, particularly the quantitative items, use common student errors to define incorrect answers, it is possible to assign partial credits to some of these incorrect responses. To do so, however, it becomes vital that instructors reach general agreement as to the level of partial credit. Using workshops with instructors, ACS Exams has identified reasons why partial credit could be assigned in an exam set of 70 test items. With partial-credit assignments thus established, polytomous scoring is applied and the effect of such scoring on the overall norm-referenced psychometrics is determined and described. While individual students move within the overall norm, the average influence of polytomous scoring as conceived by the workshops does not substantially change the ability to do norm-based comparisons.

General chemistry tests from the Examinations Institute of the Division of Chemical Education of the American Chemical Society have been analyzed to identify factors that may influence how individual test items perform. In this paper, issues of item order (position within a set of items that comprise a test) and answer order (position of correct answer relative to incorrect distractors) are discussed. Answer order is identified as potentially important, particularly for conceptually based items. When the correct answer appears earlier among the answer choices, there is some greater propensity for student performance to be better. Item-order effects are also possible, particularly when students encounter several challenging items consecutively. Performance on the next item may be lower than expected, possibly because of cognitive-load effects. (Contains 4 tables.)

This paper describes an action research approach toward an implementation of a new teaching methodology (specifically active learning) in a preparatory college chemistry classroom. The initial implementation involved the use of process-orientated guided inquiry (POGIL) activities. The treatment classes with either all POGIL activities during lecture or lectures incorporating POGIL one-third of the time were found to perform lower than the control. For the second implementation, the active learning activities were altered. The authors eliminated many of the reading components and incorporated mini-lectures into the active learning activities. Examining assessment data as a measure of student-content learning, it appears the altered activities are a better match for this preparatory chemistry course. Throughout this process, they focused on the careful analysis of intentionally collected data on student-content learning. This, accompanied by an openness to change direction on the basis of such results, represents a powerful model for teaching change. (Contains 10 tables.)

The design and use of a chemistry practice exam instrument that includes a measure for student mental effort is described in this paper. Use of such an instrument can beneficial to chemistry students and chemistry educators as well as chemical education researchers from both a content and cognitive science perspective. The method for calculating and representing cognitive efficiency in different chemistry categories at both the student-level and classroom-level is described. In addition, explanation of the information available and potential benefits to each of the target populations is discussed with respect to instrument use. (Contains 1 table and 6 figures.)