Search

Psychometricians at an organization named the Education and Training Evaluation Commission (ETEC) developed a new test scoring method called the latent D-scoring method (DSM-L) where it is believed that the new method itself is much easier and more efficient to use compared to the Item Response Theory (IRT) method. However, there are no studies that compared the methods directly regarding item fit and person parameter recovery to understand if the new method (DSM-L) provides better item fit and person parameter recovery compared to the IRT. Therefore, the purpose of this study was to compare the use of latent D-scoring method (DSM-L) models (RFM1 and RFM2) and Item Response Theory (IRT) models (1-PL and 2-PL) with respect to item fit and person parameter recovery across different test data from the ETEC. It was found that both the DSM-L and IRT methods show similar results under both Mean Absolute Error (MAE) and Mean Absolute Difference (MAD) item fit statistics methods across the different test data sets from the ETEC. Also, I found that larger sample sizes and test lengths were associated with improved item fit under the MAE item fit statistics across the different methods. I also found that a two-parameter model type (RFM2 and 2-PL models) shows lower mean MAE item fit statistics across the test data sets compared to a one-parameter model type (RFM1 and 1-PL models).Finally, the DSM-L RFM2 model showed a slightly lower RMSE rate compared to the RFM1 model. Also, both RFM1 and RFM2 models showed acceptable rate of bias (bias equal to zero). On the other hand, the IRT 1-PL model showed a lower RMSE rate compared to the 2-PL model, and 1-PL model showed an acceptable rate of bias (bias approximately equal to zero). With the study outcomes in mind, the psychometricians at the ETEC can use either method (DSM-L or IRT) if the goal is to have good item fit under the MAE item fit statistics since both methods showed similar MAE results. I also recommend using the two-parameter model type based on MAE item fit statistics because it shows lower mean MAE compared to using the one parameter model type. On the other hand, if the goal is to have lower RMSE rates and acceptable rates of bias (bias approximately equal to zero) under IRT I recommend using the 1-PL model. Also, if the goal is to have lower RMSE rate under the DSM-L I recommend using the RFM2 model, and if the goal is to have an acceptable rate of bias under the DSM-L I recommend using either the RFM1 or RFM-2 models. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Motivation is a central variable in science education research. The Science Motivation Questionnaire II (SMQ II) is an instrument designed to measure student motivation to learn science. Nearly all peer-reviewed studies (n = 26) of the SMQ II have used Classical Test Theory frameworks despite known limitations with this approach. The present study seeks to evaluate the psychometric properties of the SMQ II using a Rasch Analysis framework. We explored three questions: (1) How well do the scales and items on the SMQ II function when analyzed in the context of the Rasch model? (2) Does time of the school year impact the psychometric properties of the instrument? And (3) How do the five motivational constructs differ by gender and grade level? We administered the SMQ II to a sample of students (n = 1,728) at two points in time. In order to evaluate instrument dimensionality, a Principle Component Analysis of Rasch residuals was performed. In addition, AIC/BIC values were generated and compared across different dimensional arrangements. Person reliability, item reliability, and item fit values were calculated for each dimension. Wright maps were also generated in order to compare person-item alignments. Finally, dataset stacking was used to align scores from the June and January samples and compare motivation patterns across gender and grade band for each of the five dimensions. Results of the dimensionality tests indicated that a five-dimensional model was better supported than a unidimensional model. Item and person reliabilities were acceptable for all dimensions. Infit and outfit MNSQ statistics for items revealed acceptable values in nearly all cases. Wright Maps for the five dimensions indicated a lack of items at the upper end of the logit scales. Rasch score differences across time, gender, and grade level were apparent, most notably for grade nine. For the Self-Determination and Career Motivation dimensions, grade nine participants had lower measures compared to the other grades. The Grade Motivation dimension had the highest mean scores, and the Career Motivation dimension had the lowest mean scores. These findings advance our understanding of the strengths and weaknesses of a core measure of student motivation to learn science. Overall, the SMQ II instrument functions adequately. However, several measurement problems were documented that must be addressed before using this instrument in comparable participant populations. Multiple analyses (e.g., separation values, Wright maps) revealed a lack of measurement precision for this participant population. Moreover, a few items are problematic must be modified or eliminated. Collectively, these findings suggest that the SMQ II will produce adequate but error-prone measures of student motivation. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Genetics is universally recognized as a core aspect of biological and scientific literacy. Beyond genetics' own role as a major unifying topic within the biological sciences, understanding genetics is essential for understanding other integral ideas such as evolution and development. Genetics understanding also underlies public decision making about modern advances in health sciences and biotechnology and broader socio-scientific issues. Consequently, educators have attempted to measure student and teacher understanding of this domain. Using Rasch modeling--a superior but underutilized framework for instrument evaluation--this dissertation explored psychometric, cognitive, and demographic aspects of educational measurement in the field of genetics education in order to generate evidence-based examples to illustrate how instruments can be more carefully developed and robustly evaluated. The first study (Chapter 3) sought to expand the sources of evidence supporting validity and reliability inferences produced by a relatively new concept inventory (the Genetic Drift Inventory [GeDI]) designed for use in diagnosing undergraduate students' conceptual understanding of genetic drift. Concept Inventories (CIs) are commonly used tools for assessing student understanding of normative (scientific) and non-normative (naive) ideas, yet the body of empirical evidence supporting the inferences drawn from CI scores is often limited in scope and remains deeply rooted in Classical Test Theory (CTT) despite the availability of more robust Item Response Theory (IRT) and Rasch frameworks. Specifically, this study focused on: (1) GeDI instrument and item properties as revealed by Rasch modeling, (2) item order effects on response patterns, and (3) generalization to a new geographic sample. A sample of 336 advanced undergraduate biology majors completed one of four randomly assigned and equivalent versions of the GeDI that differed in presentation order of the GeDI item suites. Rasch analysis indicated the GeDI was unidimensional, with good fit to the Rasch model. Items had high reliability and were well matched to the ability of the sample. Person reliability was low. Rotating the GeDI's vignette-based item suites had no significant impact on overall scores, suggesting each vignette functioned independently. Scores from this new sample from the Northeast United States were comparable to those from other geographic regions and provide evidence in support of score generalizability. Suggestions for improvement include: (1) incorporation of additional items to differentiate high-ability persons and improve person reliability, and (2) re-examination of items with redundant or low difficulty levels. These results expand the range and quality of evidence in support of validity claims and illustrate changes that are likely to improve the quality of the GeDI (and other) evolution education instruments. The second study (Chapter 4) sought to determine how situational features impact inferences about participants' understanding of Mendelian genetics. Understanding how situational features of assessment tasks impact reasoning is important for many educational pursuits, notably the selection of curricular examples to illustrate phenomena, the design of formative and summative assessment items, and determination of whether instruction has fostered the development of abstract schemas divorced from particular instances. To test for context effects, an experimental research design was employed to measure differences in item difficulty among items varying only in situational features (e.g., plant, animal, human, fictitious) across five common genetics problem types. A multi-matrix test design was employed, and item packets were randomly distributed to a sample of undergraduate biology majors (n=444). Rasch analyses of participant scores produced good item fit, person reliability, and item reliability. Surprisingly, no significant differences in performance occurred among the animal, plant, and human item contexts, or between the fictitious and "real" item contexts. Also notable, incomplete dominance problems proved to be more difficult than dominant-recessive problems, and problems featuring homozygous parents were more difficult than those featuring heterozygous parents. Tests for differences in performance between genders, among ethnic groups, and by prior biology coursework revealed that none of these factors had a meaningful impact upon performance or context effects. Thus some, but not all, types of genetics problem solving or item formats are impacted by situational features. Overall, substantial evidence was generated about how current knowledge in the field of genetics education is measured and how measurement in this domain may be improved. The studies included herein exemplify some ways in which new and existing instruments can be examined to amass robust evidence for the quality of inferences generated by an instrument. Only with rigorously evaluated instruments can the educational community be confident that inferences about student learning are accurate and that consequent decisions are evidence-based. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Investigating the psychometric properties of a screening instrument for young children is necessary to ascertain its quality and accuracy. In light of the important role culture plays on human beliefs and parenting styles, a newly translated and adapted test needs to be studied. Evaluating outcomes on a translated version of a test may reveal significant information related to cultural specifications as well as the common nature of child development. The current study examined psychometric properties of the 48-month interval of the Ages and Stages Questionnaires: Social-Emotional Second Edition (ASQ:SE-2) and its Traditional Chinese version (ASQ:SE-TC), using item response theory (IRT). Participants in the U.S. included 3,005 young children/parents dyads; 1,455 dyads were collected to represent a Taiwanese sample. A two-dimensional Rasch Partial Credit Model (2D-RPCM), which was determined to present a better fit than a unidimensional Rasch Partial Credit Model, was used to examine the item fit, item difficulty, reliability, and item information curves to evaluate the psychometric properties on the ASQ:SE and ASQ:SE-TC. Further, differential item functioning was conducted to examine whether items were functioning differently in the two population groups. Lastly, the differences between the distributions of children's latent traits on the continuum of social and emotional competencies for the U.S. and Taiwanese samples were investigated. Based on findings, the adequacy of psychometric properties is discussed, providing insight into the quality of particular items. Identified differences between the two populations are explored by reviewing literature regarding cultural comparisons of child-rearing practices, parenting styles, and cultural beliefs. Future directions for research include examining the cultural equivalence between translated and original versions of other ASQ:SE-2 intervals. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]