Search

Psychometricians at an organization named the Education and Training Evaluation Commission (ETEC) developed a new test scoring method called the latent D-scoring method (DSM-L) where it is believed that the new method itself is much easier and more efficient to use compared to the Item Response Theory (IRT) method. However, there are no studies that compared the methods directly regarding item fit and person parameter recovery to understand if the new method (DSM-L) provides better item fit and person parameter recovery compared to the IRT. Therefore, the purpose of this study was to compare the use of latent D-scoring method (DSM-L) models (RFM1 and RFM2) and Item Response Theory (IRT) models (1-PL and 2-PL) with respect to item fit and person parameter recovery across different test data from the ETEC. It was found that both the DSM-L and IRT methods show similar results under both Mean Absolute Error (MAE) and Mean Absolute Difference (MAD) item fit statistics methods across the different test data sets from the ETEC. Also, I found that larger sample sizes and test lengths were associated with improved item fit under the MAE item fit statistics across the different methods. I also found that a two-parameter model type (RFM2 and 2-PL models) shows lower mean MAE item fit statistics across the test data sets compared to a one-parameter model type (RFM1 and 1-PL models).Finally, the DSM-L RFM2 model showed a slightly lower RMSE rate compared to the RFM1 model. Also, both RFM1 and RFM2 models showed acceptable rate of bias (bias equal to zero). On the other hand, the IRT 1-PL model showed a lower RMSE rate compared to the 2-PL model, and 1-PL model showed an acceptable rate of bias (bias approximately equal to zero). With the study outcomes in mind, the psychometricians at the ETEC can use either method (DSM-L or IRT) if the goal is to have good item fit under the MAE item fit statistics since both methods showed similar MAE results. I also recommend using the two-parameter model type based on MAE item fit statistics because it shows lower mean MAE compared to using the one parameter model type. On the other hand, if the goal is to have lower RMSE rates and acceptable rates of bias (bias approximately equal to zero) under IRT I recommend using the 1-PL model. Also, if the goal is to have lower RMSE rate under the DSM-L I recommend using the RFM2 model, and if the goal is to have an acceptable rate of bias under the DSM-L I recommend using either the RFM1 or RFM-2 models. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The purpose of this study was to investigate procedures for assessing model fit of IRT models for mixed format data. In this study, various IRT model combinations were fitted to data containing both dichotomous and polytomous item responses, and the suitability of the chosen model mixtures was evaluated based on a number of model fit procedures. To assess model fit for mixed format data, five fit indices were considered: PARSCALE's G[superscript 2], the generalized forms of Orlando and Thissen's (2000) S-X[superscript 2] and S-G[superscript 2], and Stone's (2000) pseudo-observed score based indices chi[superscript 2*] and G[superscript 2*]. To investigate the relative performance of the five item fit statistics, two simulation studies were conducted: Type I error and power studies. Using the model hierarchy for a set of the data generation models and calibration models, model-fitting and model-misfitting conditions were manipulated to obtain Type I error rates and power. Under the model-fitting conditions, Type I error rates were computed when identical pairs of mixed IRT models were used for data generation and calibration. On the other hand, empirical power was calculated under the model-misfitting conditions when data were calibrated using a simpler model than a generating model. Among the competing measures, the number correct score based indices S-X[superscript 2] and S-G[superscript 2] were found to be the most efficient and safest choice for assessing model fit for mixed format data across the study conditions. These indices performed well particularly with short tests. However, the pseudo-observed score indices, chi[superscript 2*] and G[superscript 2*], showed inflated Type I error rates in some simulation conditions. Thus, their applications appeared to be limited by specific measurement conditions, and need to be studied further for generalized uses. Consistent with the findings of current literature, the PARSCALE's G[superscript 2] index was rarely useful, although it provided reasonable results for long tests. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Students' responses to Word Dictation curriculum-based measurement (CBM) in writing tend to include a lot of missing values, especially items not reached due to the three-minute test time limit. A large amount of non-ignorable not-reached responses in Word Dictation can be considered using alternative item response theory (IRT) approaches. In addition, these alternative approaches could be used to estimate students' writing productivity as well as their accuracy. The purpose of this study was to evaluate the Word Dictation performance of elementary students who are struggling with writing using a classical IRT approach that considers writing accuracy only and alternative IRT approaches that consider both writing accuracy and productivity. This study used data from a larger research project that evaluated the effectiveness of a professional development program designed to support elementary teachers in implementing data-based instruction for students struggling with writing. Participants were recruited in two sites in the Midwest in the U.S. A total of 523 elementary students completed screening tests to determine eligibility for the larger project. Word Dictation CBM in writing, used for screening, was designed to measure transcription skills at the word level by asking students to write dictated words as accurately as they can. I examined the extent to which the students' results differed by comparing the classical IRT approach, latent regression model (LRM) and item response tree (IRTree) model approach. Results from different approaches considering not-reached items (IRTree model and LRM) yielded different ranges of writing ability level, even though students had the same score evaluated by the classical IRT approach. First, goodness-of-fit, item fit, person fit, and model fit were evaluated for each approach to demonstrate that the models have good fit indices. In addition, the results of conditional standardized errors of measurements (cSEM) across models revealed that the alternative IRT approaches evaluated ability more accurately and precisely than classical IRT approaches. Second, considering not-reached responses as either incorrect or missing made a difference in the ability parameters of students with different productivity performance, ultimately underestimating students performing at different levels. In addition, special education eligibility was a significant factor in comparisons of rank-order differences of several models, but not a significant factor in the comparison between classical IRT missing model and alternative IRT models (IRTree model and LRM). Third, most of the results across Word Dictation Forms A and B were consistently replicated. Major inconsistencies in the factor of English Language Learner (ELL) eligibility emerged in examining relations between child-level factors and writing productivity and accuracy and relations between rank-order difference and child-level factors. Handling missing responses is difficult, but entails important procedures that lead to more accurately estimating the ability parameters in the context of CBM. Though no one absolute evaluating approach could be identified, it is possible that abilities of students with specific status including special education eligibility and ELL eligibility were under- or overestimated when missing data were not considered. Better understanding of students' writing performance as it relates to writing productivity and accuracy could ultimately support teachers to use the instructionally meaningful data for individualized instruction. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This study used a convergent, quantitative dominant mixed methods design to investigate the construct of teacher well-being. Three-hundred-forty-five teachers participated in quantitative data collection which included the Teacher Well-being Scale (TWBS; Collie et al., 2016), the Teacher Subjective Well-being Questionnaire (TSWQ; Renshaw et al., 2016), the Workplace PERMA Profiler (Butler & Kern, 2016) and the Brief School Climate Inventory (La Salle et al., 2018). Among these participants, thirteen participated in additional semi-structured interviews. Quantitative data was analyzed using confirmatory factor analysis and item response theory. Qualitative data was analyzed using content analysis. Data were brought together through joint displays and analyzed using a novel process, integrated factor analysis. Quantitative findings suggest that existing teacher well-being scales hold pre-established structural properties and are associated with other, related scales, but item fit varied. Item response theory revealed model-fit challenges using recommended response categories. Model-fit improved for both the TWBS and TSWQ once response categories were reduced. Additionally, some items were flagged for poor fit. Qualitative findings indicated that teacher well-being includes factors at both the individual and environmental level. Specifically, individual dispositions such as having a positive attitude or ability to handle setbacks, supportive leadership, and collaborative school climates emerged as key facilitators of well-being. Job specific stress related to workload and negative relationships emerged as key barriers to well-being. Integration of quantitative and qualitative data was supported by a process of integrated factor analysis. Scale items and qualitative codes were compared for areas of convergence (alignment) and divergence (disagreement). A revised teacher well-being scale is proposed based on integrated findings. Synthesis of quantitative, qualitative, and integrated results suggest the following key findings: (a) teacher well-being is a multidimensional, psycho-social construct that must include both individual and environmental factors; (b) individual dispositions contribute to teachers' experiences and well-being at work; (c) the primary barriers to teacher well-being include job specific stress external to teaching; and (d) revising response categories can lead to stronger use and interpretation of teacher well-being scales. Implications for policy, research, and practice are discussed. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Motivation is a central variable in science education research. The Science Motivation Questionnaire II (SMQ II) is an instrument designed to measure student motivation to learn science. Nearly all peer-reviewed studies (n = 26) of the SMQ II have used Classical Test Theory frameworks despite known limitations with this approach. The present study seeks to evaluate the psychometric properties of the SMQ II using a Rasch Analysis framework. We explored three questions: (1) How well do the scales and items on the SMQ II function when analyzed in the context of the Rasch model? (2) Does time of the school year impact the psychometric properties of the instrument? And (3) How do the five motivational constructs differ by gender and grade level? We administered the SMQ II to a sample of students (n = 1,728) at two points in time. In order to evaluate instrument dimensionality, a Principle Component Analysis of Rasch residuals was performed. In addition, AIC/BIC values were generated and compared across different dimensional arrangements. Person reliability, item reliability, and item fit values were calculated for each dimension. Wright maps were also generated in order to compare person-item alignments. Finally, dataset stacking was used to align scores from the June and January samples and compare motivation patterns across gender and grade band for each of the five dimensions. Results of the dimensionality tests indicated that a five-dimensional model was better supported than a unidimensional model. Item and person reliabilities were acceptable for all dimensions. Infit and outfit MNSQ statistics for items revealed acceptable values in nearly all cases. Wright Maps for the five dimensions indicated a lack of items at the upper end of the logit scales. Rasch score differences across time, gender, and grade level were apparent, most notably for grade nine. For the Self-Determination and Career Motivation dimensions, grade nine participants had lower measures compared to the other grades. The Grade Motivation dimension had the highest mean scores, and the Career Motivation dimension had the lowest mean scores. These findings advance our understanding of the strengths and weaknesses of a core measure of student motivation to learn science. Overall, the SMQ II instrument functions adequately. However, several measurement problems were documented that must be addressed before using this instrument in comparable participant populations. Multiple analyses (e.g., separation values, Wright maps) revealed a lack of measurement precision for this participant population. Moreover, a few items are problematic must be modified or eliminated. Collectively, these findings suggest that the SMQ II will produce adequate but error-prone measures of student motivation. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Genetics is universally recognized as a core aspect of biological and scientific literacy. Beyond genetics' own role as a major unifying topic within the biological sciences, understanding genetics is essential for understanding other integral ideas such as evolution and development. Genetics understanding also underlies public decision making about modern advances in health sciences and biotechnology and broader socio-scientific issues. Consequently, educators have attempted to measure student and teacher understanding of this domain. Using Rasch modeling--a superior but underutilized framework for instrument evaluation--this dissertation explored psychometric, cognitive, and demographic aspects of educational measurement in the field of genetics education in order to generate evidence-based examples to illustrate how instruments can be more carefully developed and robustly evaluated. The first study (Chapter 3) sought to expand the sources of evidence supporting validity and reliability inferences produced by a relatively new concept inventory (the Genetic Drift Inventory [GeDI]) designed for use in diagnosing undergraduate students' conceptual understanding of genetic drift. Concept Inventories (CIs) are commonly used tools for assessing student understanding of normative (scientific) and non-normative (naive) ideas, yet the body of empirical evidence supporting the inferences drawn from CI scores is often limited in scope and remains deeply rooted in Classical Test Theory (CTT) despite the availability of more robust Item Response Theory (IRT) and Rasch frameworks. Specifically, this study focused on: (1) GeDI instrument and item properties as revealed by Rasch modeling, (2) item order effects on response patterns, and (3) generalization to a new geographic sample. A sample of 336 advanced undergraduate biology majors completed one of four randomly assigned and equivalent versions of the GeDI that differed in presentation order of the GeDI item suites. Rasch analysis indicated the GeDI was unidimensional, with good fit to the Rasch model. Items had high reliability and were well matched to the ability of the sample. Person reliability was low. Rotating the GeDI's vignette-based item suites had no significant impact on overall scores, suggesting each vignette functioned independently. Scores from this new sample from the Northeast United States were comparable to those from other geographic regions and provide evidence in support of score generalizability. Suggestions for improvement include: (1) incorporation of additional items to differentiate high-ability persons and improve person reliability, and (2) re-examination of items with redundant or low difficulty levels. These results expand the range and quality of evidence in support of validity claims and illustrate changes that are likely to improve the quality of the GeDI (and other) evolution education instruments. The second study (Chapter 4) sought to determine how situational features impact inferences about participants' understanding of Mendelian genetics. Understanding how situational features of assessment tasks impact reasoning is important for many educational pursuits, notably the selection of curricular examples to illustrate phenomena, the design of formative and summative assessment items, and determination of whether instruction has fostered the development of abstract schemas divorced from particular instances. To test for context effects, an experimental research design was employed to measure differences in item difficulty among items varying only in situational features (e.g., plant, animal, human, fictitious) across five common genetics problem types. A multi-matrix test design was employed, and item packets were randomly distributed to a sample of undergraduate biology majors (n=444). Rasch analyses of participant scores produced good item fit, person reliability, and item reliability. Surprisingly, no significant differences in performance occurred among the animal, plant, and human item contexts, or between the fictitious and "real" item contexts. Also notable, incomplete dominance problems proved to be more difficult than dominant-recessive problems, and problems featuring homozygous parents were more difficult than those featuring heterozygous parents. Tests for differences in performance between genders, among ethnic groups, and by prior biology coursework revealed that none of these factors had a meaningful impact upon performance or context effects. Thus some, but not all, types of genetics problem solving or item formats are impacted by situational features. Overall, substantial evidence was generated about how current knowledge in the field of genetics education is measured and how measurement in this domain may be improved. The studies included herein exemplify some ways in which new and existing instruments can be examined to amass robust evidence for the quality of inferences generated by an instrument. Only with rigorously evaluated instruments can the educational community be confident that inferences about student learning are accurate and that consequent decisions are evidence-based. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The common practice for testing measurement invariance is to constrain parameters to be equal over groups, and then evaluate the model-data fit to reject or fail to reject the restrictive model. Posterior predictive checking (PPC) provides an alternative approach to evaluating model-data discrepancy. This paper explores the utility of PPC in estimating measurement invariance. The simulation results show that the posterior predictive p (PP p) values of item parameter estimates respond to various invariance violations, whereas the PP p values of item-fit index may fail to detect such violations. The current paper suggests comparing group estimates and restrictive model estimates with posterior predictive distributions in order to demonstrate the pattern of misfit graphically. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

For test scores that rely on the accurate estimation of ability via an IRT model, their use and interpretation is dependent upon the assumption that the IRT model fits the data. Examinees who do not put forth full effort in answering test questions, have prior knowledge of test content, or do not approach a test with the intent of answering questions to the best of their ability are exhibiting aberrant response behaviors and the accuracy and validity of the resulting test scores are called into question. The test administrator is left with the problem of determining whether test scores are a true representation of examinee ability (Reise, 1990; Karabatsos, 2003). Model fit is typically assessed through item-fit indices. An equally important aspect of assessing model fit is determining how well an IRT model fits the response patterns of examinees, which is commonly referred to as person fit (Meijer & Sijtsma, 2001). The purpose of this research was to explore the application of person-fit analysis in the identification of cheating behavior. Specifically, issues that may impact the effectiveness of person-fit indices, also called person-fit measures, were evaluated. A primary focus of this research was the value of using multiple types of measures (scalar, response time, graphical), both individually and combined, in determining whether or not a response pattern is indicative of cheating behavior. A review of the literature on person-fit research is presented, followed by a discussion of considerations for designing a person-fit simulation study. A study was then conducted to determine the effectiveness of three person-fit measures in identifying simulated cheating behavior under various conditions. The person-fit measures used in the study were "l[subscript z]" (Drasgow, Levine & Williams, 1985), Effective Response Time (Meijer & Sotaridona, 2006), and the Person Response Curve (Trabin & Weiss, 1983). The effectiveness of the individual measures and the measures used in combination was evaluated. Study factors included IRT model, exam length, examinee ability level, amount of aberrance within an exam, and amount of aberrance within a sample or population. A real-parameter simulation study (Seo & Weiss, 2013) was conducted using Rasch and two-parameter logistic (2PL) IRT parameters estimated from a large dataset obtained from a language skills assessment. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Investigating the psychometric properties of a screening instrument for young children is necessary to ascertain its quality and accuracy. In light of the important role culture plays on human beliefs and parenting styles, a newly translated and adapted test needs to be studied. Evaluating outcomes on a translated version of a test may reveal significant information related to cultural specifications as well as the common nature of child development. The current study examined psychometric properties of the 48-month interval of the Ages and Stages Questionnaires: Social-Emotional Second Edition (ASQ:SE-2) and its Traditional Chinese version (ASQ:SE-TC), using item response theory (IRT). Participants in the U.S. included 3,005 young children/parents dyads; 1,455 dyads were collected to represent a Taiwanese sample. A two-dimensional Rasch Partial Credit Model (2D-RPCM), which was determined to present a better fit than a unidimensional Rasch Partial Credit Model, was used to examine the item fit, item difficulty, reliability, and item information curves to evaluate the psychometric properties on the ASQ:SE and ASQ:SE-TC. Further, differential item functioning was conducted to examine whether items were functioning differently in the two population groups. Lastly, the differences between the distributions of children's latent traits on the continuum of social and emotional competencies for the U.S. and Taiwanese samples were investigated. Based on findings, the adequacy of psychometric properties is discussed, providing insight into the quality of particular items. Identified differences between the two populations are explored by reviewing literature regarding cultural comparisons of child-rearing practices, parenting styles, and cultural beliefs. Future directions for research include examining the cultural equivalence between translated and original versions of other ASQ:SE-2 intervals. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The purpose of this study was to construct a holistic education school evaluation tool using Montessori Erdkinder principles, and begin the validation process of examining the proposed tool. This study addresses a vital need in the holistic education community for a school evaluation tool. The tool construction process included using Erdkinder literature to justify the development of each item through the use of an item matrix, ultimately leading to the development of the 23 item formative Montessori Erdkinder School Evaluation Survey. The validation process included a series of three Rasch Rating Scale Model analyses with data from a sample school. The validation process used item anchoring estimates from the earlier analyses in the later analyses and included determining the tool's dimensionality, reliability, item fit, possible differential item functioning, and comparing the order of item difficulty levels to the holistic model of Maslow's hierarchy of needs. Results of the study showed that six items had issues with fit and would need to be revised, and that the items in the cognitive and moral facet will need to be revised to better match Maslow's model. This study provides the foundation for the development of a holistic education evaluation or accreditation system, and constructed a resource that could be directly implemented in schools. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This simulation study compared the utility of various discrepancy measures within a posterior predictive model checking (PPMC) framework for detecting different types of data-model misfit in multidimensional Bayesian network (BN) models. The investigated conditions were motivated by an applied research program utilizing an operational complex performance assessment within a digital-simulation educational context grounded in theories of cognition and learning. BN models were manipulated along two factors: latent variable dependency structure and number of latent classes. Distributions of posterior predicted p-values (PPP-values) served as the primary outcome measure and were summarized in graphical presentations, by median values across replications, and by proportions of replications in which the PPP-values were extreme. An effect size measure for PPMC was introduced as a supplemental numerical summary to the PPP-value. Consistent with previous PPMC research, all investigated fit functions tended to perform conservatively, but Standardized Generalized Dimensionality Discrepancy Measure (SGDDM), Yen's Q3, and Hierarchy Consistency Index (HCI) only mildly so. Adequate power to detect at least some types of misfit was demonstrated by SGDDM, Q3, HCI, Item Consistency Index (ICI), and to a lesser extent Deviance, while proportion correct (PC), a chi-square-type item-fit measure, Ranked Probability Score (RPS), and Good's Logarithmic Scale (GLS) were powerless across all investigated factors. Bivariate SGDDM and Q3 were found to provide powerful and detailed feedback for all investigated types of misfit. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This study was an investigation of public elementary school students' affective changes through English Activities of international understanding lessons at Japanese public elementary school. The learners' expected affective changes were regarded as "International Posture", "Self-esteem", "Collective Self-esteem", and "Interest in Foreign Affairs" based on theoretical perspectives and the goals specified by Ministry of Education. International Understanding Questionnaire for Children (IUQC), a written instrument to measure elementary school students' "International Posture", "Self-esteem", "Collective Self-esteem", and "Interest in Foreign Affairs" was developed for the study. After the IUQC was validated in terms of its dimensionality, rating scale functioning, and item fit using factor analyses, and Rasch-based analyses, it was administered to 105 public elementary school students (61 fifth graders and 44 sixth graders) at the beginning and end of international understanding lessons in 2006. Results of the two-way mixed factorial ANOVA analyses (trial x grade) indicated that the elementary school students in this study increased their "International Posture" to a statistically significant degree after taking the international understanding lessons for one academic year. The more experienced sixth grade learners performed better on the "International Posture" items on the IUQC than the less experienced fifth graders. The sixth grade participants' "Self-esteem" did not change, while the fifth grade participants' Self-esteem decreased. All participants' "Collective Self-esteem" was not influenced by the lessons. Their "Interest in Foreign Affairs" remained the same both in the beginning and end of the lessons. The influence of "classes" on the four affective constructs was not shown by the two-way mixed factorial ANOVA analyses (trial x class). However, the qualitative analyses indicated that the elementary school students whose teacher actively supported the international understanding lessons showed a number of positive effects in the classroom. This study contributes to affective research in the field of early English education at Japanese public schools. The results of the study partially confirmed the value of English Activities in the international understanding lessons. Further, it demonstrates the effective use of factor analyses and the Rasch-based model to develop a reliable instrument to measure elementary school students' affective constructs. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]