Search

When measuring latent constructs, for example, language ability, we use statistical models to specify appropriate relationships between the latent construct and observe responses to test items. These models rely on theoretical assumptions to ensure accurate parameter estimates for valid inferences based on the test results. This dissertation specifically concerned the local independence (LI) assumption (or presence of local dependence (LD) equivalently) in two specific item response theory (IRT) models -- the partial credit model (PCM) and the generalized partial credit model (GPCM). With a primary focus on several commonly used global fit indices (GFIs), this study investigated how parameter estimates in the studied IRT models were affected by varying degrees of LD, whether the studied GFIs were responsive to model misspecifications due to LD, and any insights regarding making reasonable decisions in terms of good-fit models. The results revealed that, first, parameter estimates in both the PCM and GPCM were severely biased, especially the latent trait and item step parameter estimates. The degree of biases increased with higher levels of LD across different model and data conditions. Other general factors, including the test length and number of item response categories, could also impact biases in parameter estimates. Second, all studied GFIs were responsive to model misfit caused by the presence of LD. Specifically, studied GFIs exhibited moderate to strong associations with biases of the latent trait estimates in expected directions. Lastly, based on the performance of the studied GFIs under baseline and LD-induced conditions, the absolute fit indices (RMSEA and SRMSR) appeared to be more informative than the incremental fit indices (CFI and TLI) in assessing the global fit of the PCM and GPCM. This study provides insights into adopting specific decision criteria of good model fit under various model and data conditions, aiming to reduce potential false positive rates while maintaining reasonable true positive rates. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Social and emotional learning (SEL) has become essential in educational settings and human development. Federal policy has begun to incorporate SEL factors into education accountability metrics. Schools, institutes, and scholars have started implementing SEL in K-12 educational contexts. Social interactions and emotions are a daily part of students' lives, both in and out of school, which is why many educators are willing to teach and measure SEL competencies. Moreover, in previous research, it has been found that students' higher SEL competencies are positively associated with higher academic performance and lower risky behaviors. As interest in SEL grows, measurement tools continue to evolve. An interesting finding from the overview of existing SEL measures was that not only a general SEL scale is measured but in practice, multiple dimensions, or subscales of SEL. This finding implies the multidimensional nature of SEL measures. How- ever, researchers have not attempted to use the multidimensional item response theory to identify the multidimensionality of SEL measures and evaluate their psychometric properties. This dissertation aimed to apply MIRT models to the MSS-SEL data, identify the best-fitting model, and conduct a psychometric evaluation of the MSS-SEL items and dimensions. To mediate person-score calibration and evaluation, psychometric properties such as internal structure and item parameter estimates of the MSS-SEL items were evaluated under MIRT framework. The results showed that among the competing correlated models, MGRM fitted better than the MG- PCM to the MSS-SEL data. Additionally, among the competing bifactor models, bifactor-GRM fitted better than the bifactor-GPCM. The results indicated that for both the correlated MIRT model and bifactor model applications, the graded response modeling approach functioned better for the MSS-SEL response data than the generalized partial credit modeling approach. The results of the MGRM application showed that some MSS-SEL factors were relatively more highly correlated than others. One of the bifactor-GRM results showed that the general factor (i.e., SEL) accounted for by 66% of the common variance among items. Each specific factor was accounted almost equally well for the rest of the variance. Overall, the 37 SEL items reflected the general factor reasonably well, and most of the items were moderately/strongly explained by the general factor. This dissertation demonstrated what could have been missed if just a single model (i.e., UIRT, correlated MIRT model, or bifactor model) was fitted to the data. Furthermore, validity evidence was provided for the potential use and interpretation of the MSS-SEL scores. Several implications can be made especially for researchers and practitioners who frequently get involved in all different phases of SEL measurement. First, it is important to plan ahead for an item and scale development and to search for theoretically suitable IRT models. Second, it is crucial to examine the internal structure and use MIRT models when there are multiple SEL constructs to be measured. Third, the types of scores that are meaningful for the report and to be used should be clearly declared by the SEL measurement developers and users. Psychometric evaluation of the existing SEL measures using relevant measurement models will contribute to the validation of the SEL measures in support of the intended interpretation and use of such measures by educators and practitioners. This, of course, is a core component of validity evidence to interpret and use SEL scores at multiple levels. Moreover, this is a part of a larger effort to understand and articulate a set of psychometric principles for SEL measurement. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This dissertation informed researchers about the performance of different level-specific and target-specific model fit indices in Multilevel Latent Growth Model (MLGM) using unbalanced design and different trajectories. As the use of MLGMs is a relatively new field, this study helped further the field by informing researchers interested in using various specific model fit indices. We evaluated the descriptive information of the various model fit indices under various simulation conditions and the extent to which the fit indices could be influenced by different design factors, based on simulated data with different conditions derived from a correctly specified MLGM. Our simulation design factors included three levels of number of groups (50, 100, and 200), three levels of unbalanced GS (5/15, 10/20, and 25/75), and three trajectories (accelerating, decelerating, and linear). Based on the results, we made recommendations for practical and theoretical research about fit indices. CFI- and TFI-related fit indices performed well in the MLGM could be trustworthy to use to evaluate model fit under similar conditions found in applied settings. However, RMSEA-related fit indices, SRMR-related fit indices, and chi-square-related fit indices varied by the factors included in this study and should be used with caution for evaluating model fit in the MLGM. The use of these fit indices appears to be particularly problematic when dealing with unbalanced design and small sample sizes. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Psychometricians at an organization named the Education and Training Evaluation Commission (ETEC) developed a new test scoring method called the latent D-scoring method (DSM-L) where it is believed that the new method itself is much easier and more efficient to use compared to the Item Response Theory (IRT) method. However, there are no studies that compared the methods directly regarding item fit and person parameter recovery to understand if the new method (DSM-L) provides better item fit and person parameter recovery compared to the IRT. Therefore, the purpose of this study was to compare the use of latent D-scoring method (DSM-L) models (RFM1 and RFM2) and Item Response Theory (IRT) models (1-PL and 2-PL) with respect to item fit and person parameter recovery across different test data from the ETEC. It was found that both the DSM-L and IRT methods show similar results under both Mean Absolute Error (MAE) and Mean Absolute Difference (MAD) item fit statistics methods across the different test data sets from the ETEC. Also, I found that larger sample sizes and test lengths were associated with improved item fit under the MAE item fit statistics across the different methods. I also found that a two-parameter model type (RFM2 and 2-PL models) shows lower mean MAE item fit statistics across the test data sets compared to a one-parameter model type (RFM1 and 1-PL models).Finally, the DSM-L RFM2 model showed a slightly lower RMSE rate compared to the RFM1 model. Also, both RFM1 and RFM2 models showed acceptable rate of bias (bias equal to zero). On the other hand, the IRT 1-PL model showed a lower RMSE rate compared to the 2-PL model, and 1-PL model showed an acceptable rate of bias (bias approximately equal to zero). With the study outcomes in mind, the psychometricians at the ETEC can use either method (DSM-L or IRT) if the goal is to have good item fit under the MAE item fit statistics since both methods showed similar MAE results. I also recommend using the two-parameter model type based on MAE item fit statistics because it shows lower mean MAE compared to using the one parameter model type. On the other hand, if the goal is to have lower RMSE rates and acceptable rates of bias (bias approximately equal to zero) under IRT I recommend using the 1-PL model. Also, if the goal is to have lower RMSE rate under the DSM-L I recommend using the RFM2 model, and if the goal is to have an acceptable rate of bias under the DSM-L I recommend using either the RFM1 or RFM-2 models. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This study used a convergent, quantitative dominant mixed methods design to investigate the construct of teacher well-being. Three-hundred-forty-five teachers participated in quantitative data collection which included the Teacher Well-being Scale (TWBS; Collie et al., 2016), the Teacher Subjective Well-being Questionnaire (TSWQ; Renshaw et al., 2016), the Workplace PERMA Profiler (Butler & Kern, 2016) and the Brief School Climate Inventory (La Salle et al., 2018). Among these participants, thirteen participated in additional semi-structured interviews. Quantitative data was analyzed using confirmatory factor analysis and item response theory. Qualitative data was analyzed using content analysis. Data were brought together through joint displays and analyzed using a novel process, integrated factor analysis. Quantitative findings suggest that existing teacher well-being scales hold pre-established structural properties and are associated with other, related scales, but item fit varied. Item response theory revealed model-fit challenges using recommended response categories. Model-fit improved for both the TWBS and TSWQ once response categories were reduced. Additionally, some items were flagged for poor fit. Qualitative findings indicated that teacher well-being includes factors at both the individual and environmental level. Specifically, individual dispositions such as having a positive attitude or ability to handle setbacks, supportive leadership, and collaborative school climates emerged as key facilitators of well-being. Job specific stress related to workload and negative relationships emerged as key barriers to well-being. Integration of quantitative and qualitative data was supported by a process of integrated factor analysis. Scale items and qualitative codes were compared for areas of convergence (alignment) and divergence (disagreement). A revised teacher well-being scale is proposed based on integrated findings. Synthesis of quantitative, qualitative, and integrated results suggest the following key findings: (a) teacher well-being is a multidimensional, psycho-social construct that must include both individual and environmental factors; (b) individual dispositions contribute to teachers' experiences and well-being at work; (c) the primary barriers to teacher well-being include job specific stress external to teaching; and (d) revising response categories can lead to stronger use and interpretation of teacher well-being scales. Implications for policy, research, and practice are discussed. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Many popular global model-data fit indices (GFIs), such as Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residuals (SRMSR) are proposed and widely used in the context of structural equation modeling (SEM) with continuous data. The proposed cutoffs of those indices are for continuous data, which are not directly applicable to categorical data (Xu et al., 2017; Xia, 2016; Maydeu-Olivares & Joe, 2014; Yu, 2002). Furthermore, the existing GFIs are developed and have been used with a limited information estimation approach, while a full information estimation approach is the most widely used estimation method in item response theory (IRT). Not many GFIs are available for categorical data when the full information IRT estimation is conducted (Maydeu-Olivares & Joe, 2014). When there is an effect of guessing, as far as previous literature was searched, there is no GFI that demonstrates good performance in detecting the model-data fit for the comparative evaluation of the two-parameter logistic (2PL) and the three-parameter logistic (3PL) models. In addition, there is a lack of studies that conduct a systematic full evaluation of those popular GFIs in the unidimensional IRT (UIRT) and multidimensional IRT (MIRT) modeling (Maydeu-Olivares & Joe, 2014; Xu et al., 2017). This study proposes descriptive GFIs using squared or absolute (raw or standardized) item response residuals from the whole item response data when an IRT model is estimated with the full information estimation. This study also compares the performances of the newly proposed GFIs with the existing GFIs in UIRT and MIRT modeling, especially the sensitivity of the indices in detecting the 2PL and the 3PL model fitting when data are contaminated with examinee guessing. As a secondary interest, Akaike's Information Criterion (AIC), Bayesian Information Criterion (BIC), and M2 test are also included in the comparisons. The newly proposed RI.R2s, residual variation reduction index based on the absolute residual summary for standardized residuals, showed the best performance in distinguishing the 3PL and the 2PL model fitting in unidimensional and multidimensional data, which contained test-taker guessing. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The Next Generation Science Standards propose a multidimensional model of science learning, comprised of Core Disciplinary Ideas, Science and Engineering Practices, and Crosscutting Concepts (NGSS Lead States, 2013). Accordingly, there is a need for student assessment aligned with the new standards. Creating assessments that validly and reliably measure multidimensional science ability is a challenge for the measurement community (Pellegrino, et al., 2014). Multidimensional assessment tasks may need to go beyond typical item designs of standalone multiple-choice and short-answer items. Furthermore, scoring and modeling of student performance should account for the multidimensionality of the construct. This research contributes to knowledge about best practices for multidimensional science assessment by exploring three areas of interest: 1) item design, 2) scoring rubrics, and 3) measurement models. This study investigated multidimensional scaffolding and response format by comparing alternative item designs on an elementary assessment of matter. Item variations had a different number of item prompts and/or response formats. Observations about student cognition and performance were collected during cognitive interviews and a pilot test. Items were scored using a holistic rubric and a multidimensional rubric, and interrater agreement was examined. Assessment data was scaled with multidimensional scores and holistic scores, using unidimensional and multidimensional Rasch models, and model-data fit was compared. Results showed that scaffolding is associated with more thorough responses, especially among low ability students. Students tended to utilize different cognitive processes to respond to selected-response items and constructed-response items, and were more likely to respond to selected-response arguments. Interrater agreement was highest when the structure of the item aligned with the structure of the scoring rubric. Holistic scores provided similar reliability and precision as multidimensional scores, but item and person fit was poorer. Multidimensional subscales had lower reliability, less precise student estimates than the unidimensional model, and interdimensional correlations were high. However, the multidimensional rubric and model provide nuanced information about student performance and better fit to the response data. Recommendations about optimal combinations of scaffolding, rubric, and measurement models are made for teachers, policymakers, and researchers. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The common practice for testing measurement invariance is to constrain parameters to be equal over groups, and then evaluate the model-data fit to reject or fail to reject the restrictive model. Posterior predictive checking (PPC) provides an alternative approach to evaluating model-data discrepancy. This paper explores the utility of PPC in estimating measurement invariance. The simulation results show that the posterior predictive p (PP p) values of item parameter estimates respond to various invariance violations, whereas the PP p values of item-fit index may fail to detect such violations. The current paper suggests comparing group estimates and restrictive model estimates with posterior predictive distributions in order to demonstrate the pattern of misfit graphically. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

For test scores that rely on the accurate estimation of ability via an IRT model, their use and interpretation is dependent upon the assumption that the IRT model fits the data. Examinees who do not put forth full effort in answering test questions, have prior knowledge of test content, or do not approach a test with the intent of answering questions to the best of their ability are exhibiting aberrant response behaviors and the accuracy and validity of the resulting test scores are called into question. The test administrator is left with the problem of determining whether test scores are a true representation of examinee ability (Reise, 1990; Karabatsos, 2003). Model fit is typically assessed through item-fit indices. An equally important aspect of assessing model fit is determining how well an IRT model fits the response patterns of examinees, which is commonly referred to as person fit (Meijer & Sijtsma, 2001). The purpose of this research was to explore the application of person-fit analysis in the identification of cheating behavior. Specifically, issues that may impact the effectiveness of person-fit indices, also called person-fit measures, were evaluated. A primary focus of this research was the value of using multiple types of measures (scalar, response time, graphical), both individually and combined, in determining whether or not a response pattern is indicative of cheating behavior. A review of the literature on person-fit research is presented, followed by a discussion of considerations for designing a person-fit simulation study. A study was then conducted to determine the effectiveness of three person-fit measures in identifying simulated cheating behavior under various conditions. The person-fit measures used in the study were "l[subscript z]" (Drasgow, Levine & Williams, 1985), Effective Response Time (Meijer & Sotaridona, 2006), and the Person Response Curve (Trabin & Weiss, 1983). The effectiveness of the individual measures and the measures used in combination was evaluated. Study factors included IRT model, exam length, examinee ability level, amount of aberrance within an exam, and amount of aberrance within a sample or population. A real-parameter simulation study (Seo & Weiss, 2013) was conducted using Rasch and two-parameter logistic (2PL) IRT parameters estimated from a large dataset obtained from a language skills assessment. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

More robust and rigorous psychometric models, such as multidimensional Item Response Theory models, have been advocated for survey applications. However, item responses may be influenced by construct-irrelevant variance factors such as preferences for extreme response options. Through empirical and simulation methods, this study evaluates the use of the IRTree Model, the multidimensional nominal response model, and the modified generalized partial credit model designed to account for extreme response tendencies. The modified generalized partial credit model was found to have the best overall fit in terms of test-level, item-level, and person-level posterior predictive model checks performed. Estimation of this model also resulted in the lowest mean squared error between observed total score and expected total score. The multidimensional nominal response model had the lowest deviance information criterion among the three models. The empirical study, data validation from the simulation study, and the simulation results provided evidence that the IRTree Model was measuring a unique construct-irrelevant variance factor compared to the two other methods. For all simulation conditions of sample size (500, 1000), survey length (10, 20), and number of response options (4, 6), the modified generalized partial credit model had the most adequate model fit with respect to mean item mean squared error. The multidimensional nominal response model was found equally suitable for surveys measuring one substantive trait when responses to 10 4-option forced-choice Likert-type items were explored. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The purpose of the present study is to apply the item response theory (IRT) and testlet response theory (TRT) models to a reading comprehension test. This study applied the TRT models and the traditional IRT model to a seventh-grade reading comprehension test (n = 8,815) with eight testlets. These three models were compared to determine the best model for a testlet-based reading comprehension assessment. The goodness-of-fit indices such as -2 log likelihood, Akaike information criterion, and Bayesian information criterion were utilized as model comparison indices. The standardized local dependence X[superscript 2] statistic was computed for a comparison of local dependence among the three different models. Scatter plots were obtained to evaluate parameter-estimation consistency among models. Correlations and mean differences between the estimated parameters were also examined to detect and quantify the magnitude of inaccuracy due to the use of a worse-fitting model. Finally, items were evaluated based on the item parameters from the TRT models and compared to the results from the Coh Metrix. According to the three goodness-of-fit indices, the bi-factor model was the best-fitting model and the testlet-effects model was the second best-fitting model among the three models, showing a superiority of the TRT models for the testlet-based assessments compared to the more traditional IRT model. The standardized local dependence X2 statistic revealed significant local dependencies when the traditional IRT model was applied to the data, while the application of the TRT models resolved the problematic local dependencies. Items requiring the comprehension ability of context-related vocabulary showed high local dependencies when the 3-plm was applied. Comparisons of the item and person parameters and their standard error estimates showed a slight underestimation and less stable standard errors of the item discrimination parameters under the locally dependent condition. The results of the item parameters from the TRT models supported the advantages of the TRT models in selecting good items for accurate assessments. This study provided replicable evidence that the TRT models are crucial for testlet-based reading comprehension assessments. For future study, various comprehension tests containing more questions for each testlet as well as smaller sample sizes might be useful. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Investigating the psychometric properties of a screening instrument for young children is necessary to ascertain its quality and accuracy. In light of the important role culture plays on human beliefs and parenting styles, a newly translated and adapted test needs to be studied. Evaluating outcomes on a translated version of a test may reveal significant information related to cultural specifications as well as the common nature of child development. The current study examined psychometric properties of the 48-month interval of the Ages and Stages Questionnaires: Social-Emotional Second Edition (ASQ:SE-2) and its Traditional Chinese version (ASQ:SE-TC), using item response theory (IRT). Participants in the U.S. included 3,005 young children/parents dyads; 1,455 dyads were collected to represent a Taiwanese sample. A two-dimensional Rasch Partial Credit Model (2D-RPCM), which was determined to present a better fit than a unidimensional Rasch Partial Credit Model, was used to examine the item fit, item difficulty, reliability, and item information curves to evaluate the psychometric properties on the ASQ:SE and ASQ:SE-TC. Further, differential item functioning was conducted to examine whether items were functioning differently in the two population groups. Lastly, the differences between the distributions of children's latent traits on the continuum of social and emotional competencies for the U.S. and Taiwanese samples were investigated. Based on findings, the adequacy of psychometric properties is discussed, providing insight into the quality of particular items. Identified differences between the two populations are explored by reviewing literature regarding cultural comparisons of child-rearing practices, parenting styles, and cultural beliefs. Future directions for research include examining the cultural equivalence between translated and original versions of other ASQ:SE-2 intervals. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Item response theory (IRT) has gained an increasing popularity in large-scale educational and psychological testing situations because of its theoretical advantages over classical test theory. Unidimensional graded response models (GRMs) are useful when polytomous response items are designed to measure a unified latent trait. They are limited in practical instances where the test structure is not readily available or items are not necessarily measuring the same underlying trait. To overcome the problem, this dissertation proposes a multi-unidimensional normal ogive GRM under the fully Bayesian framework via the use of Markov chain Monte Carlo (MCMC). The performance of the proposed model was evaluated using the Monte Carlo simulations. It was further compared with conventional GRMs under simulated and real test situations. Results from simulation studies as well as a real data example suggest that (1) the proposed MCMC method for the proposed model provides fairly accurate and efficient parameter estimates, including correlations between latent dimensions, (2) compared with the conventional IRT models, the proposed model consistently performs well, if not better as far as the model-data fit is concerned. Therefore, the proposed multi-unidimensional model offers generosity, flexibility, and a better way to represent test situations when the latent dimensionality is not a priori clear or more than one latent trait is involved. In addition, the proposed model is not limited to tests in educational and psychological measurement. Instead, it can be applied to other disciplines, such as business and medicine, where Likert-type items are adopted in an instrument. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The current study introduced a general modeling framework, multilevel mixture IRT (MMIRT) which detects and describes characteristics of population heterogeneity, while accommodating the hierarchical data structure. In addition to introducing both continuous and discrete approaches to MMIRT, the main focus of the current study was to distinguish continuous and discrete MMIRT models from a model comparison perspective. A simulation study was conducted to evaluate the impact of class separation, cluster size, proportion of mixture, and between-group ability variance on the model performance of a set of MMIRT models. The behavior of information-based fit criteria in distinguishing between discrete and continuous MMIRT models was also investigated. An empirical analysis was presented to illustrate the application of MMIRT models. Results suggested that class separation, and between-group ability variance had significant impact on MMIRT model performance. Discrete MMIRT models with fewer group-level latent classes performed consistently better on parameter and classification recovery than the continuous MMIRT model and the discrete models with more latent classes at the group level. Despite the poor performance of the continuous MMIRT model, it was favored over the discrete models by most fit indices. The AIC, AIC3, AICC, and the modification of AIC and ssBIC were more sensitive to the discreteness in random effect distribution, compared to the CAIC, BIC, their modifications, and ssBIC. The latter ones had a higher tendency to select continuous MMIRT model as the best fitting model, regardless of the true distribution of random effects. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This study employed Bachman and Palmer's (2010) Assessment Use Argument framework to investigate to what extent the use of a second language oral test as an exit test in a Hong Kong university can be justified. It also aimed to help test developers of this oral test identify the most critical areas in the current test design that might need improvement. Candidates' oral responses to five integrated speaking tasks in this oral test were rated on five dimensions: Task fulfillment and relevance (TFR), Clarity of presentation (CoP), Grammar and Vocabulary (GV), Pronunciation (Pron), and Confidence and Fluency (CoFlu). To provide backing for the meaningfulness of interpretations, confirmatory factor analysis (CFA) and item response theory (IRT) analyses were used to analyze 999 candidates' scores and raters' verbal reports were also analyzed to provide complementary information to the results of the quantitative analyses. Several CFA models were first tested and compared in terms of their statistical fit and substantive interpretability. And a graded response model was applied to the test data. The CFA results showed that the superior fit of the Higher-order trait-Uncorrelated Method model validated the test design, confirmed the current multicomponential view of language ability in the literature, and provided the most parsimonious explanation of the relationships among the five dimensions and overall speaking proficiency. The analytic scores were found to have much larger factor loadings on the trait factors than on the method factors, providing evidence that the component test scores could be meaningfully interpreted as indicators of the five dimensions. The presence of a higher-order speaking ability factor governing the five trait factors also supported the practice of reporting one composite score. Task Fulfillment and Relevance (TFR) measured on Task 4 had the highest method loading (0.60) on Task 4 and the lowest trait factor loading (0.36) on TFR, which suggested TFR4 might be too task specific and weak in measuring students' speaking ability to fulfill a speaking task in a relevant way. The trace lines of the graded response model also confirmed this. The raters' verbal reports showed that most raters did not have much difficulty differentiating across the performance levels. Hence, the problem of TFR4 can only be due to the nature of the task itself and its low discrimination. Both CFA and IRT results indicated that task types had great effects on test takers' speaking abilities especially TFR and that this language ability component might be too task specific. In order to investigate the impartiality of interpretations, multi-group CFA and differential item functioning (DIF) were conducted to examine the extent to which the oral test had test bias and item bias across (1) gender and (2) disciplines. The multi-sample CFA results indicated that the factor structure was significantly different between males and females. However, the comparison of the factor loadings between females and males showed that only the factor loading of one item for the male group was significantly different from the female group at the 0.05 level. DIF results also suggested that the majority of the items displayed no DIF. The source of DIF may be attributed to the group mean difference on the latent trait and their real differences on certain aspects of language ability measured in this test. This provided backing for the impartiality of score interpretations, indicating that the rating-based interpretations from GSLPA SLT are impartial to a large extent across subgroups of test takers (males vs. females; business vs. non-business). In order to examine the consistency of test scores, Generalizability theory (G theory) analyses were performed to investigate whether the test was dependable and whether the five dimensions were separable. G theory results showed that the phi coefficient for the whole test fell between 0.76 and 0.85 and Grammar and Vocabulary and Pronunciation proved to be the most dependable dimensions. G theory and CFA results both confirmed that the five speaking dimension were highly correlated with each other. The possible reasons of these findings were further discussed with reference to the raters' verbal reports. Based on the above results, it can be concluded that the meaningfulness, impartiality, and consistency could be justified to a large extent. Some critical areas to be improved in the test design and administration were identified. Theoretical and practical implications were addressed and methodological limitations were also discussed. Overall, this study highlights the usefulness of Bachman and Palmer's Assessment Use Argument (2010) to justify the use of an existing language assessment. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Over the past decade, educational policy trends have shifted to a focus on examining students' growth from kindergarten through twelfth grade (K-12). One way states can track students' growth is with a vertical scale. Presently, every state that uses a vertical scale bases the scale on a unidimensional IRT model. These models make a strong but implausible assumption that a single construct is measured, in the same way, across grades. Additionally, research has found that variations of psychometric methods within the same model can result in different vertical scales. The purpose of this study was to examine the impact of three IRT models (unidimensional model, U3PL; bifactor model with grade specific subfactors, BG-M3PL; and a bifactor model with content specific subfactors, BC-M3PL); three calibration methods (separate, hybrid, and concurrent), and two scoring methods (EAP pattern and EAP summed scoring; EAPSS) on the resulting vertical scales. Empirical data based on a states' assessment program were used to create vertical scales for Mathematics and Reading from Grades 3-8. Several important results were found. First, the U3PL model always resulted in the worst model-data fit. The BC-M3PL fit the data best in Mathematics and the BG-M3PL fit the data best in Reading. Second, calibration methods led to minor differences in the resulting vertical scale. Third, examinee proficiency estimates based on the primary factor for each model were generally highly correlated (0.97+) across all conditions. Fourth, meaningful classification differences were observed across models, calibration methods, and scoring methods. Overall, I concluded that none of the models were viable for developing operational vertical scales. Multidimensional models are promising for addressing the current limitations of unidimensional models for vertical scaling but more research is needed to identify the correct model specification within and across grades. Implications for these results are discussed within the context of research, operational practice, and educational policy. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

Three decision consistency and accuracy (DC/DA) methods, the Livingston and Lewis (LL) method, LEE method, and the Hambleton and Han (HH) method, were evaluated. The purposes of the study were: (1) to evaluate the accuracy and robustness of these methods, especially when their assumptions were not well satisfied, (2) to investigate the "true" DC/DA indices in various conditions, and (3) to assess the impact of choice of reliability estimate on the LL method. Four simulation studies were conducted: Study 1 looked at various test lengths. Study 2 focused on local item dependency (LID). Study 3 checked the consequences of IRT model-data misfit and Study 4 checked the impact of using different scoring metrics. Finally, a real data study was conducted where no advantages were given to any models or assumptions. The results showed that the factors of LID and model misfit had a negative impact on "true" DA index, and made all selected methods over-estimate DA index. On the contrary, the DC estimates had minimal impacts from the above factors, although the LL method had poorer estimates in short tests and the LEE and HH methods were less robust to tests with a high level of LID. Comparing the selected methods, the LEE and HH methods had nearly identical results across all conditions, while the HH method had more flexibility in complex scoring metrics. The LL method was found sensitive to the choice of test reliability estimate. The LL method with Cronbach's alpha consistently underestimated DC estimates while LL with stratified alpha functioned noticeably better with smaller bias and more robustness in various conditions. Lastly it is hoped to make the software be available soon to permit the wider use of the HH method. The other methods in the study are already well supported by easy to use software. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

This study compared the model-data fit of a parametric item response theory (PIRT) model to a nonparametric item response theory (NIRT) model to determine the best-fitting model for use with ordinal-level alternate assessment ratings. The PIRT Generalized Graded Unfolding Model (GGUM) was compared to the NIRT Mokken model. Chi-square statistics were examined to determine the fit of the GGUM. To determine the fit of the Mokken model, Loevinger's H coefficients and the number of monotone homogeneity (MH) and double monotonicity (DM) violations were examined. Participants were 4,449 students ages 7 to 21 with autism spectrum disorder, a mild intellectual disability, moderate intellectual disability, or severe intellectual disability. These students participated in Indiana's alternate assessment, the Indiana Assessment System of Educational Proficiencies (IASEP) in the years 2000, 2001, or 2002. The IASEP consists of five subscales, consisting of 20 items, each, in the domains of Information Acquisition and Use (Language Arts and Math), Personal Adjustment, Social Adjustment, Recreation and Leisure, and Vocational Experience. Students are rated by teachers using a universal rubric ranging from 0 to 4. The GGUM showed better overall model-data fit with most items showing good fit across all disability groups. The Mokken model's stricter assumption of double monotonicity (DM) did not hold. The Mokken model's required property of monotone homogeneity (MH) held across most items and disability groups, making this model useful for selected applications. Researchers or practitioners may benefit when choosing which IRT model to use for alternate assessment ratings. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

The purpose of this study was to investigate procedures for assessing model fit of IRT models for mixed format data. In this study, various IRT model combinations were fitted to data containing both dichotomous and polytomous item responses, and the suitability of the chosen model mixtures was evaluated based on a number of model fit procedures. To assess model fit for mixed format data, five fit indices were considered: PARSCALE's G[superscript 2], the generalized forms of Orlando and Thissen's (2000) S-X[superscript 2] and S-G[superscript 2], and Stone's (2000) pseudo-observed score based indices chi[superscript 2*] and G[superscript 2*]. To investigate the relative performance of the five item fit statistics, two simulation studies were conducted: Type I error and power studies. Using the model hierarchy for a set of the data generation models and calibration models, model-fitting and model-misfitting conditions were manipulated to obtain Type I error rates and power. Under the model-fitting conditions, Type I error rates were computed when identical pairs of mixed IRT models were used for data generation and calibration. On the other hand, empirical power was calculated under the model-misfitting conditions when data were calibrated using a simpler model than a generating model. Among the competing measures, the number correct score based indices S-X[superscript 2] and S-G[superscript 2] were found to be the most efficient and safest choice for assessing model fit for mixed format data across the study conditions. These indices performed well particularly with short tests. However, the pseudo-observed score indices, chi[superscript 2*] and G[superscript 2*], showed inflated Type I error rates in some simulation conditions. Thus, their applications appeared to be limited by specific measurement conditions, and need to be studied further for generalized uses. Consistent with the findings of current literature, the PARSCALE's G[superscript 2] index was rarely useful, although it provided reasonable results for long tests. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]

An important aspect of empirical research is the construction of a model that represents the data. In psychological and educational measurement, models are typically evaluated regarding their ability to fit well to the observed data. Philosophers of science have long recognized that goodness-of-fit to the realized data is an insufficient metric of a model’s usefulness; models should also be appraised regarding their generalizability to unseen data. Frequentist statistics, Bayesian inference, and information theory seem to offer philosophically and methodologically dissimilar perspectives on model evaluation. However, this dissertation develops a simple theoretical framework that integrates these three perspectives. Within this framework, the information-theoretic principle of minimum description length is explored in the context of item response theory modeling. The findings reveal that complexity in item response theory is defined not by the number of freely estimated parameters in a model, but by its functional form. The frequentist, Bayesian, and information-theoretic approaches are then utilized in evaluating the usefulness of a unidimensional 3-parameter logistic model of item response data from the Program for International Student Assessment. Philosophical ramifications, future research directions, and implications for educational and psychological measurement are discussed.

The publisher of the Degrees of Reading Power test of reading comprehension (DRP) calibrate their test using an item response model called the Rasch or one-parameter logistic model. The relationship between the use of the Rasch model in calibration of the DRP and the use of the DRP as a component of the Virginia Literacy Passport Testing Program (LPT) is addressed. Analyses concentrate on sixth grade students who were administered the DRP in 1991. The question that arises is whether the Rasch model is the appropriate model to use to calibrate the DRP in this high-stakes setting. The majority of research that has been reported by the publisher of the DRP to assess the adequacy of the Rasch model have not included direct checks on model assumptions, model features or model predictions. Instead, they have relied almost exclusively on statistical tests in assessment of model fit. This study will assess the adequacy of fitting DRP test data to the Rasch model through direct examination of the assumptions, features and predictions of the IRT model. This is accomplished by comparing the Rasch model to the less restrictive two- and three-parameter logistic models. Robust IRT-based goodness-of-fit techniques are conducted. When the DRP is used in a high stakes setting, guessing is likely for those in jeopardy of failing. Under these circumstances, we must attend to the possibility that guessing may be a factor and thereby calibrate the DRP with the three-parameter model, as this model takes guessing into account.