Search

This study evaluated the psychometric quality of a computerized adaptive testing (CAT) version of the general cognitive ability test (GCAT), using a simulation study protocol put forth by Han, K. T. (2018a). For the needs of the analysis, three different sets of items were generated, providing an item pool of 165 items. Before evaluating the efficiency of the GCAT, all items in the final item pool were linked (equated), following a sequential approach. Data were generated using a standard normal for 10,000 virtual individuals (M = 0 and SD = 1). Using the measure's 165-item bank, the ability value ([theta]) for each participant was estimated. Maximum Fisher information (MFI) and maximum likelihood estimation with fences (MLEF) were used as item selection and score estimation methods, respectively. For item exposure control, the fade away method (FAM) was preferred. The termination criterion involved a minimum SE [less than or equal to]0.33. The study revealed that the average number of items administered for 10,000 participants was 15. Moreover, the precision level in estimating the participant's ability score was very high, as demonstrated by the CBIAS, CMAE, and CRMSE). It is concluded that the CAT version of the test is a promising alternative to administering the corresponding full-length measure since it reduces the number of administered items, prevents high rates of item exposure, and provides accurate scores with minimum measurement error.

Measurement specialists strive to shorten assessment time without compromising precision of scores. Computerized Adaptive Testing (CAT) has rapidly gained ground over the past decades to fulfill this goal. However, parameters for implementation of CATs need to be explored in simulations before implementation so that it can be determined whether expectations can be met. CATs can become costly if trial-and-error strategies are followed and especially if constraints are included in the algorithms, simulations can save time and money. In this study it was found that for both a multiple-choice question test and a rating scale questionnaire, simulations not only predicted outcomes for CATs very well, but also illustrated the efficiency of CATs when compared to fixed length tests.

Computerized Adaptive Testing (CAT) is now widely used. However, inserting new items into the question bank of a CAT requires a great effort that makes impractical the wide application of CAT in classroom teaching. One solution would be to use the tacit knowledge of the teachers or experts for a pre-classification and calibrate during the execution of tests with these items. Thus, this research consists of a comparative case study between a Stratified Adaptive Test (SAT), based on the tacit knowledge of a teacher, and a CAT based on Item Response Theory (IRT). The tests were applied in seven Computer Networks courses. The results indicate that levels of "anxiety" expressed in the use of the SAT were better than those using the CAT, in addition to being simpler to implement. In this way, it is recommended the implementation of a SAT, where the strata are initially based on the tacit knowledge of the teacher and later, as a result of an IRT calibration.

The purpose of this simulation study was to assess the accuracy of a classical test theory (CTT)-based procedure for estimating the alternate-forms reliability of scores on a multistage test (MST) having 3 stages. We generated item difficulty and discrimination parameters for 10 parallel, nonoverlapping forms of the complete 3-stage test and ability parameters for a population of 30,000 simulated test takers. Using these parameters to generate item responses, we ran each of the 30,000 simulated test takers through each of the 10 simulated forms of the 3-stage test and computed the correlation of the scores on each pair of simulated test forms. We then computed the CTT estimate of the reliability, in the full population of 30,000 simulated test takers, of the total scaled scores resulting from the multistage testing procedure. We computed the estimate separately from the simulated responses to each of the 10 simulated forms of the 3-stage test. The reliability estimates from each simulated form of the MST differed by less than 0.005 from the average of the correlations between scores on that form and on the other 9 simulated forms of the MST.

Prior research on the Measures of Academic Progress (MAP), a computer-adaptive test distributed by the Northwest Evaluation Association, has primarily focused on the Reading MAP for screening/benchmarking in elementary grades. The purpose of this study was to explore the functional form of growth and the extent to which student variability in growth was captured by the Mathematics MAP across a single school year using linear mixed-effects regression. Participants were 1608 middle school students from a school district in the Upper Midwest. Results supported a statistically significant nonlinear growth term for sixth grade and statistically significant, but small, monthly linear growth terms for sixth, seventh, and eighth grades. The Math MAP moderately captured individual variability across grades. Implications for research and practice are discussed.

Artificial Intelligence is at the heart of modern society with computers now capable of making process decisions in many spheres of human activity. In education, there has been intensive growth in systems that make formal and informal learning an anytime, anywhere activity for billions of people through online open educational resources and massive online open courses. Moreover, new developments in Artificial Intelligence-related educational assessment are attracting increasing interest as means of improving assessment efficacy and validity, with much attention focusing on the analysis of the large volumes of process data being captured from digital assessment contexts. In evaluating the state of play of Artificial Intelligence in formative and summative educational assessment, this paper offers a critical perspective on the two core applications: automated essay scoring systems and computerized adaptive tests, along with the Big Data analysis approaches to machine learning that underpin them.

An adaptive weight estimation approach is proposed to provide robust latent ability estimation in computerized adaptive testing (CAT) with response revision. This approach assigns different weights to each distinct response to the same item when response revision is allowed in CAT. Two types of weight estimation procedures, nonfunctional and functional weight, are proposed to determine the weight adaptively based on the compatibility of each revised response with the assumed statistical model in relation to remaining observations. The application of this estimation approach to a data set collected from a large-scale multistage adaptive testing demonstrates the capability of this method to reveal more information regarding the test taker's latent ability by using the valid response path compared with only using the very last response. Limited simulation studies were concluded to evaluate the proposed ability estimation method and to compare it with several other estimation procedures in literature. Results indicate that the proposed ability estimation approach is able to provide robust estimation results in two test-taking scenarios.

The use of computer-based tests (CBTs), for both formative and summative purposes, has greatly increased over the past years. One major advantage of CBTs is the easy integration of multimedia. It is unclear, though, how to design such CBT environments with multimedia. The purpose of the current study was to examine whether guidelines for designing multimedia "instruction" based on the Cognitive Load Theory (CLT) and Cognitive Theory of Multimedia Learning (CTML) will yield similar effects in CBT. In a within-subject design, thirty-three vocational students completed a computer-based arithmetic exam, in which half of the items were presented in an original design format, and the other half was redesigned based on the CTML principles for instructional design (i.e., adapted format). Results showed that applying CTML principles to a CBT decreased the difficulty of the test items, i.e., students scored significantly higher on the adapted test items. Moreover, eye-tracking data showed that the adapted items required less visual search and increased attention for the question and answer. Finally, cognitive load, measured as silent pauses during a secondary think-aloud task, decreased. Mean fixation duration (a different indicator of cognitive load), however, did not significantly differ between adapted and original items. These results indicate that applying multimedia principles to CBTs can be beneficial. It seems to prevent cognitive overload and helps students to focus on important parts of the test items (e.g., the question), leading to better test results.

An essential question when computing test--retest and alternate forms reliability coefficients is how many days there should be between tests. This article uses data from reading and math computerized adaptive tests to explore how the number of days between tests impacts alternate forms reliability coefficients. Results suggest that the highest alternate forms reliability coefficients were obtained when the second test was administered at least 2 to 3 weeks after the first test. Even though reliability coefficients after this amount of time were often similar, results suggested a potential tradeoff in waiting longer to retest as student ability tended to grow with time. These findings indicate that if keeping student ability similar is a concern that the best time to retest is shortly after 3 weeks have passed since the first test. Additional analyses suggested that alternate forms reliability coefficients were lower when tests were shorter and that narrowing the first test ability distribution of examinees also impacted estimates. Results did not appear to be largely impacted by differences in first test average ability, student demographics, or whether the student took the test under standard or extended time. It is suggested that for math and reading tests, like the ones analyzed in this article, the optimal retest interval would be shortly after 3 weeks have passed since the first test.

Many recent studies have looked at the viability of applying recurrent neural networks (RNNs) to educational data. In most cases, this is done by comparing their performance to existing models in the artificial intelligence in education (AIED) and educational data mining (EDM) fields. While there is increasing evidence that, in many situations, RNN models can improve on the performance of these existing methods, in this work we take a different approach. Rather than directly comparing RNNs with other models, we are instead interested in the results when RNNs are combined with one of these existing models. In particular, we attempt to improve the performance of ALEKS ("A ssessment and LE arning in K nowledge S paces"), an adaptive learning and assessment system based on Knowledge Space Theory, through the use of RNN models. Using data from more than 1.4 million ALEKS assessments, we first build an RNN classifier that attempts to predict the final result of each assessment. After verifying the accuracy of these predictions, we develop our stopping algorithm, with the goal of improving the efficiency of the ALEKS assessment by reducing the total number of questions that are asked. Based on this stopping algorithm, we give a comprehensive analysis of the possible effects it would have on students. We show that the combination of an RNN with the ALEKS assessment can reduce the average assessment length by over 26%, while a high degree of accuracy is maintained.

Purpose: This study aims to reveal effects of content balancing and item selection method on ability estimation in computerized adaptive tests by comparing Fisher's maximum information (FMI) and likelihood weighted information (LWI) methods. Research Methods: Four groups of examinees (250, 500, 750, 1000) and a bank of 500 items with 10 different content domains were generated through Monte Carlo simulations. Examinee ability was estimated by fixing all settings except for the item selection methods mentioned. True and estimated ability (?) values were compared by dividing examinees into six subgroups. Moreover, the average number of items used was compared. Findings: The correlations decreased steadily as examinee ? level increased among all examinee groups when LWI was used. FMI had the same trend with the 250 and 500 examinees. Correlations for 750 examinees decreased as ? level increased as well, but they were somewhat steady with FMI. For 1000 examinees, FMI was not successful in estimating examinee ? accurately after ? subgroup 4. Moreover, when FMI was used, ? estimates had less error than LWI. The figures regarding the average items used indicated that LWI used fewer items in subgroups 1, 2, 3 and that FMI used less items in subgroups 4, 5, and 6. Implications for Research and Practice: The findings indicated that when content balancing is put into use, LWI is more suitable to estimate examinee ? for examinees between -3 and 0 and that FMI is more stable when examinee ? is above 0. An item selection algorithm combining these two item selection methods is recommended.

This article aims to provide a theoretical framework for computerized adaptive tests (CAT) and item response theory models for polytomous items. Besides that, it aims to introduce the simulation and live CAT software to the related researchers. Computerized adaptive test algorithm, assumptions of item response theory models, nominal response model, partial credit and generalized partial credit models and graded response model are described carefully to reach that aim. Likewise, item selection methods, such as maximum Fisher information, maximum expected information, minimum expected posterior variance, maximum expected posterior weighted-information, and ability prediction methods, such as expected a posteriori and maximum a posteriori, are expounded as well as stopping rules for the computerized adaptive tests.

The adaptation of experimental cognitive tasks into measures that can be used to quantify neurocognitive outcomes in translational studies and clinical trials has become a key component of the strategy to address psychiatric and neurological disorders. Unfortunately, while most experimental cognitive tests have strong theoretical bases, they can have poor psychometric properties, leaving them vulnerable to measurement challenges that undermine their use in applied settings. Item response theory-based computerized adaptive testing has been proposed as a solution but has been limited in experimental and translational research due to its large sample requirements. We present a generalized latent variable model that, when combined with strong parametric assumptions based on mathematical cognitive models, permits the use of adaptive testing without large samples or the need to precalibrate item parameters. The approach is demonstrated using data from a common measure of working memory--the N-back task--collected across a diverse sample of participants. After evaluating dimensionality and model fit, we conducted a simulation study to compare adaptive versus nonadaptive testing. Computerized adaptive testing either made the task 36% more efficient or score estimates 23% more precise, when compared to nonadaptive testing. This proof-of-concept study demonstrates that latent variable modeling and adaptive testing can be used in experimental cognitive testing even with relatively small samples. Adaptive testing has the potential to improve the impact and replicability of findings from translational studies and clinical trials that use experimental cognitive tasks as outcome measures.

This study proposed a recursion-based analytical approach to assess measurement precision of ability estimation and classification accuracy in multistage adaptive tests (MSTs). A simulation study was conducted to compare the proposed recursion-based analytical method with an analytical method proposed by Park, Kim, Chung, and Dodd and with the more commonly used Monte Carlo (MC) simulation approaches. The results from the simulation study indicate that the recursion-based analytical method produced more stable measurement properties in MST than the MC-based simulation method. It also produced more credible measurement performance than the analytical approach of Park, Kim, Chung, and Dodd. We expect that the recursion-based analytical method would be especially efficient in instances when multiple MST designs need to be compared to determine which one has better measurement performance.

Adaptive tests are more efficient than fixed-length tests through the use of item response theory; adaptive tests also present students questions that are tailored to their proficiency level. Although the adaptive algorithm is straightforward, developing a multidimensional computer adaptive test (MCAT) measure is complex. Evidence-centered design (ECD) can provide a highly structured framework to guide the development of a MCAT. In this paper, the development of the adaptive reading motivation measure (ARMM) demonstrates the process of applying ECD to the development of a MCAT measure. This paper focuses on the conceptual assessment framework layer of the ECD that guides the technical aspects of MCAT development. The five models of conceptual assessment framework layer are student model, task model, evidence model, assembly model, and presentation model. How each model guided the development of the ARMM is described in detail. In this paper, we demonstrate how the conceptual assessment framework in ECD can provide a useful structure for developing MCAT measures.

Technology-Enhanced Items (TEIs) have been purported to be more motivating and engaging to test takers than traditional multiple-choice items. The claim of enhanced engagement, however, has thus far received limited research attention. This study examined the rates of rapid-guessing behavior received by three types of items (multiple-choice, multiple select, and TEIs) on a commonly-used K-12 adaptive achievement test. Across three subject areas, the TEIs consistently showed the lowest rapid guessing rate, suggesting that their use may help mitigate the problem of disengaged test taking.

Multistage testing (MST), an adaptive test delivery mode that involves algorithmic selection of predefined item modules rather than individual items, offers a practical alternative to linear and fully computerized adaptive testing. However, interactions across stages between item modules and examinee groups can lead to challenges in item calibration with MST. This study used simulated data based on an operational program to investigate the performance of four item calibration methods under a 1-3 MST design. Conditions included routing module length, routing rule, and sample size. Calibration methods were evaluated based on item and person parameter recovery and classification accuracy. Results indicated that calibration with fixed common item parameters and concurrent calibration assuming a single ability distribution similarly outperformed both separate calibration with linking and concurrent calibration with the multiple-group procedure.

Designing a multidimensional adaptive test (M-MST) based on a multidimensional item response theory (MIRT) model is critical to make full use of the advantages of both MST and MIRT in implementing multidimensional assessments. This study proposed two types of automated test assembly (ATA) algorithms and one set of routing rules that can facilitate the development of an M-MST. Different M-MST designs were developed based on the proposed ATA algorithms and routing rules and were evaluated through two sets of simulation studies. Study 1 used simulated item banks and considered a variety of testing factors, and results from which can inform us the theoretical performance of the proposed M-MST designs. Study 2 was designed based on a real multidimensional assessment. In addition, a MCAT was simulated as a baseline design to demonstrate the advantage of the proposed M-MST design in each study. Our simulation results indicate that the proposed ATA algorithms and routing rule can generate M-MSTs with a good-quality control and the same or even better ability estimation results than the MCAT given the same condition. This demonstrates the advantage of using M-MST for multidimensional assessment especially for the one that needs to satisfy many nonstatistical constraints.

As a method to shorten the test time of the Force Concept Inventory (FCI), we suggest the use of computerized adaptive testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. In so doing, the test length can be significantly shortened. As a step to develop a CAT-based version of the FCI (FCI-CAT), we examined the optimal test length of the FCI-CAT such that accuracy and precision [which we measure in terms of bias, standard error, and root-mean-square error (RMSE)] of Cohen's d would be comparable to that of the full FCI for a given class size. First, we estimated the item parameters of the FCI items based on the three-parameter logistic model of item response theory, which are used in the algorithm of CAT. For this estimation, we used 2882 responses of Japanese university students. Second, we conducted a Monte Carlo simulation to analyze how the bias, standard error, and RMSE of Cohen's "d" depend upon the test length. Third, we conducted a "post hoc" simulation to examine the consistency of the Monte Carlo results with what would have been obtained using empirical responses. For this comparison, we used 86 pairs of pre- and post- test responses of Japanese university students. As a result, we found that for a class size of 40, we may reduce the test length of the FCI-CAT to 15-19 items, thereby reducing the test time of the FCI to 50%-63%, with an accompanying decrease in accuracy and precision of only 5%-10%. The results of the Monte Carlo study and the "post hoc" simulation were consistent, which supports the adequacy of our Monte Carlo study and its relevance in terms of administering the FCI-CAT in real classrooms.

Low examinee effort is a major threat to valid uses of many test scores. Fortunately, several methods have been developed to detect noneffortful item responses, most of which use response times. To accurately identify noneffortful responses, one must set response time thresholds separating those responses from effortful ones. While other studies have compared the efficacy of different threshold-setting methods, they typically do so using simulated or small-scale data. When large-scale data are used in such studies, they often are not from a computer-adaptive test (CAT), use only a handful of items, or do not comprehensively examine different threshold-setting methods. In this study, we use reading test scores from over 728,923 3rd-8th-grade students in 2056 schools across the United States taking a CAT consisting of nearly 12,000 items to compare threshold-setting methods. In so doing, we help provide guidance to developers and administrators of large-scale assessments on the tradeoffs involved in using a given method to identify noneffortful responses.

A key consideration when giving any computerized adaptive test (CAT) is how much adaptation is present when the test is used in practice. This study introduces a new framework to measure the amount of adaptation of Rasch-based CATs based on looking at the differences between the selected item locations (Rasch item difficulty parameters) of the administered items and target item locations determined from provisional ability estimates at the start of each item. Several new indices based on this framework are introduced and compared to previously suggested measures of adaptation using simulated and real test data. Results from the simulation indicate that some previously suggested indices are not as sensitive to changes in item pool size and the use of constraints as the new indices and may not work as well under different item selection rules. The simulation study and real data example also illustrate the utility of using the new indices to measure adaptation at both a group and individual level. Discussion is provided on how one may use several of the indices to measure adaptation of Rasch-based CATs in practice.

The more frequent collection of response time data is leading to an increased need for an understanding of how such data can be included in measurement models. Models for response time have been advanced, but relatively limited large-scale empirical investigations have been conducted. We take advantage of a large data set from the adaptive NWEA MAP Growth Reading Assessment to shed light on emergent features of response time behavior. We identify two behaviors in particular. The first, response acceleration, is a reduction in response time for responses that occur later in the assessment. We note that such reductions are heterogeneous as a function of estimated ability (lower ability estimates are associated with larger increases in acceleration) and that reductions in response time lead to lower accuracy relative to expectation for lower ability students. The second is within-person variation in the association between time usage and accuracy. Idiosyncratic within-person changes in response time have inconsistent implications for accuracy; in some cases additional response time predicts higher accuracy but in other cases additional response time predicts declines in accuracy. These findings have implications for models that incorporate response time and accuracy. Our approach may be useful in other studies of adaptive testing data.

Trust-based e-assessment systems are increasingly important in the digital age for both academic institutions and students, including students with special educational needs and disabilities (SEND). Recent literature indicates a growing number of studies about e-authentication and authorship verification for quality assurance with more flexible modes of assessment. Yet understanding the acceptability of e-authentication systems among SEND students is underexplored. This study examines SEND students' views about the use of e-authentication systems, including perceived advantages and disadvantages of new technology-enhanced assessment. This study aims to shed light on this area by examining the attitudes of 267 SEND students who used, or were aware of, an authentication system known as adaptive trust-based e-assessment system for learning (TeSLA). The results suggest a broadly positive acceptability of these e-authentication technologies by SEND students. In the view of these students, the key advantages are the ability of proving the originality of their work, and trust-based e-assessment results; the key disadvantages are the possibility that the technology might not work or present wrong outputs in terms of cheating.

Retrieval practice promotes retention more than restudying (i.e., the "testing effect") and is applied to many educational settings. However, little research has investigated means to enhance this effect in educational settings. Theoretical accounts assume retrieval practice to be the most effective whenever retrieval is difficult but successful. Therefore, we developed a novel retrieval practice procedure, which adapts to learners' abilities and can be applied irrespective of learning content. This adaptive procedure aims to make retrieval gradually easier whenever students provide an incorrect answer. In a field experiment, students read book chapters as part of a weekly university course. In three consecutive weeks, they then practiced reading assignments by (a) adaptive testing, (b) non-adaptive testing and (c) restudy. In Week 4, a surprise criterial test took place. Restudy outperformed both testing conditions, whereas adaptive testing performed equally well as non-adaptive testing. However, exploratory analyses revealed that with increasing retention intervals, the superiority of restudy disappeared. Furthermore, whenever participants fully read the assignments and retention intervals increased, adaptive testing outperformed non-adaptive testing. In sum, adaptive retrieval practice did not prove to be generally superior, but retention interval and students' preparation for class might be conditions rendering adaptive retrieval useful in educational settings.

The purpose of this study is to evaluate the extent to which item response theory (IRT) proficiency estimation methods are robust to the presence of aberrant responses under the "GRE"® General Test multistage adaptive testing (MST) design. To that end, a wide range of atypical response behaviors affecting as much as 10% of the test items were simulated using a generic GRE 2-stage MST and the 2-parameter logistic (2PL) IRT model. As expected, some differences were found among the 5 estimators in terms of the recovery of the true theta ability; for example, Bayesian estimators had lower error variance and their estimates were regressed to the mean.Once the IRT theta estimates were scaled onto a comparable reporting score scale, however, it was found that all the estimation methods investigated, including the one currently used to score GRE MSTs, were equally robust under the simulated conditions.

Exploratory and confirmatory factor analyses of domestic data from the" GRE"® revised General Test, introduced in 2011, were conducted separately for the verbal (VBL) and quantitative (QNT) reasoning measures to evaluate the unidimensionality and local independence assumptions required by item response theory (IRT). Results based on data from the period immediately after the launch of the revised test and data from a year later showed that very little local item dependence was present and that a predominant single factor accounted for each of the data sets. These results provide evidence supporting the assumptions that underlie the use of the unidimensional 2-parameter IRT model for scoring and contribute evidence for the validity of GRE scores and their use.

In this paper we propose a method for the adaptive complex assessment (ACA) of the higher-order thinking (HOT) skills needed by students for problem solving, and we examine the impact of the method on the development of HOT skills in a problem-based learning (PBL) environment. Complexity in the assessment is provided by initial, formative, and adaptive assessments of HOT skills; assessments of collaborative skills; and summative assessments of HOT skills. Adaptability in the assessment is provided through the dynamics of an instructor's assessments based on developing HOT skills; through a flexible choice of control tests and instructional problems for students and collaborative groups; and through the self-formation of heterogeneous, collaborative HOT skill groups. The assessment fosters the development of HOT and collaborative skills through a combination of personalized and collaborative problem-based learning (PBL). The three-stage assessment process guides the development of HOT skills during subject study through PBL. The focus of the first stage is on developing the HOT skills of students through personalized PBL. The second stage is devoted to developing HOT skills and collaborative skills through collaborative PBL. The third stage is devoted to assessing collaborative skills and constructing summative assessments of students. The proposed calculations for the coefficients of HOT skill development serve as a constructive means of exploring the impact of the ACA method on the HOT skill development of students.

Polytomous items are typically described by multiple category-related parameters; situations, however, arise in which a single index is needed to describe an item's location along a latent trait continuum. Situations in which a single index would be needed include item selection in computerized adaptive testing or test assembly. Therefore single location indices for ordinal polytomous items are proposed and studied. The proposed location indices (LIs) for polytomous items are mathematically derived based on the item category response functions (ICRFs) and item response function (IRF) for polytomous items. The ICRF approach resulted in three indices: LI[subscript mean], LI[subscript trimmed mean], and LI[subscript median], and the IRF approach resulted in one proposed index, LI[subscript IRF]. An empirical example of real items is presented to help comprehension of the new location indices. Possible testing applications where the proposed item location indices are useful are discussed.

This study aimed to investigate the effects of calibration sample size and item bank size on examinee ability estimation in computerized adaptive testing (CAT). For this purpose, a 500-item bank pre-calibrated using the three-parameter logistic model with 10,000 examinees was simulated. Calibration samples of varying sizes (150, 250, 350, 500, 750, 1,000, 2,000, 3,000, and 5,000) were selected from the parent sample, and item banks that represented small (100) and medium size (200 and 300) banks were drawn from the 500-item bank. Items in these banks were recalibrated using the drawn samples, and their estimated parameters were used in post-hoc simulations to re-estimate ability parameters for the simulated 10,000 examinees. The findings showed that ability estimates in CAT are robust against fluctuations in item parameter estimation and that accurate ability parameter estimates can be obtained with a calibration sample of 150 examinees. Moreover, a 200-item bank pre-calibrated with as few as 150 examinees can be used for some purposes in CAT as long as it has sufficient information at targeted ability levels.

The present study extended the "p"-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the "p"-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the "p"-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.

Purpose: This study introduces a framework to produce very short versions of the MacArthur-Bates Communicative Development Inventories (CDIs) by combining the Bayesian-inspired approach introduced by Mayor and Mani (2019) with an item response theory-based computerized adaptive testing that adapts to the ability of each child, in line with Makransky et al. (2016). Method: We evaluated the performance of our approach--dynamically selecting maximally informative words from the CDI and combining parental response with prior vocabulary data--by conducting real-data simulations using four CDI versions having varying sample sizes on Wordbank--the online repository of digitalized CDIs: American English (a very large data set), Danish (a large data set), Beijing Mandarin (a medium-sized data set), and Italian (a small data set). Results: Real-data simulations revealed that correlations exceeding 0.95 with full CDI administrations were reached with as few as 15 test items, with high levels of reliability, even when languages (e.g., Italian) possessed few digitalized administrations on Wordbank. Conclusions: The current approach establishes a generic framework that produces very short (less than 20 items) adaptive early vocabulary assessments--hence considerably reducing their administration time. This approach appears to be robust even when CDIs have smaller samples in online repositories, for example, with around 50 samples per month-age.

Reading screening assessments help educators identify students who are at risk of reading and determine the need for intervention and supports. However, some schools screen and assess students more often than needed, and the additional information does not improve the accuracy of decisions. This may be especially true for students at the upper elementary grades who have already taken high-stakes tests. This exploratory study evaluated how accurately a variety of screening measures predicted performance on a high-stakes end of year test for fourth- and fifth-grade students. Results of this study indicated that previous scores on the statewide assessment and computer-adaptive assessment best predicted student performance on a high-stakes reading test (Minnesota Comprehensive Assessment--Third Edition). When comparing screening models, a two-gate approach appeared to be the best method for identifying student risk.

For the training of academic skills, digital educational games with integrated adaptivity are promising. Adaptive games are considered superior to non-adaptive games, because they constantly assess children's performance, and accordingly adapt the difficulty of the tasks corresponding to the children's individual level. However, empirical evidence with regard to the effectivity of adaptive compared to non-adaptive games is limited. A study was conducted with 191 children from the third year of Kindergarten who were enrolled in one of three conditions, that is, playing an adaptive version of the reading game (RG), a non-adaptive version of the RG or training with pen-and-paper exercises. In all three conditions, children trained emergent reading (phonological awareness and letter knowledge) once a week for 30?min over a period of 5?weeks. Children's performance on cognitive (phonological awareness, letter knowledge, reading fluency) and non-cognitive (motivation, self-concept) factors was assessed. Results revealed a significant improvement in phonological awareness and letter knowledge in all conditions. However, no differences between the conditions were observed with respect to children's improvement on phonological awareness and letter knowledge or on their post-test scores for reading fluency. With regard to motivation and self-concept, again, no differences in these non-cognitive factors were observed across conditions.

In large-scale educational assessment data consistent with a simple-structure multidimensional item response theory (MIRT) model, where every item measures only one latent variable, separate unidimensional item response theory (UIRT) models for each latent variable are often calibrated for practical reasons. While this approach can be valid for data from a linear test, unacceptable item parameter estimates are obtained when data arise from a multistage test (MST). We explore this situation from a missing data perspective and show mathematically that MST data will be problematic for calibrating multiple UIRT models but not MIRT models. This occurs because some items that were used in the routing decision are excluded from the separate UIRT models, due to measuring a different latent variable. Both simulated and real data from the National Assessment of Educational Progress are used to further confirm and explore the unacceptable item parameter estimates. The theoretical and empirical results confirm that only MIRT models are valid for item calibration of multidimensional MST data.

Universal screening is the first mechanism by which students are identified as at risk of failure in the context of multitiered systems of supports. This study examined the validity and diagnostic accuracy of a reading computer-adaptive test as a screener to identify state achievement test performance for third through fifth graders (N = 1,696). Single time points and slopes within year and longitudinally were examined. Validity results for single points were moderate (0.60-0.79, p < 0.002). Validity for slopes and the state test were weak or not significant. Diagnostic accuracy cut scores that maximized sensitivity and specificity yielded high accuracy for single points whereas sensitivity was inadequate for slopes. Practical implications and future directions are presented.

With the widespread use of computers in modern assessment, online calibration has become increasingly popular as a way of replenishing an item pool. The present study discusses online calibration strategies for a joint model of responses and response times. The study proposes likelihood inference methods for item paramter estimation and evaluates their performance along with optimal sampling procedures. An extensive simulation study indicates that the proposed online calibration strategies perform well with relatively small samples (e.g., 500~800 examinees). The analysis of estimated parameters suggests that response time information can be used to improve the recovery of the response model parameters. Among a number of sampling methods investigated, "A"-optimal sampling was found most advantageous when the item parameters were weakly correlated. When the parameters were strongly correlated, "D"-optimal sampling tended to achieve the most accurate parameter recovery. The study provides guidelines for deciding sampling design under a specific goal of online calibration given the characteristics of field-testing items.

While exam-style questions are a fundamental educational tool serving a variety of purposes, manual construction of questions is a complex process that requires training, experience, and resources. This, in turn, hinders and slows down the use of educational activities (e.g. providing practice questions) and new advances (e.g. adaptive testing) that require a large pool of questions. To reduce the expenses associated with manual construction of questions and to satisfy the need for a continuous supply of new questions, automatic question generation (AQG) techniques were introduced. This review extends a previous review on AQG literature that has been published up to late 2014. It includes 93 papers that were between 2015 and early 2019 and tackle the automatic generation of questions for educational purposes. The aims of this review are to: provide an overview of the AQG community and its activities, summarise the current trends and advances in AQG, highlight the changes that the area has undergone in the recent years, and suggest areas for improvement and future opportunities for AQG. Similar to what was found previously, there is little focus in the current literature on generating questions of controlled difficulty, enriching question forms and structures, automating template construction, improving presentation, and generating feedback. Our findings also suggest the need to further improve experimental reporting, harmonise evaluation metrics, and investigate other evaluation methods that are more feasible.

The Bayesian way of accounting for the effects of error in the ability and item parameters in adaptive testing is through the joint posterior distribution of all parameters. An optimized Markov chain Monte Carlo algorithm for adaptive testing is presented, which samples this distribution in real time to score the examinee's ability and optimally select the items. Thanks to extremely rapid convergence of the Markov chain and simple posterior calculations, the algorithm is ready for use in real-world adaptive testing with running times fully comparable with algorithms that fix all parameters at point estimates during testing.