Search

The main purpose of the study was to investigate the distinctness and reliability of analytic (or multitrait) rating dimensions and their relationships to holistic scores and "e-rater"® essay feature variables in the context of the TOEFL® computer-based test (CBT) writing assessment. Data analyzed in the study were analytic and holistic essay scores provided by human raters and essay feature variable scores computed by "e-rater" (version 2.0) for two TOEFL CBT writing prompts. It was found that (a) all of the six analytic scores were not only correlated among themselves but also correlated with the holistic scores, (b) high correlations obtained among holistic and analytic scores were largely attributable to the impact of essay length on both analytic and holistic scoring, (c) there may be some potential for profile scoring based on analytic scores, and (d) some strong associations were confirmed between several "e-rater" variables and analytic ratings. Implications are discussed for improving the analytic scoring of essays, validating automated scores, and refining "e-rater" essay feature variables.

This study assessed the factor structure of the LanguEdge™ test and the invariance of its factors across language groups. Confirmatory factor analyses of individual tasks and subsets of items in the four sections of the test, Listening, Reading, Speaking, and Writing, was carried out for Arabic-, Chinese-, and Spanish-speaking test takers. Two factors were identified, Speaking and a fusion of the other sections of the test. The number of factors, the factor loadings, and the factors' error variances were invariant in the three samples, although the correlations between the factors differed. The failure to find separate factors for each section of the LanguEdge test necessarily raises questions about the test's functioning that need to be resolved.

Possible integrated and independent tasks were pilot tested for the writing section of a new generation of TOEFL® (Test of English as a Foreign Language™) examination. This study examines the impact of various rating designs as well as the impact of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the perspective of generalizability theory (G-theory). Both univariate and multivariate G-theory analyses were conducted. It was found that (a) in terms of maximizing the score reliability, it would be more efficient to increase the number of tasks rather than the number of ratings per essay; (b) two particular single-rating designs having different tasks for the same examinee rated by different raters [p × (R:T), R:(p × T)] achieved relatively higher score reliabilities than other single-rating designs; and (c) a somewhat larger gain in composite score reliability was achieved when the number of listening-writing tasks was larger than the number of reading-writing tasks.

Three polytomous DIF detection techniques--the Mantel test, logistic regression, and polySTAND--were used to identify GRE® Analytical Writing prompts ("Issue" and "Argument") that are differentially difficult for (a) female test takers; (b) African American, Asian, and Hispanic test takers; and (c) test takers whose strongest language skills are in a language other than English. Prompts in the gender comparison showed low differential item functioning (DIF) values. Higher DIF values were found in the White versus African American comparison (for Argument prompts), while a smaller number of prompts showed moderate DIF to the disadvantage of the Hispanic (for Argument prompts), Asian American (for Issue prompts), and English as a second language groups (for Issue prompts). No DIF values were found that were large enough to warrant the removal of prompts from the item pool. Several potential causes of high DIF values for some prompts are discussed with respect to the content characteristics of these prompts.

This study has investigated the comparability of computer-based testing (CBT) writing prompts in the Test of English as a Foreign Language™ (TOEFL®) for examinees of different native language backgrounds. A total of 81 writing prompts introduced from July 1998 through August 2000 were examined using a three-step logistic regression procedure for ordinal items. An English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scale scores. This ELA variable was used to match examinees of East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) language groups. Although about one third of the 81 prompts were initially flagged because of statistically significant group effects, the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect.

Eighty-three Test of English as a Foreign Language™ (TOEFL®) CBT writing prompts that were administered between July 1998 and August 2000 were examined in order to identify differences in scores that could be attributed to the response mode chosen by examinees (handwritten or word processed). Differences were examined statistically using polytomous logistic regression. An English language ability (ELA) variable was developed from the multiple-choice components of the TOEFL examination and used as a matching variable. Although there was little observed difference in mean writing scores, when examinees were matched on English language ability, small differences were observed in effect sizes consistently favoring the handwritten response mode. The difference favoring the handwritten response mode occurred for all of the writing prompts analyzed; however, the differences for individual writing prompts were small. This difference suggests a general effect for response mode.

Diagnostic language assessment (DLA) is gaining a lot of attention from language teachers, testers, and applied linguists. With a recent surge of interest in DLA, there seems to be an urgent need to assess where the field of DLA stands at the moment and develop a general sense of where it should be moving in the future. The current article, as the first article in this special issue, aims to provide a general theoretical background for discussion of DLA and address some fundamental issues surrounding DLA. More specifically, the article (a) examines some of the defining characteristics of DLA and its major components, (b) reviews the current state of DLA in conjunction with these components, and (c) identifies some promising areas of future research and development of DLA where important breakthroughs can be made in the future. Some of the major obstacles and challenges facing DLA are identified and discussed, along with some possible solutions to them.

This study examined lexical processing in English by native speakers of Korean and Chinese, compared to that of native speakers of English, using normal, alternated, and inverse fonts. Sixty four adult students participated in a lexical decision task. The findings demonstrated similarities and differences in accuracy and latency among the three L1 groups. The participants, regardless of L1, had a greater advantage in nonwords than words for the normal fonts because they were able to efficiently detect the illegal letter strings. However, word advantages were observed in the visually distorted stimuli (i.e., alternated and inverse fonts). These results were explained from the perspectives of the theory of psycholinguistic grain size, L1-L2 distance, and the mechanism of familiarity discrimination. The native speakers of Chinese were more sensitive to visual distortions than the Korean counterpart, suggesting that the linguistic template established in L1 might play a role in word processing in English.

Objectives: Postoperative urinary retention (POUR) is influenced by many factors, and its reported incidence rate varies widely. This study aimed to investigate the occurrence and risk factors for urinary retention following general anesthesia for endoscopic nasal surgery in male patients aged >60 years. Methods: A retrospective review of medical records between January 2015 and December 2019 identified 253 patients for inclusion in our study. Age, body mass index (BMI), a history of diabetes/hypertension, American Society of Anesthesiologists (ASA) classification, and urologic history were included as patient-related factors. Urologic history was subdivided into 3 groups according to history of benign prostate hyperplasia (BPH)/lower urinary tract symptoms (LUTS) and current medication. The following was analyzed as perioperative variables for POUR development: duration of anesthesia and surgery; amount of fluid administered; rate of fluid administration; intraoperative requirement for fentanyl, ephedrine, and dexamethasone; postoperative pain; and analgesic use. Preoperatively measured prostate size and uroflowmetry parameters of patients on medication for symptoms were compared according to the incidence of urinary retention. Results: Thirty-seven (15.7%) patients developed POUR. Age (71.4 vs 69.6 years), BMI (23.9 vs 24.9 kg/m2), a history of diabetes/hypertension, ASA classification, and perioperative variables were not significantly different between patients with and without POUR. Only urologic history was identified as a factor affecting the occurrence of POUR (P =.03). The incidence rate among patients without urologic issues was 5.9%, whereas that among patients with BPH/LUTS history was 19.8%. Among patients taking medication for symptoms, the maximal and average velocity of urine flow were significantly lower in patients with POUR. Conclusions: General anesthesia for endoscopic nasal surgery may be a potent trigger for urinary retention in male patients aged >60 years. The patient's urological history and urinary conditions appear to affect the occurrence of POUR. [ABSTRACT FROM AUTHOR]

The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT[R] (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors--Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation. (Contains 9 tables, 4 figure and 1 footnote.)

The main purpose of the study was to investigate the distinctness and reliability of analytic (or multi-trait) rating dimensions and their relationships to holistic scores and "e-rater"[R] essay feature variables in the context of the TOEFL[R] computer-based test (TOEFL CBT) writing assessment. Data analyzed in the study were holistic and multi-trait essay scores provided by human raters and essay feature variable scores computed by "e-rater"-[R] (version 2.0) for two TOEFL CBT writing prompts. It was found that (i) all of the six multi-trait scores were not only correlated among themselves but also correlated with the holistic score, (ii) high correlations obtained among holistic and multi-trait scores were largely attributable to the impact of essay length on both holistic and multi-trait scoring, and (iii) some strong associations were confirmed between several "e-rater" variables and multi-trait rating dimensions. Implications are discussed for improving the multi-trait scoring of essays, refining "e-rater" essay feature variables, and validating automated essay scores.

This article presents the authors' response on the two commentators' (Fred Davidson and Charles Alderson) reviews of the 2009 Special Issue of "Language Assessment Quarterly" on "Cognitive Diagnosis and Q-matrices in Language Assessment." The Special Issue represents several years of research work on cognitive diagnosis approaches (CDA) carried out by the authors of the four main articles in the issue. The reviewers' comments assure the authors that they have accomplished their initial goals of informing and updating language testers of groundbreaking work on CDA and recent developments in CDA applications to language assessment. Their goals in this article are two-fold: (1) to respond as co-editors to the general comments made by the two commentators regarding the entire issue; and (2) to respond, as co-authors, of the first (Lee & Sawaki, 2009a) and fourth (Lee & Sawaki, 2009b) articles in the issue, to the specific concerns raised by Charles Alderson regarding these two articles.

A variety of cognitive diagnosis approaches (CDAs) have been used by researchers to extract fine-grained information about the linguistic knowledge states and skill mastery levels of test takers based on their performance on language tests. Although some of these approaches have been in use in the psychometric community for quite some time, many of these are relatively new to language testers and have not been widely researched or discussed in the field. The goal of this article is to describe how a CDA framework could be applied to language assessment. We start with a brief history and an overview of the major components of CDAs and then compare the characteristics of psychometric models for CDAs and discuss score interpretation and reporting issues. We conclude this article by outlining some of the challenges ahead and future research avenues for the application of CDAs in language assessment. (Contains 3 tables and 1 figure.)

The present study investigated the functioning of three psychometric models for cognitive diagnosis--the general diagnostic model, the fusion model, and latent class analysis--when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL[R] iBT (Test of English as a Foreign Language[TM]). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into "flat" profiles (i.e., "nonmasters of all skills" and "masters of all skills"). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well. (Contains 5 footnotes, 3 tables, and 6 figures.)

Possible integrated and independent tasks were pilot tested for the writing section of a new generation of the TOEFL[R] (Test of English as a Foreign Language[TM]). This study examines the impact of various rating designs and of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the perspective of generalizability theory (G-theory). Both univariate and multivariate G-theory analyses were conducted. It was found that (a) in terms of maximizing the score dependability, it would be more efficient to increase the number of tasks rather than the number of raters per essay; (b) two particular single-rating designs of "having different tasks for the same examinee rated by different raters" [p x (R:T), R:(p x T)] achieved relatively higher score dependability than other single-rating designs; and (c) a somewhat larger gain in composite score reliability was achieved when the number of listening-writing tasks was larger than that of reading-writing tasks. (Contains 4 figures, 6 tables and 8 footnotes.)

The objective of the present investigation was to examine the comparability of writing prompts for different gender groups in the context of the computer-based Test of English as a Foreign Language[TM] (TOEFL[R]-CBT). A total of 87 prompts administered from July 1998 through March 2000 were analyzed. An extended version of logistic regression for polytomous items was used to investigate both uniform and non-uniform gender effects. An English Language Ability variable was developed from the multiple-choice components of the TOEFL[R]-CBT examination and used as a matching variable. Initially, most of the prompts were flagged because of statistically significant uniform gender effects, with some prompts displaying non-uniform effects as well. Nevertheless, the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect. These findings are discussed in relation to prompt content review, gender format differences, and second language learning theories. (Contains 4 tables and 6 figures. Appended is: Logistic Regression Model for Polytomous Items: The Proportional Odds-Ratio Model.)

A multitask speaking measure consisting of both integrated and independent tasks is expected to be an important component of a new version of the TOEFL test. This study considered two critical issues concerning score dependability of the new speaking measure: How much would the score dependability be impacted by (1) combining scores on different task types into a composite score and (2) rating each task only once? To answer these questions, generalizability theory (G-theory) procedures were used to examine the impact of the numbers of tasks and raters per speech sample and of subsection lengths on the dependability of speaking scores. Univariate and multivariate G-theory analyses were conducted on rating data collected for 261 examinees for the study. The finding in the univariate analyses was that it would be more efficient to increase the number of tasks rather than the number of ratings per speech sample in maximizing the score dependability. The multivariate G-theory analyses also revealed that (1) the universe (or true) scores among the task-type subsections were very highly correlated and that (2) slightly larger gains in composite score reliability would result from increasing the number of listening-speaking tasks for the fixed section lengths.

This study has investigated the comparability of computer-based testing writing prompts in the Test of English as a Foreign LanguageTM (TOEFL) for examinees of different native language backgrounds. A total of 81 writing prompts introduced from July 1998 through August 2000 were examined using a 3-step logistic regression procedure for ordinal items. An English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scale scores. This ELA variable was used to match examinees of East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) language groups. Although about one third of the 81 prompts were initially flagged because of statistically significant group effects; the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect.