Search

This study investigates whether re-thinking the separation of lexis and grammar in language testing could lead to more valid inferences about proficiency across modes. As argued by Römer, typical scoring rubrics ignore important information about proficiency encoded at the lexis-grammar interface, in particular how the co-selection of lexical and grammatical features is mediated by communicative function. This is especially evident when assessing oral versus written exam tasks, where the modality of a task may intersect with register-induced variation in linguistic output. This article presents the results of an empirical study in which we measured the diversity and sophistication of four-word lexical bundles extracted from a corpus of French proficiency exams. Analysis revealed that the diversity of noun-based bundles was a significant predictor of written proficiency scores and the sophistication of verb-based bundles was a significant predictor of proficiency scores across both modes, suggesting that communicative function as well as the constraints of online planning mediated the effect of lexicogrammatical phenomena on proficiency scores. Importantly, lexicogrammatical measures were better predictors of proficiency than solely lexical-based measures, which speaks to the potential utility of considering lexicogrammatical competence on scoring rubrics.

Derivational morphology (DM) and how it can be assessed have been investigated relatively rarely in language learning and testing research. The goal of this study is to add to the understanding of the nature of DM knowledge, exploring whether and how it is separable from vocabulary breadth. Eight L2 (second or foreign language) English DM knowledge measures and three measures of the size of the English vocabulary were administered to 120 learners. We conducted two confirmatory factor analyses, one with one underlying factor and the other treating vocabulary breadth and DM as separate. As neither model had a satisfactory fit without introducing a residual covariance to the two-factor model, we conducted an exploratory factor analysis, which suggested two separate DM factors in addition to vocabulary breadth. Regardless, the analysis demonstrated that the DM knowledge was separate from learners' vocabulary breadth. However, learners' vocabulary breadth factor still explained a substantial amount of variance in learners' performance on DM measures. We discuss theoretical implications and implications for L2 assessment.

This study reviewed conceptualizations and operationalizations of second language (L2) listening constructs. A total of 157 peer-reviewed papers published in 19 journals in applied linguistics were coded for (1) publication year, author, source title, location, language, and reliability and (2) listening subskills, cognitive processes, attributes, and listening functions potentially measured or investigated. Only 39 publications (24.84%) provided theoretical definitions for listening constructs, 38 of which were general or had a narrow construct coverage. Listening functions such as discriminative, empathetic, and analytical listening were largely unattended to in construct conceptualization in the studies. In addition, we identified 24 subskills, 27 cognitive processes, and 54 listening attributes (total = 105) operationalized in the studies. We developed a multilayered framework to categorize these features. The subskills and cognitive processes were categorized into five principal groups each (10 groups total), while the attributes were divided into three main groups. This multicomponential framework will be useful in construct delineation and operationalization in L2 listening assessment and teaching. Finally, limitations of the extant research and future directions for research and development in L2 listening assessment are discussed.

In online environments, listening involves being able to pause or replay the recording as needed. Previous research indicates that control over the listening input could improve the measurement accuracy of listening assessment. Self-pacing also supports the second language (L2) comprehension processes of test-takers with specific learning difficulties (SpLDs) or, more specifically, of learners with reading-related learning difficulties who might have slower processing speed and limited working memory capacity. Our study examined how L1 literacy skills influence L2 listening performance in the standard single-listening and self-paced administration mode of the listening section of the Test of English as a Foreign Language (TOEFL) Junior Standard test. In a counterbalanced design, 139 Austrian learners of English completed 15 items in a standard single-listening condition and another 15 in a self-paced condition. L1 literacy skills were assessed via a standard reading, non-word reading, word-naming, and non-word repetition test. Generalized Linear Mixed-Effects Modelling revealed that self-pacing had no statistically significant effect on listening scores nor did it boost the performance of test-takers with lower L1 literacy scores indicative of reading-related SpLDs. The results indicate that young test-takers might require training in self-pacing or that self-paced conditions may need to be carefully implemented when they are offered to candidates with SpLDs.

This article describes the implementation of a special accommodations policy for a suite of localised English language and numeracy tests, the Workplace Literacy and Numeracy (WPLN) Assessments. The WPLN are computer-delivered assessments, part of the WPLN training and assessment programme, which exists to provide access to workforce skills training across the Singaporean population. Operational and interview data were analysed to investigate three main areas: the extent to which WPLN accommodations are considered appropriate and effective for the test-taker population; the impact of the accommodations on test-takers' future opportunities; and the main factors perceived by stakeholders as playing an important role in test accommodations. The findings indicate that while the accommodations provided are generally considered to be appropriate and effective for the diverse WPLN test-taker population and facilitate improved future educational and workplace opportunities, some areas of the process are problematic or worthy of further consideration. Specifically, recommendations are made for future improvement in special accommodations policy development, dissemination, and implementation in large-scale test systems to safeguard access and inclusion.

Published work on vocabulary assessment has grown substantially in the last 10 years, but it is still somewhat outside the mainstream of the field. There has been a recent call for those developing vocabulary tests to apply professional standards to their work, especially in validating their instruments for specified purposes before releasing them for widespread use. A great deal of work on vocabulary assessment can be seen in terms of the somewhat problematic distinction between breadth and depth of vocabulary knowledge. Breadth refers to assessing vocabulary size, based on a large sample of words from a frequency list. New research is raising questions about the suitability of word frequency norms derived from large corpora, the choice of the word family as the unit of analysis, the selection of appropriate test formats, and the role of guessing in test-taker performance. Depth of knowledge goes beyond the basic form-meaning link to consider other aspects of word knowledge. The concept of word association has played a dominant role in the design of such tests, but there is a need to create test formats to assess knowledge of word parts as well as a range of multi-word items apart from collocation.

Among the variety of selected response formats used in L2 reading assessment, multiple-choice (MC) is the most commonly adopted, primarily due to its efficiency and objectiveness. Given the impact of assessment results on teaching and learning, it is necessary to investigate the degree to which the MC format reliably measures learners' L2 reading comprehension in the classroom context. While researchers have claimed that the longer the reading test (i.e., more test items and passages), the higher its overall reliability, few studies have investigated the optimal number of items and passages required for reliable classroom-based L2 reading assessment. To address this research gap, I adopted generalizability (G) theory to investigate the score reliability of the MC format in classroom-based L2 reading tests. A total of 108 ESL students at an American college completed an English reading test that included four passages, each of which was accompanied by five MC comprehension questions. The results showed that the score reliability of the L2 reading test was critically influenced by the number of items and passages, inasmuch as a different combination of the number of passages and items altered the degree of reliability. Implications for practitioners and educational researchers are discussed.

As access to smartphones and emerging technologies has become ubiquitous in our daily lives and in language learning, technology-mediated social interaction has become common in teaching and assessing L2 speaking. The changing ecology of L2 spoken interaction provides language educators and testers with opportunities for renewed test design and the gathering of context-sensitive validity evidence of interactive speaking assessment. First, I review the current research on interactive speaking assessment focusing on commonly used test formats and types of validity evidence. Second, I discuss recent research that reports the use of artificial intelligence and technologies in teaching and assessing speaking in order to understand how and what evidence of interactive speaking is elicited. Based on the discussion, I argue that it is critical to identify what features of interactive speaking are elicited depending on the types of technology-mediated interaction for valid assessment decisions in relation to intended uses. I further discuss opportunities and challenges for future research on test design and eliciting validity evidence of interactive speaking using technology-mediated interaction.

Language proficiency testing serves an important function of classifying examinees into different categories of ability. However, misclassification is to some extent inevitable and may have important consequences for stakeholders. Recent research suggests that classification efficacy may be enhanced substantially using computerized adaptive testing (CAT). Using real data simulations, the current study investigated the classification performance of CAT on the reading section of an English language proficiency test and made comparisons with the paper-based version of the same test. Classification analysis was carried out to estimate classification accuracy (CA) and classification consistency (CC) by applying different locations and numbers of cutoff points. The results showed that classification was suitable when a single cutoff score was used, particularly for high- and low-ability test takers. Classification performance declined significantly when multiple cutoff points were simultaneously employed. Content analysis also raised important questions about construct coverage in CAT. The results highlight the potential for CAT to serve classification purposes and outline avenues for further research.

Instrument measurement conducted with Rasch analysis is a common process in language assessment research. A recent systematic review of 215 studies involving Rasch analysis in language testing and applied linguistics research reported that 23 different software packages had been utilized. However, none of the analyses were conducted with one of the numerous R-based Rasch analysis software packages, which generally employ one of the three estimation methods: conditional maximum likelihood estimation (CMLE), joint maximum likelihood estimation (JMLE), or marginal maximum likelihood estimation (MMLE). For this study, eRm, a CMLE-based R package, was utilized to conduct a dichotomous Rasch analysis of a Yes/No vocabulary test based on the academic word list. The resulting parameters and diagnostic statistics were compared with the equivalent results from four other R-based Rasch measurement software packages and Winsteps. Finally, all of the packages were utilized in the analysis of 1000 simulated datasets to investigate the extent to which results generated from the contrasting estimation methods converged or diverged. Overall, the differences between the results produced with the three estimation methods were negligible, and the discrepancies observed between datasets were attributable to the software choice as opposed to the estimation method.

Despite consistent calls for authentic stimuli in listening tests for better construct representation, unscripted texts have been rarely adopted in high-stakes listening tests due to perceived inefficiency. This study details how a local academic listening test was developed using authentic unscripted audio-visual texts from the local target language use (TLU) domain without compromising the reliability of the test results and validity of the score interpretations. The purpose of the listening test was to identify international students who need additional language support at a U.S. university. We show that efficiency persists when using authentic unscripted texts that are representative of the local context both at the test development phase and at the classification phase where placement decisions are made in a dependable manner. Expert judgments highlighted the improved correspondence of the listening test using locally sourced audio-visual texts to the local TLU domain, providing additional support for using the listening test for local placement purposes. Additionally, dimensionality assessments demonstrated that test design decisions inevitably entailed with using authentic unscripted texts did not threaten the internal structure of the test. We argue that local resources are indispensable in developing authentic test stimuli and in supporting the validity of local test interpretation and use.

Over the past decade, testing and assessing spoken-language interpreting has garnered an increasing amount of attention from stakeholders in interpreter education, professional certification, and interpreting research. This is because in these fields assessment results provide a critical evidential basis for high-stakes decisions, such as the selection of prospective students, the certification of interpreters, and the confirmation/refutation of research hypotheses. However, few reviews exist providing a comprehensive mapping of relevant practice and research. The present article therefore aims to offer a state-of-the-art review, summarizing the existing literature and discovering potential lacunae. In particular, the article first provides an overview of interpreting ability/competence and relevant research, followed by main testing and assessment practice (e.g., assessment tasks, assessment criteria, scoring methods, specificities of scoring operationalization), with a focus on operational diversity and psychometric properties. Second, the review describes a limited yet steadily growing body of empirical research that examines rater-mediated interpreting assessment, and casts light on automatic assessment as an emerging research topic. Third, the review discusses epistemological, psychometric, and practical challenges facing interpreting testers. Finally, it identifies future directions that could address the challenges arising from fast-changing pedagogical, educational, and professional landscapes.

In this study we investigated the potential for a shared-first-language (shared-L1) effect on second language (L2) listening test scores using differential item functioning (DIF) analyses. We did this in order to understand how accented speech may influence performance at the item level, while controlling for key variables including listening passages, item type, and the degrees of intelligibility, comprehensibility, and speaker accentedness. A total of 386 undergraduate and graduate students from China, Korea, and India, who were enrolled in a university in the United States, each took two listening tests. In the first session, they took a standardized listening comprehension test comprising texts recorded by native English speakers. In the second session, they took a listening comprehension test consisting of counterbalanced sets of American English-, Indian-, and Chinese-accented lectures. The results show that the shared-L1 effect is minimal. Effects are consistent for only a few narrow, detail-oriented items, on which Chinese and Korean listeners performed relatively poorly when listening to Indian speakers.

The last three decades have seen an increase of tests aimed at measuring an individual's vocabulary level or size. The target words used in these tests are typically sampled from word frequency lists, which are in turn based on language corpora. Conventionally, test developers sample items from frequency bands of 1000 words; different tests employ different sampling ratios. Some have as few as 5 or 10 items representing the underlying population of words, whereas other tests feature a larger number of items, such as 24, 30, or 40. However, very rarely are the sampling size choices supported by clear empirical evidence. Here, using a bootstrapping approach, we illustrate the effect that a sample-size increase has on confidence intervals of individual learner vocabulary knowledge estimates, and on the inferences that can safely be made from test scores. We draw on a unique dataset consisting of adult L1 Japanese test takers' performance on two English vocabulary test formats, each featuring 1000 words. Our analysis shows that there are few purposes and settings where as few as 5 to 10 sampled items from a 1000-word frequency band (1K) are sufficient. The use of 30 or more items per 1000-word frequency band and tests consisting of fewer bands is recommended.

Speech fluency has been extensively researched as a core construct for second language (L2) speaking assessment. Despite the broad consensus on its multifaceted nature, few researchers have empirically explored the dimensionality of this construct. Operationalizations of fluency vary across research and practice, using both holistic and fine-grained features. To address the dimensionality of speech fluency, in this study we examined an array of fluency features of speaking performances on the Aptis test. We conducted both univariate and multivariate statistical analyses to investigate the relationship between individual fluency features and language proficiency, as well as the relationships among fluency, complexity, and accuracy features. We found differences in the holistic and fine-grained fluency features, suggesting that they might reflect different dimensions of speech fluency and be associated with different components of language proficiency. Based on the findings, we labeled these two types of fluency features as macro and micro fluencies. Whereas macro fluency features tend to entail a holistic representation of fluency, micro fluency features tend to be more closely related to the automatic processing of lexico-grammar, constituting a more direct reflection of the cognitive processes in speech production. The findings support the multidimensionality of speech fluency and the need to include both macro and micro fluency features in the scoring, scale development, and validation of L2 speaking assessment.

This test review focuses on the current version (2009) of [Chinese characters omitted] (Hanyu Shuiping Kaoshi), literally translated as the Chinese Language Proficiency Test and abbreviated as HSK. Tailored to non-native speakers of the Chinese language, this test consists of six proficiency levels (Levels 1 and 2 as beginners, Levels 3 and 4 as intermediate, and Levels 5 and 6 as advanced), assessing listening, reading, and writing skills. The purpose of HSK is twofold. First, it aims to promote Chinese language teaching and learning globally. Second, it aims to provide a useful measure of test takers' Chinese language proficiency for making various decisions. The authors adopt Bachman and Palmer's (2010) Assessment Use Argument (AUA) to review and appraise the current version of HSK. AUA consists of a set of claims about test takers' performance on a language test (i.e., assessment tasks and assessment record) linked to the use of the test scores (i.e., interpretations, decisions, and consequences). By using an AUA, they are able to link all aspects of the current version of HSK and make a comprehensive review of the validity of the interpretation and use of the HSK test.

Vocabulary's relationship to reading proficiency is frequently cited as a justification for the assessment of L2 written receptive vocabulary knowledge. However, to date, there has been relatively little research regarding which modalities of vocabulary knowledge have the strongest correlations to reading proficiency, and observed differences have often been statistically non-significant. The present research employs a bootstrapping approach to reach a clearer understanding of relationships between various modalities of vocabulary knowledge to reading proficiency. Test-takers (N = 103) answered 1000 vocabulary test items spanning the third 1000 most frequent English words in the New General Service List corpus (Browne, Culligan, & Phillips, 2013). Items were answered under four modalities: Yes/No checklists, form recall, meaning recall, and meaning recognition. These pools of test items were then sampled with replacement to create 1000 simulated tests ranging in length from five to 200 items and the results were correlated to the Test of English for International Communication (TOEIC®) Reading scores. For all examined test lengths, meaning-recall vocabulary tests had the highest average correlations to reading proficiency, followed by form-recall vocabulary tests. The results indicated that tests of vocabulary recall are stronger predictors of reading proficiency than tests of vocabulary recognition, despite the theoretically closer relationship of vocabulary recognition to reading.

The main purpose of this study is to investigate guessing in the Yes/No (YN) format vocabulary test. One-hundred-and-five university students took a YN test, a translation task and a multiple-choice vocabulary size test (MC VST). With matched lexical properties between the real words and the pseudowords, pseudowords could index guessing in the YN test as correlations between false alarms and real word guessing were high (> 0.80). This finding provides important validity evidence for correction formulas that employ false alarm information to adjust the YN test scores. Another important finding is that the neighborhood size of pseudowords was related to false alarm rates, which has useful implications for the designing of future YN tests. Finally, the corrected scores of the YN test correlated highly (> 0.70) with those of the translation task. In comparison, correlations between the YN corrected scores and the MC VST scores were lower, indicating a difference in guessing between the two tests.

The American Council on the Teaching of Foreign Languages (ACTFL) oral proficiency interview -- computer (OPIc) testing system represents an ambitious effort in language assessment: Assessing oral proficiency in over a dozen languages, on the same scale, from virtually anywhere at any time. Especially for users in contexts where multiple foreign languages are relevant, this utility cannot be overstated. Quality assurance has not been unduly sacrificed in ACTFL's implementation of the OPIc, as shown by studies of rating quality. Although there are some concerns with the mechanism for form selection and the comparability of forms and prompts (within and across languages), ACTFL has largely succeeded in bridging the traditional strengths of the OPI with the practical affordances of the OPIc. ACTFL and Language Testing International (LTI) can also be commended for offering to the general public modular assessments of language ability (separate and independent tests of speaking, reading, writing, and listening), so that stakeholders who require measures of language skills à la carte can obtain them.

Reading comprehension tests are often assumed to measure the same, or at least similar, constructs. Yet, reading is not a single but a multidimensional form of processing, which means that variations in terms of reading material and item design may emphasize one aspect of the construct at the cost of another. The educational systems in Denmark, Norway, and Sweden share a number of traits, and in the recent decade, the development of national test instruments, especially for reading, has been highly influenced by international surveys of student achievement. In this study, national tests of L1 reading comprehension in secondary school in the three Scandinavian countries are compared in order to reveal the present range of diversity/commonality within the three test domains. The analysis employs both qualitative and quantitative aspects of data, including frameworks, text samples, task samples, and scoring guidelines from 2011 to 2014. Findings indicate that the three tests differ substantially from each other, not only in terms of the intentional and operative constructs of reading to be measured, but also in terms of testing methods and stability over time. Implications for the future development of reading comprehension assessment are discussed.

This study compared three common vocabulary test formats, the Yes/No test, the Vocabulary Knowledge Scale (VKS), and the Vocabulary Levels Test (VLT), as measures of vocabulary difficulty. Vocabulary difficulty was defined as the item difficulty estimated through Item Response Theory (IRT) analysis. Three tests were given to 165 Japanese students, resulting in five measures of vocabulary knowledge and four measures of word difficulty. Analyses included item and score factor analysis, unidimensionality, local independence, and correlations. Results indicate that these are reliable tests. Tests of unidimensionality suggest these tests are essentially measuring one major latent trait, which can be interpreted as a factor for word knowledge. Strong correlations of the scores with each other provide evidence of concurrent validity, and for the interpretation of the scores as indicative of word knowledge. Correlations with other methods of estimating word difficulty, such as transformed frequency, length of word, or number of syllables, suggest that of these methods, the log of frequencies from very large corpora gives the best estimate of word difficulty. However, direct testing of vocabulary difficulty appears to, in the words of Kreuz (1987), "provide a better account of recognition latencies than do counts based on printed word frequency" (p. 159).

The rise in the affordability of quality video production equipment has resulted in increased interest in video-mediated tests of foreign language listening comprehension. Although research on such tests has continued fairly steadily since the early 1980s, studies have relied on analyses of raw scores, despite the growing prevalence of item response theory in the field of language testing as a whole. The present study addresses this gap by comparing data from identical, counter-balanced multiple-choice listening test forms employing three text types (monologue, conversation, and lecture) administered to 164 university students of English in Japan. Data were analyzed via many-facet Rasch modeling to compare the difficulties of the audio and video formats; to investigate interactions between format and text-type, and format and proficiency level; and to identify specific items biased toward one or the other format. Finally, items displaying such differences were subjected to differential distractor functioning analyses. No interactions between format and text-type, or format and proficiency level, were observed. Four items were discovered displaying format-based differences in difficulty, two of which were found to correspond to possible acting anomalies in the videos. The author argues for further work focusing on item-level interactions with test format.

Implementing assessment reform can be challenging. Proposed new assessments must be seen by stakeholders to be fit for purpose, and sometimes the perceptions of key stakeholders, such as teachers and students, may differ from the assessment developers. This article considers the recent introduction of a new high-stakes assessment of spoken proficiency for students of foreign languages in New Zealand high schools. The new assessment aims to measure spoken proficiency through the recording of a range of unstaged peer-to-peer interactions as they take place throughout the year. It contrasts with an earlier assessment that drew on a summative teacher-led interview. The article presents findings from a survey of teachers (n = 152), completed two years into the assessment reform, in which teachers were asked to consider the relative usefulness of the two assessment formats. Findings suggest that teachers consider the new assessment to be, in most respects, significantly more useful than the earlier model, and that the new assessment is working relatively well. Some challenges emerge, however, in particular around the feasibility and fairness of collecting ongoing evidence of spontaneous peer-to-peer performance. Findings raise issues to be considered if the new assessment is to work more successfully.

Partial dictation is a measure of EFL listening proficiency that can be easily constructed, administered, and scored by EFL teachers. However, it is controversial whether this form of test measures lower-order abilities exclusively or involves both lower- and higher-order abilities. In order to answer this question, a study was designed to examine the difference between partial dictation and test forms believed to measure more higher-order abilities. In a series of confirmatory factor analyses, the simplex, second-order, and bi-factor models were fitted to the scores of 367 college-level EFL learners in China in a listening test composed of partial dictation, gap-filling and constructed response tasks. The bi-factor model was identified as the best-fitting and this supports the view that partial dictation measures the same construct as test forms believed to measure more higher-order abilities. Concomitant statistical analyses also showed that the partial dictation tasks were suited to the ability level of the test takers and had high internal consistency. (Contains 3 tables and 3 figures.)

We adapted three practice College Scholastic Ability Tests (CSAT) of English listening, each with five-option items, to create four- and three-option versions by asking 73 Korean speakers or learners of English to eliminate the least plausible options in two rounds. Two hundred and sixty-four Korean high school English-language learners formed three groups. Each took three of the nine tests, one with five-option items, one with four-, and one with three-, with administrations counterbalanced to control for order and practice effects. Mean test scores of the three-option tests were significantly higher than those of four- and five-option tests. While no difference was found in mean item discriminations across the three different test formats, reliability coefficients showed inconsistent patterns depending on the number of options and test versions. One possible interpretation of the low correlations among the scores of three test formats is that items with different numbers of options tap into skills other than listening. The findings suggest that statistically, three options may or may not be optimal depending on the point of view taken--from that of the test score users, or from that of the test stakeholders. Test developers must consider multiple statistical, affective, and contextual factors in determining the optimal number of options. (Contains 5 tables, 2 figures and 2 notes.)

Despite a number of research studies investigating the Yes-No vocabulary test format, one main question remains unanswered: What is the best scoring procedure to adjust for testee overestimation of vocabulary knowledge? Different scoring methodologies have been proposed based on the inclusion and selection of nonwords in the test. However, there is currently no consensus on the best adjustment procedure using these nonwords. Two studies were conducted to examine a new methodology for scoring Yes-No tests based on testees' response times (RTs) to the words in the test, on the assumption that faster responses would be more certain and accurate whereas more hesitant and inaccurate ones would be reflected in slower RTs. Participants performed a timed Yes-No test and were then interviewed to ascertain their actual vocabulary knowledge. Study 1 explored the viability of this approach and Study 2 examined whether the RT approach presented any advantage over the more traditional nonword approaches. Results showed that there was no clear advantage for any of the approaches under comparison, but their effectiveness depended on factors like the false alarm rate and the size of participants' overestimation of their lexical knowledge. (Contains 9 tables and 2 notes.)

The Assessment of Language Competence (ALC) certificates is an annual, international testing program developed by the Australian Council for Educational Research to test the listening and reading comprehension skills of lower to middle year levels of secondary school. The tests are developed for three levels in French, German, Italian and Japanese, and at two levels in Chinese and Indonesian. There is a mixture of target language and English questions in the Level 2 and 3 tests. Some teachers have raised this as a concern in the belief that all questions should only be offered in English for the sake of fairness. Their view is that the tests are unduly difficult when they are designed with questions in the language. Arising from this concern, the aim of the research to be reported in this paper was to investigate the effects of the language of the question on student performance. We drew on data from a trial test and a final listening test, a questionnaire administered to students to gauge their perceptions of the tests and an examination of public documentation about the ALC. For the statistical analysis, we used Item Response Theory for calibrating items and for comparing item difficulty estimates, and fit statistics to verify how well items with English and target language question formats worked together. We found that where the questions involved listening for simple, explicitly stated information, students found the items in the target language relatively easier. In questions that required students to listen for global meaning, language choice either did not matter or tended to favour items in English rather than the target language. Furthermore, each of the six tests with some items in English and others in the target language showed a high level of reliability and fit to the single latent scale, indicating that items were functioning consistently regardless of the language of the test question. (Contains 1 table, 5 figures and 3 notes.)

This study tracked gender differential item functioning (DIF) on the English subtest of the Korean College Scholastic Aptitude Test (KCSAT) over a nine-year period across three data points, using both the Mantel-Haenszel (MH) and item response theory likelihood ratio (IRT-LR) procedures. Further, the study identified two factors (i.e. reading strategy and perceived interest) that explained a portion of the variance in the magnitude of gender DIF via a series of multiple linear regression analyses. The results indicated (1) an interaction between item type and gender DIF and (2) a significant relationship between gender differences in the examinee's perceived interest in test items and the magnitude of gender DIF. The study discusses the results based on previous DIF research and presents pedagogical implications and some avenues for further research. (Contains 8 tables.)

The Modern Language Aptitude Test (Paper-and-Pencil Version, henceforth, the MLAT) measures "an individual's ability to learn a foreign language." It targets English-speaking adults (over Grade 9) who are literate. The test has only one form, which has not changed since it was first published by the Psychological Corporation in 1959. The test can be taken either in short or in long form. The long form consists of Parts I to V, requires a CD player, and takes about 70 minutes to complete. The short form consists of Parts III to V, does not require a CD player, and takes about 30 minutes to complete. Users can score the test using a hand-scoring stencil. Information about the mean scores and percentile rankings of different types of learners is available in the manual. With the deaths of the first author, Dr. John B. Carroll, in 2003, and the second author, Dr. Stanley M. Sapon, in 2010, the 53-year-old test has now lost both of its creators. Given that the single original form of the MLAT has remained unchanged since its first release in 1959, it is truly remarkable that the test is still widely used for both practical and research purposes. This review presents the test in historical perspective, bridging its past, present, and future. (Contains 1 table.)

Pictures are widely used to elicit expressive language skills, and pictures must be established as parallel before changes in ability can be demonstrated by assessment using pictures prompts. Why parallel prompts are required and what it is necessary to do to ensure that prompts are in fact parallel is not widely known. To date, evidence of equivalence has been documented for only a handful of picture instruments. The present study aims to demonstrate that two picture series, one describing hiking on a mountain, the other a picnic at a beach, designed to elicit narrative writing from children, are statistically parallel. To verify their being parallel, a random group design and a covariate were used. MANCOVA and confirmatory factor analysis confirmed that the two alternate picture forms "were parallel" in terms of means, variances, and factor structures with regard to multiple writing components. This study makes a unique contribution to picture-based assessment, first by publishing this parallel picture set and making it available as a tool for assessing children's productive skills using longitudinal designs, and second by demonstrating a process by which any other test forms, pictorial or otherwise, can be validated as parallel. (Contains 4 figures, 6 tables and 8 notes.)

A meta-analysis was conducted on the effects of multiple-choice and open-ended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items. (Contains 5 tables.)

In this article, the author reviews the TOEFL iBT which is the latest version of the TOEFL, whose history stretches back to 1961. The TOEFL iBT was introduced in the USA, Canada, France, Germany and Italy in late 2005. Currently the TOEFL test is offered in two testing formats: (1) Internet-based testing (iBT); and (2) paper-based testing (PBT). The TOEFL iBT measures the ability of non-native speakers of English to use and understand English as it is spoken, written and heard in academic settings. Crucially, TOEFL iBT is in many ways radically different from earlier versions of TOEFL, in that it is said to be "better aligned to the variety of language use tasks that examinees are expected to encounter in everyday academic life." It includes mandatory speaking and writing sections, and integrated tasks that require test takers to read a text, listen to a lecture or conversation and write or speak a response to what has been read or heard. The separate test of Structure and Written Expression is no longer given, and test takers read or listen to longer written and spoken texts in the reading and listening sections and are allowed to take and refer to notes. TOEFL iBT is delivered via an Internet-based delivery system in Educational Testing Service (ETS)-certified test centres and candidates use the keyboard to input their answers, including writing tasks, and a microphone to record their responses to speaking tasks. The test is linear, not computer-adaptive, and the listening and reading sections have both a long version and a short version. The long version includes experimental, pre-test, research, or equating questions. Each test taker receives a long version of either listening or reading, but not both, thus ensuring that all test takers are given tests that are comparable in length. As it is impossible to tell which questions are experimental, candidates have to do their best on all. TOEFL iBT represents a major step forward in testing language proficiency for academic purposes. Challenges remain, inevitably, in terms of the low reliability of the Writing test (the addition of e-rater[R] may enhance reliability), possible construct-irrelevant variance in the Reading and Listening tests, and construct under-representation in the Speaking and Writing tests, but a continuing programme of research will hopefully continue to refine and improve the test further. (Contains 3 footnotes.

Speaking tasks involving peer-to-peer candidate interaction are increasingly being incorporated into language proficiency assessments, in both large-scale international testing contexts, and in smaller-scale, for example course-related, ones. This growth in the popularity and use of paired and group orals has stimulated research, particularly into the types of discourse produced and the possible impact of candidate background factors on performance. However, despite the fact that the strongest argument for the validity of peer-to-peer assessment lies in the claim that such tasks allow for the assessment of a broader range of interactional skills than the more traditional interview-format tests do, there is surprisingly little research into the "judgments" that are made of such performances. The fact that raters, and rating criteria, are in a crucial mediating position between output and outcomes, warrants investigation into how raters construe the interaction in these tasks. Such investigations have the potential to inform the development of interaction-based rating scales and ensure that validity claims are moved beyond the content level to the construct level. This paper reports the findings of a verbal protocol study of teacher-raters viewing the paired test discourse of 17 beginner dyads in a university-based Spanish as a foreign language course. The findings indicate that the raters identified three interaction parameters: non-verbal interpersonal communication, interactive listening, and interactional management. The findings have implications for our understanding of the construct of effective interaction in paired candidate speaking tests, and for the development of appropriate rating scales. (Contains 2 figures and 3 footnotes.)

In 2002 the University of Cambridge Local Examinations Syndicate (UCLES) implemented a revised version of the Certificate of Proficiency in English (CPE). CPE, which is the highest level of the Main Suite of Cambridge ESOL exams, comprises five modules, "Reading," "Writing," "Use of English," "Listening" and "Speaking," the latter of which is the focus of this review. Among the innovations introduced in the revised CPE exam were the introduction of a paired speaking format with two candidates and two examiners, revised tasks including a collaborative task carried out by the candidates together, a script for the participating examiner to follow, and revised assessment scales. The CPE speaking test consists of three parts and takes 19 minutes for a pair of candidates. The test tasks increase in difficulty as the test proceeds and three types of stimulus are used: (1) verbal (all parts, but exclusively in Part 1); (2) visual (Part 2); and (3) written (Part 3). In the test as a whole, the construct consists of three main interactional modes: (1) "question and answer" (examiner to candidate/s); (2) "goal-oriented conversation" (candidate to candidate); and (3) "long turn" (candidate to examiner and other candidate). These are characterized by different turn-taking patterns and power relationships. The authors conclude that the test review process was careful, lengthy, and consultative. There are a number of commendable additions in the revised test--the use of two examiners, the enhanced reliability afforded by the examiner script, the appeal of the paired format among test-takers, and the variety of interaction patterns and language functions--but there are also areas where ongoing validation work should focus. These include: (1) continued research on the effect of different pairings on candidates' scores; (2) a clear articulation of how the paired-candidate format broadens the speaking construct; and (3) a re-evaluation of the separability of the assessment criteria. (Contains 6 footnotes.)

This study, framed within sociocultural theory, examines the interaction of adult ESL test-takers in two tests of oral proficiency: one in which they interacted with an examiner (the individual format) and one in which they interacted with another student (the paired format). The data for the eight pairs in this study were drawn from a larger study comparing the two test formats in the context of high-stakes exit testing from an Academic Preparation Program at a large Canadian university. All of the test-takers participated in both test formats involving a discussion with comparable speaking prompts. The findings from the quantitative analyses show that overall the test-takers performed better in the paired format in that their scores were on average higher than when they interacted with an examiner. Qualitative analysis of the test-takers' speaking indicates that the differences in performance in the two test formats were more marked than the scores suggest. When test-takers interacted with other students in the paired test, the interaction was much more complex and revealed the co-construction of a more linguistically demanding performance than did the interaction between examiners and students. The paired testing format resulted in more interaction, negotiation of meaning, consideration of the interlocutor and more complex output. Among the implications for test theory and practice is the need to account for the joint construction of performance in a speaking test in both construct definitions and rating scales. (Contains 2 figures, 4 tables, and 5 footnotes.)

The assessment of so-called depth of word knowledge has been the focus of research for some years now. In this article the construct of deep word knowledge is further specified as the decontextualized knowledge of word meanings and word associations. Most studies so far have involved adolescent and adult second language learners. In this article, we extend the focus to younger first and second language learning children in primary school in the Netherlands. A well-established test format involving word associations has been adapted for this younger target group and the feasibility of assessing deep word knowledge in primary school children is evaluated empirically. The results showed that the test provides a reliable and efficient method of assessing the children's vocabulary knowledge. Evidence of validity was obtained from a concurrent definition task, and from an analysis of the scores in terms of language background, which is known to be associated with different levels of vocabulary knowledge. (Contains 1 figure, 6 tables, and 9 footnotes.)

Some educational contexts almost mandate the application of multiple-choice (MC) testing techniques, even if they are deplored by many practitioners in the field. In such contexts especially, research into how well these types of item perform and how their performance may be characterised is both appropriate and desirable. The focus of this paper is on a modified type of MC item dubbed "multitrak", which was used to test grammar in the Hungarian national admissions test for English between 1993 and 2004. Multitrak items were evaluated in relation to three better-known MC item types and were shown to provide more information about candidates in the ability range represented by Levels B2 and C1 on the Common European Framework of Reference (CEFR), as well as allowing a focus on more difficult content than the other item types in the study. (Contains 9 tables and 3 footnotes.)

This study examined the multiple true-false (MTF) test format in second language testing by comparing multiple-choice (MCQ) and multiple true-false (MTF) test formats in two language areas of general English: vocabulary and reading. Two counter-balanced experimental designs--one for each language area--were examined in terms of the number of MCQ and MTF differentially responded to, MTF item dependency, reliability, and concurrent validity. The data were analysed by classical test theory (CTT) and Rasch analysis. The results showed a two- and three-fold increase in vocabulary and reading items answered, respectively. Participants responded to significantly more MTF items than MCQ, and further analysis revealed no item dependency for both language domains. Reliability increases were found in the reading tests. Item conversions did not alter the basic functioning of the MTF items, and common person equating plots demonstrated a steady relationship between MCQ and MTF person ability estimates.

Examination boards are often criticized for their failure to provide evidence of comparability across forms, and few such studies are publicly available. This study aims to investigate the extent to which three forms of the General English Proficiency Test Intermediate Speaking Test (GEPTS-I) are parallel in terms of two types of validity evidence: parallel-forms reliability and content validity. The three trial test forms, each containing three different task types (read-aloud, answering questions and picture description), were administered to 120 intermediate-level EFL learners in Taiwan. The performance data from the different test forms were analysed using classical procedures and Multi-Faceted Rasch Measurement (MFRM). Various checklists were also employed to compare the tasks in different forms qualitatively in terms of content. The results showed that all three test forms were statistically parallel overall and Forms 2 and 3 could also be considered parallel at the individual task level. Moreover, sources of variation to account for the variable difficulty of tasks in Form 1 were identified by the checklists. Results of the study provide insights for further improvement in parallel-form reliability of the GEPTS-I at the task level and offer a set of methodological procedures for other exam boards to consider.

Studies of placement tests are typically narrowly concerned with their validation as instruments for the efficient grouping of students. They rarely explore the assumption that placement test content can be related to classroom tasks and so inform instructional decisions. This study focuses on a trial version of the Global Placement Test (GPT), a measure of grammatical knowledge being developed for use in language schools worldwide. The central concern is the extent to which the GPT can enable schools to make valid inferences regarding mastery of the grammatical structures to be taught on the relevant courses and hence guide diagnostic intervention. In this study the GPT, together with another test specifically designed to measure knowledge of grammatical items expressing temporal relations in English, was administered to 1070 EFL (English as a Foreign Language) learners studying in the UK, Greece and Japan. The dimensionality of the GPT as a measurement scale was investigated by IRT and factor-analytic methods. The analysis suggests that item difficulty is affected more by item type than by any inherent linguistic difficulty of the element of grammatical competence being tested. The results cast doubt on whether quick grammar-based placement tests, which traditionally emphasize efficiency and practicality, can provide sufficient information on which to base diagnostic mediation or sequencing of instructional materials.

Investigates the effects of text organization and response format on second language learners' performance on reading comprehension tests. Analyzes the results of reading comprehension tests that were delivered to Japanese University students. Found that text organization and test format had a significant impact on students' performance. (Author/VWL)

Evaluates the characteristics of Yes/No tests as a measure for receptive vocabulary size in a second language (L2). The evaluation was conducted both on theoretical grounds as well as on the basis of a large corpus of data collected with French learners of Dutch. Focuses on the internal qualities of the format in comparison with other more classical test formats. (Author/VWL)

Suggests that performance tests as currently represented in the American Council on the Teaching of Foreign Languages (ACTFL)-Oral Proficiency Interview (OPI) may not adequately address the basic concerns brought about by the perceived shortcomings of academic second language programs. Supports this argument with a critical analysis of the ACTFL proficiency guidelines for speaking and the ACTFL-OPI Tester lines for speaking. (Author/VWL)

Describes a study of multiple-choice and open-ended questions, each presented in the native language and in the second language, on the same second language text. Different methods resulted in different scores. Some methods were found to be more difficult than others and to have a greater effect on students of low-level proficiency. (SED)

Discusses authenticity in language testing in relation to the language proficiency movement. Looks at both sociolinguistic and psychometric principles as they are concerned with authenticity and validity as well as the inferential distance that separates face validity from techical validities. Criticizes the belief that some test type possess inherent validity. (SED)

Studies the application of the letter-deletion procedure (LDP), which matches the advantages of both rational deletion and C-Tests and avoids their disadvantages to English-as-a-Second-Language tests. LDP flexibly reduces redundancy in a text by deleting letters in item words according to specified principles. (Author/LMO)

Compares student attitudes toward oral or written tests and the relationship of attitude to performance of 170 students enrolled in the advanced reading course of English as a foreign language at Haifa University (Israel). Students preferred written over oral tests. No meaningful relationship was observed between students' attitudes and test performance. (Author/LMO)

Investigates the relationship between field independence and language measures. Results indicate varying relationships of field independence with cloze, dictation, and multiple-choice language tests. These relationships also differ for native speakers in regular or remedial English classes, and for nonnative speakers. 53 references cited. Cloze and dictation tests are appended. (Author/LMO)