Search

The use of common tasks and rating procedures when assessing the communicative skills of students from highly diverse linguistic and cultural backgrounds poses particular measurement challenges, which have thus far received little research attention. If assessment tasks or criteria are found to function differentially for particular subpopulations within a test candidature with the same or a similar level of criterion ability, then the test is open to charges of bias in favour of one or other group. While there have been numerous studies involving dichotomous language test items (see e.g. Chen and Henning, 1985 and more recently Elder, 1996) few studies have considered the issue of bias in relation to performance based tasks which are assessed subjectively, via analytic and holistic rating scales. The paper demonstrates how Rasch analytic procedures can be applied to the investigation of item bias or differential item functioning (DIF) in both dichotomous and scalar items on a test of English for academic purposes. The data were gathered from a pilot English language test administered to a representative sample of undergraduate students (N= 139) enrolled in their first year of study at an English-medium university. The sample included native speakers of English who had completed up to 12 years of secondary schooling in their first language (L1) and immigrant students, mainly from Asian language backgrounds, with varying degrees of prior English language instruction and exposure. The purpose of the test was to diagnose the academic English needs of incoming undergraduates so that additional support could be offered to those deemed at risk of failure in their university study. Some of the tasks included in the assessment procedure involved objectively-scored items (measuring vocabulary knowledge, text-editing skills and reading and listening comprehension) whereas others (i.e. a report and an argumentative writing task) were subjectively-scored. The study models a methodology for estimating bias with both dichotomous and scalar items using the programs Quest (Adams and Khoo, 1993) for the former and ConQuest (Wu, Adams and Wilson, 1998) for the latter. It also offers answers to the practical questions of whether a common set of assessment criteria can, in an academic context such as this one, be meaningfully applied to all subgroups within the candidature and whether analytic criteria are more susceptible to biased ratings than holistic ones. Implications for test fairness and test validity are discussed.

Applied linguists have developed complex theories of the ability to communicate in a second language (L2). However, the perspectives on L2 communication ability of speakers who are not trained language professionals have been incorporated neither into theories of communication ability nor in the criteria for assessing performance on general-purpose oral proficiency tests. This potentially weakens the validity of such tests because the ultimate arbiters of L2 speakers' oral performance are not trained language professionals. This study investigates the perspectives of these linguistic laypersons on L2 communication ability. Twenty-three native and non-native English-speaking linguistic laypersons judged L2 speakers' oral performances and verbalized the reasons for their judgments. The results showed that the participants focus not only on the linguistic aspects of the speaker's output but also on features that applied linguists have less paid attention to. Even where speaker's linguistic errors were acknowledged, message conveyance and comprehensibility of the message contributed to their judgment. The study has implications for language testing and the development of tests reflecting the construct of English as a lingua franca.

This article addresses the suitability of the CEFR as the basis for decisions about the readiness of individuals to engage in academic writing tasks in undergraduate university courses, and as a guide to progress. The CEFR offers potentially relevant general scales and subscales, but also more specific subscales for writing in the academic context. However, recent challenges to traditional views of academic writing have potential implications for assessment frameworks such as the CEFR when they are used to identify readiness for, and progress in, academic study. In this article we explore the views of students on what it means to "do" academic writing. Questionnaires, interviews, and short reflective texts were used to investigate the changing perceptions of first-year undergraduate students at an Australian university. The analysis of student data confirms the reality of the more complex view of academic writing suggested by the recent literature. The article then considers what implications this has for the adequacy of the definitions provided in the CEFR. It suggests that the CEFR descriptors underrepresent the complexity of the challenges of academic writing, particularly its cognitive demands. A new and rather different approach will be required to inform assessments used to manage the admission of students in`to academic writing contexts and the monitoring of their progress.

This report documents two coordinated exploratory studies into the nature of oral English-for-academic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine field experts? rating orientations, and Study II investigated the quality of test-taker discourse on two different Test of English as a Foreign Language? (TOEFL®) task types (independent and integrated) at different levels of proficiency. Study I showed that, with no guidance, domain experts distinguished and described qualitatively different performances using a common set of criteria very similar to those included in draft rating scales developed for the tasks at ETS. Study II provided empirical support for the criteria applied by the judges. The findings indicate that raters take a range of performance features into account within each conceptual category and that holistic ratings are driven by all of the assessment categories rather than, as has been suggested in earlier studies, predominantly by grammar.

This article brings together the theoretical field of language testing and the practical field of language analysis for the determination of the origin of asylum seekers. It considers what it would mean to think of language analysis as a form of language test, subject to the same validity constraints, and proposes a research agenda.

Gaining insights from domain experts into how they view communication in real world settings is recognized as an important authenticity consideration in the development of criteria to assess language proficiency for specific academic or occupational purposes. These "indigenous" criteria represent an articulation of the test construct and should therefore reflect what is germane to the particular domain of language use rather than general language-focused criteria familiar from other language tests. The methodological question of how to elicit such insights is, however, complex and has been addressed by various researchers using different methodological and theoretical frameworks.

This paper explores the views of nursing and medical domain experts in considering the standards for a specific-purpose English language screening test, the Occupational English Test (OET), for professional registration for immigrant health professionals. Since individuals who score performances in the test setting are often language experts rather than domain experts, there are possible tensions between what is being measured by a language test and what is deemed important by domain experts. Another concern is a lack of qualitative research on the process of the standard setting. To date, no published qualitative work has been identified about the contributions of domain experts in the standard setting for healthcare communication. In this study, a standard-setting exercise was conducted for the speaking component of the OET, using judgements of nursing and medical clinical educators and supervisors. In all, 13 medical and 18 nursing clinical educators and supervisors rated medical and nursing candidate performances respectively. These performances were audio-recorded OET role-plays that were selected across a range of proficiency levels. Domain experts were invited to comment on the basis of their decisions and the extent of alignment between these decisions and the criteria used to assess performance on the OET. Nursing and medical domain experts showed that they attended to all of the OET criteria in making their decisions about standards. However, clinical scenario simulation also invited judgements of clinical competence from participants, even where they knew that clinical competence should be excluded from their decision-making. Another concern related to the authenticity limitations of the role-play tasks as evidence of readiness to handle communication in the workplace. Overall, findings support the value of qualitative evidence from the standard setting in providing insight into the factors informing and impeding decision-making.

This paper considers how to establish the minimum required level of professionally relevant oral communication ability in the medium of English for health practitioners with English as an additional language (EAL) to gain admission to practice in jurisdictions where English is the dominant language. A theoretical concern is the construct of clinical communicative competence and its separability (or not) from other aspects of professional competence, while a methodological question examines the technical difficulty of determining a defensible minimum standard. The paper reports on a standard-setting study to set a minimum standard of professionally relevant oral competence for three health professions--medicine, nursing, and physiotherapy--as measured by the speaking sub-test of the Occupational English Test, a profession-specific test of clinically related communicative competence. While clinical educators determined the standard, it is to be implemented by raters trained as teachers of EAL; therefore, the commensurability of the views of each group is a central issue. This also relates to where the limits of authenticity lie in the context of testing language for specific purposes: to represent the views of domain experts, a sufficient alignment of their views with scores given by the raters of test performances is vital. The paper considers the construct of clinical communicative competence and describes the standard-setting study, which used the analytical judgement method. The method proved successful in capturing sufficiently consistent judgements to define defensible standards. Findings also indicate that raters can act as proxies for occupational experts, although it remains unclear whether the views of performances held by these two groups are directly comparable. The new minimum standards represented by the cut scores were found to be somewhat harsher than those in current use, particularly in medicine.

Those who work in second- and foreign-language testing often find Koretz's concern for validity inferences under high-stakes (VIHS) conditions both welcome and familiar. While the focus of the article is more narrowly on the potential for two instructional responses to test-based accountability, "reallocation" and "coaching," to undermine the validity of score-based inferences about achievement by inflating scores and exaggerating mastery of the domain of interest, rather than the broader set of problems around test-based accountability (Haertel, 2013), the issues raised have also been extensively discussed for several years in the field of language testing, using somewhat different terminology. "Washback" (or "backwash") is the specific term used to refer to the effects of high-stakes assessment on teaching and learning and is considered a subset of test impact more generally (Cheng, 2008). The inflationary or deflationary effect of nonsubstantive (construct irrelevant) "performance elements" on scores (associated with coaching in the article) is known as "test method effect" (Bachman, 1990). The context of the article, where outcomes-based control of educational effort is found to have a distorting effect on teaching and learning efforts, has many parallels in language testing, not only in the United States, where the negative impact on bilingual education of test-based accountability under No Child Left Behind and more recently the Common Core Standards has been widely discussed (Menken, 2008), but internationally, particularly in studies of the problematic impact of the Common European Framework of Reference (Council of Europe 2001) and PISA testing on language education (McNamara, 2011). Issues around validity inferences under high-stakes conditions arise in language testing in more contexts than the kind of test based accountability scenarios that are the focus of this article. Since the appearance of a special volume in "Language Testing" (the leading journal in the field) featuring an article on washback by Messick (1996), there has been a significant amount of research undertaken on washback and the broader impacts of language assessment, including a suite of studies sponsored by the major language testing agencies themselves. However, the high-stakes language assessments forming the focus of these studies are typically used for selection and/or accreditation purposes rather than for school and teacher accountability: language tests act as gatekeeping devices to control access to employment, education, migration, and citizenship. There is a general consensus in language-testing research that high-stakes language assessment influences "what" is taught but not necessarily "how" it is taught.

The biblical story of the shibboleth is widely cited in language testing as emblematic of the social and political function of language tests. But the meaning of the shibboleth has also been explored within poststructuralism, specifically within Derrida's discussion of the dilemmas of identity in the work of the German Jewish poet Paul Celan. Derrida discusses language itself as shibboleth, and emphasizes the ambiguity and indeterminacy of the linguistic sign, its "undecidability". This article considers the implications of Derrida's argument for the area of language testing in order to see what his interpretation of the shibboleth might mean for understanding its practices, typically framed as they are within a modernist paradigm. Examples are drawn from various areas of language assessment, including the use of language analysis in the determination of the claims of asylum seekers. What implications for understanding such assessment practices does a poststructuralist perspective offer?

This paper examines the uptake of Rasch measurement in language testing through a consideration of research published in language testing research journals in the period 1984 to 2009. Following the publication of the first papers on this topic, exploring the potential of the simple Rasch model for the analysis of dichotomous language test data, a debate ensued as to the assumptions of the theory, and the place of the model both within Item Response Theory (IRT) more generally and as appropriate for the analysis of language test data in particular. It seemed for some time that the reservations expressed about the use of the Rasch model might prevail. Gradually, however, the relevance of the analyses made possible by multi-faceted Rasch measurement to address validity issues within performance-based communicative language assessments overcame language testing researchers' initial resistance. The paper outlines three periods in the uptake of Rasch measurement in the field, and discusses the research which characterized each period. (Contains 4 tables and 12 notes.)

This paper presents a comprehensive framework for researching classroom-based assessment (CBA) processes, and is based on a detailed empirical study of two Australian school classrooms where students aged 11 to 13 were studying Indonesian as a foreign language. The framework can be considered innovative in several respects. It goes beyond the scope of earlier models in addressing a number of gaps in previous research, including consideration of the epistemological bases for observed assessment practices and a specific learner and learning focus. Moreover, by adopting the broadest possible definition of CBA, the framework allows for the inclusion of a diverse range of data, including the more intuitive forms of teacher decision-making found in CBA (Torrance & Pryor, 1998). Finally, in contrast to previous studies the research motivating the development of the framework took place in a school-based foreign language setting. We anticipate that the framework will be of interest to both researchers and classroom practitioners. (Contains 8 tables, 6 figures and 2 notes.)

The suggested role for assessment in developing "Roadmaps for Learning" has potentially important implications for the learning of second or foreign languages in school, a major concern of applied linguistics. In this response, the authors consider how the findings of a detailed ethnographic study of classroom-based assessment in two foreign language classrooms in Australia can illuminate a number of the issues and suggestions in the article. They also evaluate the success of attempts within language testing so far to map growth along the lines suggested in the article, and the implications this has for the applicability of the suggested approach to "skill-oriented" areas of the curriculum (such as proficiency in the practical use of languages, the current target of school languages curricula), as distinct from the "knowledge-oriented" areas of the curriculum (such as mathematics or science). (Contains 1 table.)

A feature of language assessment internationally is its role in the enforcement of language policies established by governments and other educational and cultural agencies. This trend has led to the near-universal adoption of curriculum and assessment frameworks, the clearest example of which is the Common European Framework of Reference for Languages (CEFR). What does this movement represent? What is the source of the authority of frameworks such as the CEFR, and why are they so appealing to governments? The paper argues that the determination of test constructs within policy-related frameworks leads to inflexibility, and considers the case of the testing of competence in English as a lingua franca (ELF) communication, in the context of international civil aviation communication.

The paper by Wilson and Moore (this volume), based on the Messick Lecture delivered in 2006 at the annual Language Testing Research Colloquium in Melbourne, may present a familiar challenge to some language testers: of reading outside one's comfort zone. The distinctive character of language testing lies in its combination of two primary fields of expertise: (1) applied linguistics; and (2) measurement. The tradition of the Messick Lecture at the annual Language Testing Research Colloquium (LTRC) has been to invite those with expertise in measurement, often from outside language testing, to share their current work with language testing practitioners. This paper is admirably clear and patient in its setting forth of the logic and detail of the psychometric models it presents, and illustrating these with data from language testing. On a technical note, the paper gives a lucid and practically useful discussion of effect sizes in Rasch analyses, and also helps one to understand the point and the logic of model goodness of fit tests for language test data. It is also important to note that the logic of the approach outlined in the paper is not restricted to Rasch models but applies to non-Rasch models as well, including two-parameter logistic models and multidimensional models. The extensive research effort within language testing to provide empirical support for the practice of content-referencing of language tests has clear implications for the validity of content-referencing more generally, and deserves to be more widely known. It also demonstrates the advantage of ongoing dialogue between subject matter assessment specialists, such as language testers, and psychometricians, of the kind triggered by Wilson and Moore's stimulating paper. (Contains 1 note.)

Discussions of multilingualism in education, as exemplified by the articles in this issue, can be critically reevaluated using perspectives available from poststructuralism. These perspectives focus on the potential ambiguity of language and language practices. This, in turn, encourages us to question simple notions of the relationship of learners to the languages they speak, especially the "mother tongue," to see the individual's relation to language as a relation to power, and to recognize the polyvalent role of language tests in the context of multilingual education as, on the one hand, enforcing the relations of power in language and, on the other, disrupting them. The article focuses on the extent to which these themes are acknowledged in the articles in this issue, exploring the impact of the contrasting contexts represented, and presents examples of other discussions, especially in colonial and postcolonial contexts, that reflect alternative views of multilingualism in education. These include Derrida's reflections on his own language socialization as a colonial subject in Algeria during the time of the Vichy regime and studies of the role of language in contemporary education systems in Africa and Southeast Asia.

This article proposes a distinction between "fairness" and "justice" in relation to language tests. Basing its discussion on the validity theory of Samuel Messick, it discusses the way in which these terms have been used in the existing literature, and their relationship to the concept of validity. Fairness, broadly speaking, refers to the technical quality of the test; justice encompasses the values implicit in test constructs, and the social uses to which language tests may be put. It illustrates the potential usefulness of the distinction by discussing the growing international practice of the use of language tests within citizenship procedures, and considers in detail the introduction of a formal citizenship test in Australia by a conservative government in the context of growing public anxiety about security, and about cultural diversity within the Australian population. (Contains 5 footnotes, 2 tables, and 1 figure.)

In its late colonial history and early years as an independent nation, Australia practised a policy of ruthless exclusion of immigrants on the basis of race by means of a language test: the notorious Dictation Test. In the 50 years following World War II, Australia adopted policies encouraging immigration with bipartisan political support. However, following the election of John Howard as Prime Minister in 1996, immigration was again politicised in the context of the emergence of an anti-immigrant right wing populist movement, which threatened Howard's political base. This, plus the security anxieties following 9/11 and heightened intergroup tensions within Australia itself, led to a renewal of discourses of national belonging and exclusion. One result was the introduction of a new linguistically demanding citizenship test in 2007, again with bipartisan support. The role of language testers in the light of such policy developments is considered. (Contains 7 footnotes.)

The study reported in this paper is an investigation of the nature of speaking proficiency in English as a second language in the context of a larger project to develop a rating scale for a new international test of English for Academic Purposes, TOEFL iBT (Brown et al. 2005). We report on a large-scale study of the relationship between detailed features of the spoken language produced by test-takers and holistic scores awarded by raters to these performances. Spoken test performances representing five different tasks and five different proficiency levels (200 performances in all) were analyzed using a range of measures of grammatical accuracy and complexity, vocabulary, pronunciation, and fluency. The results showed that features from each category helped distinguish overall levels of performance, with particular features of vocabulary and fluency having the strongest impact. Overall, the study contributes important insights into the nature of spoken proficiency as it develops and can be measured in rating scales for speaking, and has implications for methodological issues of the appropriateness of the use in language testing research contexts of measures developed in research on second language acquisition.

The thought of Samuel Messick has influenced language testing in 2 main ways: in proposing a new understanding of how inferences made based on tests must be challenged, and in drawing attention to the consequences of test use. The former has had a powerful impact on language-testing research, most notably in Bachman's work on validity and the design of language tests. Messick's writing on test consequences has informed debate on ethics, impact, accountability, and washback in language testing in the work of several researchers. But the character of Messick's work challenges us in many additional ways. Messick located validity theory in the area of values. This article explores the implications of this position, highlighting the social construction of language test constructs. Language test constructs are increasingly the target of policy, a development that threatens to render traditionally conceived validation work of only marginal relevance. The less obvious, covert social construction of language test constructs is explored in the light of Butler's theory of performativity. The article concludes with a consideration of recent adaptations of Messick's work in the influential validation models of Mislevy and Kane, and considers their failure to address questions of values and the social context of assessment properly. Tackling these questions is the ongoing challenge of Messick's legacy. (Contains 4 footnotes and 2 tables.)

Focuses on the increase in discourse-based studies of oral proficiency assessment techniques. Discusses research carried out on a number of factors in the assessment setting, including the role of interlocutor, candidate, and rater, and the impact of tasks, task performance conditions, and rating criteria. (Author/VWL)

Investigated the impact of performance conditions on perceptions of task difficulty in a test of spoken language, in light of the cognitive complexity framework proposed by Skehan (1998). Candidates performed narrative tasks whose characteristics and the conditions under which they were performed were manipulated, and the impact of these on task performance was analyzed. (Author/VWL)

Argues that a growing awareness of the fundamentally social character of language assessment challenges a rethinking of priorities and responsibilities in language testing research. This awareness is the result of the treatment of the social character of educational assessment in Messick's (1989) work on validity and by actual changes triggered by postmodernism. (Author/VWL)

Addresses the question of whether there are different characteristics and language performance conditions (involving assumed different levels of cognitive demand) associated with different levels of fluency, complexity, or accuracy in test candidate responses. Participants were pre-university students taking English courses. (Author/VWL)

Examines the linguistic orientation of traditional language-assessment procedures by focusing on an Australian study of immigrant health professionals in which a mismatch was found between successful performance on a test for occupation-specific English-language ability and clinical supervisors' perceptions of their supervisees' English communication skills. (Author/VWL)