Search

This article discusses the familiar notion of the shibboleth in situations of exclusion, focusing on the non-use, rather than the use, of language, for which I propose the term the anti-shibboleth. The article begins with an introduction to the concept of the shibboleth, giving examples from situations of violent conflict and suppression such as the Holocaust and the Cambodian genocide, and goes on to develop the notion of the anti-shibboleth, using further examples from such contexts. It then considers the situation of Aboriginal Australians in the period up until 1967 in the light of the concept. The article concludes with a discussion of the way in which we may understand the anti-shibboleth theoretically, drawing on insights from poststructuralism.

This paper introduces the conceptual framing of studies of trauma. It considers how, on the one hand, applied linguistics may contribute to this study, responding to the suggestion that trauma 'can be best understood through plural, multi-disciplinary perspectives' (Luckhurst 2008: 214), and, on the other hand, the extent to which linguistic studies of trauma can contribute to a better understanding of what Coupland and Coupland (1997: 117) have called 'discourses of the unsayable'. It argues that the tools of linguistic analysis may be used to understand the role of language in how individuals may experience, recount, and potentially recover from psychological trauma, in personal, literary, and institutional contexts, as exemplified by the papers in this volume.

Applied linguists have developed complex theories of the ability to communicate in a second language (L2). However, the perspectives on L2 communication ability of speakers who are not trained language professionals have been incorporated neither into theories of communication ability nor in the criteria for assessing performance on general-purpose oral proficiency tests. This potentially weakens the validity of such tests because the ultimate arbiters of L2 speakers' oral performance are not trained language professionals. This study investigates the perspectives of these linguistic laypersons on L2 communication ability. Twenty-three native and non-native English-speaking linguistic laypersons judged L2 speakers' oral performances and verbalized the reasons for their judgments. The results showed that the participants focus not only on the linguistic aspects of the speaker's output but also on features that applied linguists have less paid attention to. Even where speaker's linguistic errors were acknowledged, message conveyance and comprehensibility of the message contributed to their judgment. The study has implications for language testing and the development of tests reflecting the construct of English as a lingua franca.

All educational testing is intended to have consequences, which are assumed to be beneficial, but tests may also have unintended, negative consequences (Messick, 1989). The issue is particularly important in the case of large-scale standardized tests, such as Australia's "National Assessment Program--Literacy and Numeracy" (NAPLAN), the intended benefits of which are increased accountability and improved educational outcomes. The NAPLAN purpose is comparable to that of other state and national 'core skills' testing programs, which evaluate cross-sections of populations in order to compare results between population sub-groupings. Such comparisons underpin 'accountability' in the era of population-level testing. This study investigates the impact of NAPLAN testing on one population grouping that is prominent in the NAPLAN results' comparisons and public reporting: children in remote Indigenous communities. A series of interviews with principals and teachers documents informants' first-hand experiences of the use and effects of NAPLAN in schools. In the views of most participants, the language and content of the test instruments, the nature of the test engagement, and the test washback have negative impacts on students and staff, with little benefit in terms of the usefulness of the test data. The primary issue is the fact that meaningful participation in the tests depends critically on proficiency in Standard Australian English (SAE) as a first language. This study contributes to the broader discussion of how reform-targeted standardized testing for national populations affects sub-groups who are not treated equitably by the test instrument or reporting for accountability purposes. It highlights a conflict between consequential validity and the notion of accountability that drives reform-targeted testing.

This article addresses the suitability of the CEFR as the basis for decisions about the readiness of individuals to engage in academic writing tasks in undergraduate university courses, and as a guide to progress. The CEFR offers potentially relevant general scales and subscales, but also more specific subscales for writing in the academic context. However, recent challenges to traditional views of academic writing have potential implications for assessment frameworks such as the CEFR when they are used to identify readiness for, and progress in, academic study. In this article we explore the views of students on what it means to "do" academic writing. Questionnaires, interviews, and short reflective texts were used to investigate the changing perceptions of first-year undergraduate students at an Australian university. The analysis of student data confirms the reality of the more complex view of academic writing suggested by the recent literature. The article then considers what implications this has for the adequacy of the definitions provided in the CEFR. It suggests that the CEFR descriptors underrepresent the complexity of the challenges of academic writing, particularly its cognitive demands. A new and rather different approach will be required to inform assessments used to manage the admission of students in`to academic writing contexts and the monitoring of their progress.

This report documents two coordinated exploratory studies into the nature of oral English-for-academic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine field experts? rating orientations, and Study II investigated the quality of test-taker discourse on two different Test of English as a Foreign Language? (TOEFL®) task types (independent and integrated) at different levels of proficiency. Study I showed that, with no guidance, domain experts distinguished and described qualitatively different performances using a common set of criteria very similar to those included in draft rating scales developed for the tasks at ETS. Study II provided empirical support for the criteria applied by the judges. The findings indicate that raters take a range of performance features into account within each conceptual category and that holistic ratings are driven by all of the assessment categories rather than, as has been suggested in earlier studies, predominantly by grammar.

This article brings together the theoretical field of language testing and the practical field of language analysis for the determination of the origin of asylum seekers. It considers what it would mean to think of language analysis as a form of language test, subject to the same validity constraints, and proposes a research agenda.

Gaining insights from domain experts into how they view communication in real world settings is recognized as an important authenticity consideration in the development of criteria to assess language proficiency for specific academic or occupational purposes. These "indigenous" criteria represent an articulation of the test construct and should therefore reflect what is germane to the particular domain of language use rather than general language-focused criteria familiar from other language tests. The methodological question of how to elicit such insights is, however, complex and has been addressed by various researchers using different methodological and theoretical frameworks.

This paper explores the views of nursing and medical domain experts in considering the standards for a specific-purpose English language screening test, the Occupational English Test (OET), for professional registration for immigrant health professionals. Since individuals who score performances in the test setting are often language experts rather than domain experts, there are possible tensions between what is being measured by a language test and what is deemed important by domain experts. Another concern is a lack of qualitative research on the process of the standard setting. To date, no published qualitative work has been identified about the contributions of domain experts in the standard setting for healthcare communication. In this study, a standard-setting exercise was conducted for the speaking component of the OET, using judgements of nursing and medical clinical educators and supervisors. In all, 13 medical and 18 nursing clinical educators and supervisors rated medical and nursing candidate performances respectively. These performances were audio-recorded OET role-plays that were selected across a range of proficiency levels. Domain experts were invited to comment on the basis of their decisions and the extent of alignment between these decisions and the criteria used to assess performance on the OET. Nursing and medical domain experts showed that they attended to all of the OET criteria in making their decisions about standards. However, clinical scenario simulation also invited judgements of clinical competence from participants, even where they knew that clinical competence should be excluded from their decision-making. Another concern related to the authenticity limitations of the role-play tasks as evidence of readiness to handle communication in the workplace. Overall, findings support the value of qualitative evidence from the standard setting in providing insight into the factors informing and impeding decision-making.

This paper considers how to establish the minimum required level of professionally relevant oral communication ability in the medium of English for health practitioners with English as an additional language (EAL) to gain admission to practice in jurisdictions where English is the dominant language. A theoretical concern is the construct of clinical communicative competence and its separability (or not) from other aspects of professional competence, while a methodological question examines the technical difficulty of determining a defensible minimum standard. The paper reports on a standard-setting study to set a minimum standard of professionally relevant oral competence for three health professions--medicine, nursing, and physiotherapy--as measured by the speaking sub-test of the Occupational English Test, a profession-specific test of clinically related communicative competence. While clinical educators determined the standard, it is to be implemented by raters trained as teachers of EAL; therefore, the commensurability of the views of each group is a central issue. This also relates to where the limits of authenticity lie in the context of testing language for specific purposes: to represent the views of domain experts, a sufficient alignment of their views with scores given by the raters of test performances is vital. The paper considers the construct of clinical communicative competence and describes the standard-setting study, which used the analytical judgement method. The method proved successful in capturing sufficiently consistent judgements to define defensible standards. Findings also indicate that raters can act as proxies for occupational experts, although it remains unclear whether the views of performances held by these two groups are directly comparable. The new minimum standards represented by the cut scores were found to be somewhat harsher than those in current use, particularly in medicine.

Language has featured prominently in contemporary social theory, but the relevance of this fact to the concerns of Applied Linguistics, with its necessary orientation to practical issues of language in context, represents an ongoing challenge. This article supports the need for a greater engagement with theory in Applied Linguistics. It considers the ongoing debate about the nature of Applied Linguistics as a field and the role of theory within it. The article concludes with a call for re-engagement with theory in the humanities, in particular poststructuralism.

Those who work in second- and foreign-language testing often find Koretz's concern for validity inferences under high-stakes (VIHS) conditions both welcome and familiar. While the focus of the article is more narrowly on the potential for two instructional responses to test-based accountability, "reallocation" and "coaching," to undermine the validity of score-based inferences about achievement by inflating scores and exaggerating mastery of the domain of interest, rather than the broader set of problems around test-based accountability (Haertel, 2013), the issues raised have also been extensively discussed for several years in the field of language testing, using somewhat different terminology. "Washback" (or "backwash") is the specific term used to refer to the effects of high-stakes assessment on teaching and learning and is considered a subset of test impact more generally (Cheng, 2008). The inflationary or deflationary effect of nonsubstantive (construct irrelevant) "performance elements" on scores (associated with coaching in the article) is known as "test method effect" (Bachman, 1990). The context of the article, where outcomes-based control of educational effort is found to have a distorting effect on teaching and learning efforts, has many parallels in language testing, not only in the United States, where the negative impact on bilingual education of test-based accountability under No Child Left Behind and more recently the Common Core Standards has been widely discussed (Menken, 2008), but internationally, particularly in studies of the problematic impact of the Common European Framework of Reference (Council of Europe 2001) and PISA testing on language education (McNamara, 2011). Issues around validity inferences under high-stakes conditions arise in language testing in more contexts than the kind of test based accountability scenarios that are the focus of this article. Since the appearance of a special volume in "Language Testing" (the leading journal in the field) featuring an article on washback by Messick (1996), there has been a significant amount of research undertaken on washback and the broader impacts of language assessment, including a suite of studies sponsored by the major language testing agencies themselves. However, the high-stakes language assessments forming the focus of these studies are typically used for selection and/or accreditation purposes rather than for school and teacher accountability: language tests act as gatekeeping devices to control access to employment, education, migration, and citizenship. There is a general consensus in language-testing research that high-stakes language assessment influences "what" is taught but not necessarily "how" it is taught.

This paper uses two autobiographical accounts of language learning within the context of highly significant personal relationships to argue that the functional emphasis of communicative language teaching has narrowed our understanding of the potential personal and educational meaning of what it is to learn a language. The author's learning of Hebrew and German at different periods of his life is interpreted within his close relationship (as a person raised within the conventionally anti-Semitic cultural environment of Australian Irish Catholicism) with European Jewish friends whose lives have been shaped by the Holocaust and its painful legacy. Language learning in each case has involved psychological and social exploration of the meaning of cultural boundaries and how they are policed. The narrative supports a traditional rationale for the educational benefit of language learning, one that is currently little heard: the development of the learner as a human being.

The biblical story of the shibboleth is widely cited in language testing as emblematic of the social and political function of language tests. But the meaning of the shibboleth has also been explored within poststructuralism, specifically within Derrida's discussion of the dilemmas of identity in the work of the German Jewish poet Paul Celan. Derrida discusses language itself as shibboleth, and emphasizes the ambiguity and indeterminacy of the linguistic sign, its "undecidability". This article considers the implications of Derrida's argument for the area of language testing in order to see what his interpretation of the shibboleth might mean for understanding its practices, typically framed as they are within a modernist paradigm. Examples are drawn from various areas of language assessment, including the use of language analysis in the determination of the claims of asylum seekers. What implications for understanding such assessment practices does a poststructuralist perspective offer?

This paper examines the uptake of Rasch measurement in language testing through a consideration of research published in language testing research journals in the period 1984 to 2009. Following the publication of the first papers on this topic, exploring the potential of the simple Rasch model for the analysis of dichotomous language test data, a debate ensued as to the assumptions of the theory, and the place of the model both within Item Response Theory (IRT) more generally and as appropriate for the analysis of language test data in particular. It seemed for some time that the reservations expressed about the use of the Rasch model might prevail. Gradually, however, the relevance of the analyses made possible by multi-faceted Rasch measurement to address validity issues within performance-based communicative language assessments overcame language testing researchers' initial resistance. The paper outlines three periods in the uptake of Rasch measurement in the field, and discusses the research which characterized each period. (Contains 4 tables and 12 notes.)

This paper presents a comprehensive framework for researching classroom-based assessment (CBA) processes, and is based on a detailed empirical study of two Australian school classrooms where students aged 11 to 13 were studying Indonesian as a foreign language. The framework can be considered innovative in several respects. It goes beyond the scope of earlier models in addressing a number of gaps in previous research, including consideration of the epistemological bases for observed assessment practices and a specific learner and learning focus. Moreover, by adopting the broadest possible definition of CBA, the framework allows for the inclusion of a diverse range of data, including the more intuitive forms of teacher decision-making found in CBA (Torrance & Pryor, 1998). Finally, in contrast to previous studies the research motivating the development of the framework took place in a school-based foreign language setting. We anticipate that the framework will be of interest to both researchers and classroom practitioners. (Contains 8 tables, 6 figures and 2 notes.)

The suggested role for assessment in developing "Roadmaps for Learning" has potentially important implications for the learning of second or foreign languages in school, a major concern of applied linguistics. In this response, the authors consider how the findings of a detailed ethnographic study of classroom-based assessment in two foreign language classrooms in Australia can illuminate a number of the issues and suggestions in the article. They also evaluate the success of attempts within language testing so far to map growth along the lines suggested in the article, and the implications this has for the applicability of the suggested approach to "skill-oriented" areas of the curriculum (such as proficiency in the practical use of languages, the current target of school languages curricula), as distinct from the "knowledge-oriented" areas of the curriculum (such as mathematics or science). (Contains 1 table.)

A feature of language assessment internationally is its role in the enforcement of language policies established by governments and other educational and cultural agencies. This trend has led to the near-universal adoption of curriculum and assessment frameworks, the clearest example of which is the Common European Framework of Reference for Languages (CEFR). What does this movement represent? What is the source of the authority of frameworks such as the CEFR, and why are they so appealing to governments? The paper argues that the determination of test constructs within policy-related frameworks leads to inflexibility, and considers the case of the testing of competence in English as a lingua franca (ELF) communication, in the context of international civil aviation communication.

The paper by Wilson and Moore (this volume), based on the Messick Lecture delivered in 2006 at the annual Language Testing Research Colloquium in Melbourne, may present a familiar challenge to some language testers: of reading outside one's comfort zone. The distinctive character of language testing lies in its combination of two primary fields of expertise: (1) applied linguistics; and (2) measurement. The tradition of the Messick Lecture at the annual Language Testing Research Colloquium (LTRC) has been to invite those with expertise in measurement, often from outside language testing, to share their current work with language testing practitioners. This paper is admirably clear and patient in its setting forth of the logic and detail of the psychometric models it presents, and illustrating these with data from language testing. On a technical note, the paper gives a lucid and practically useful discussion of effect sizes in Rasch analyses, and also helps one to understand the point and the logic of model goodness of fit tests for language test data. It is also important to note that the logic of the approach outlined in the paper is not restricted to Rasch models but applies to non-Rasch models as well, including two-parameter logistic models and multidimensional models. The extensive research effort within language testing to provide empirical support for the practice of content-referencing of language tests has clear implications for the validity of content-referencing more generally, and deserves to be more widely known. It also demonstrates the advantage of ongoing dialogue between subject matter assessment specialists, such as language testers, and psychometricians, of the kind triggered by Wilson and Moore's stimulating paper. (Contains 1 note.)

Discussions of multilingualism in education, as exemplified by the articles in this issue, can be critically reevaluated using perspectives available from poststructuralism. These perspectives focus on the potential ambiguity of language and language practices. This, in turn, encourages us to question simple notions of the relationship of learners to the languages they speak, especially the "mother tongue," to see the individual's relation to language as a relation to power, and to recognize the polyvalent role of language tests in the context of multilingual education as, on the one hand, enforcing the relations of power in language and, on the other, disrupting them. The article focuses on the extent to which these themes are acknowledged in the articles in this issue, exploring the impact of the contrasting contexts represented, and presents examples of other discussions, especially in colonial and postcolonial contexts, that reflect alternative views of multilingualism in education. These include Derrida's reflections on his own language socialization as a colonial subject in Algeria during the time of the Vichy regime and studies of the role of language in contemporary education systems in Africa and Southeast Asia.

This article proposes a distinction between "fairness" and "justice" in relation to language tests. Basing its discussion on the validity theory of Samuel Messick, it discusses the way in which these terms have been used in the existing literature, and their relationship to the concept of validity. Fairness, broadly speaking, refers to the technical quality of the test; justice encompasses the values implicit in test constructs, and the social uses to which language tests may be put. It illustrates the potential usefulness of the distinction by discussing the growing international practice of the use of language tests within citizenship procedures, and considers in detail the introduction of a formal citizenship test in Australia by a conservative government in the context of growing public anxiety about security, and about cultural diversity within the Australian population. (Contains 5 footnotes, 2 tables, and 1 figure.)

In its late colonial history and early years as an independent nation, Australia practised a policy of ruthless exclusion of immigrants on the basis of race by means of a language test: the notorious Dictation Test. In the 50 years following World War II, Australia adopted policies encouraging immigration with bipartisan political support. However, following the election of John Howard as Prime Minister in 1996, immigration was again politicised in the context of the emergence of an anti-immigrant right wing populist movement, which threatened Howard's political base. This, plus the security anxieties following 9/11 and heightened intergroup tensions within Australia itself, led to a renewal of discourses of national belonging and exclusion. One result was the introduction of a new linguistically demanding citizenship test in 2007, again with bipartisan support. The role of language testers in the light of such policy developments is considered. (Contains 7 footnotes.)

The study reported in this paper is an investigation of the nature of speaking proficiency in English as a second language in the context of a larger project to develop a rating scale for a new international test of English for Academic Purposes, TOEFL iBT (Brown et al. 2005). We report on a large-scale study of the relationship between detailed features of the spoken language produced by test-takers and holistic scores awarded by raters to these performances. Spoken test performances representing five different tasks and five different proficiency levels (200 performances in all) were analyzed using a range of measures of grammatical accuracy and complexity, vocabulary, pronunciation, and fluency. The results showed that features from each category helped distinguish overall levels of performance, with particular features of vocabulary and fluency having the strongest impact. Overall, the study contributes important insights into the nature of spoken proficiency as it develops and can be measured in rating scales for speaking, and has implications for methodological issues of the appropriateness of the use in language testing research contexts of measures developed in research on second language acquisition.

The thought of Samuel Messick has influenced language testing in 2 main ways: in proposing a new understanding of how inferences made based on tests must be challenged, and in drawing attention to the consequences of test use. The former has had a powerful impact on language-testing research, most notably in Bachman's work on validity and the design of language tests. Messick's writing on test consequences has informed debate on ethics, impact, accountability, and washback in language testing in the work of several researchers. But the character of Messick's work challenges us in many additional ways. Messick located validity theory in the area of values. This article explores the implications of this position, highlighting the social construction of language test constructs. Language test constructs are increasingly the target of policy, a development that threatens to render traditionally conceived validation work of only marginal relevance. The less obvious, covert social construction of language test constructs is explored in the light of Butler's theory of performativity. The article concludes with a consideration of recent adaptations of Messick's work in the influential validation models of Mislevy and Kane, and considers their failure to address questions of values and the social context of assessment properly. Tackling these questions is the ongoing challenge of Messick's legacy. (Contains 4 footnotes and 2 tables.)

Used Rasch analytic procedures to study item bias or differential item functioning in both dichotomous and scalar items on a test of English for academic purposes. Results for 139 college students on a pilot English language test model the approach and illustrate the measurement challenges posed by a diagnostic instrument to measure English language achievement. (SLD)