Search

The elicited imitation task (EIT), in which language learners listen to a series of spoken sentences and repeat each one verbatim, is a commonly used measure of language proficiency in second language acquisition research. The "TOEFL® Essentials"™ test includes an EIT as a holistic measure of speaking proficiency, referred to as the "Listen and Repeat" task type. In this report, we describe the design considerations that informed the development of the EIT for TOEFL Essentials. We also report the results of a series of investigations conducted during the prototyping and pilot phases of test development, which were undertaken with the goal of confirming task design specifications, evaluating scoring performance, and obtaining initial validity evidence to support score interpretation and use of the EIT in the TOEFL Essentials test. We found that task design variables generally performed as expected. The length of input sentence was strongly associated with performance (Pearson r = 0.88), consistent with the construct measured by the EIT, while other task variables not directly related to the EIT construct did not impact performance (e.g., graphics, speaker accent, and response time). Scorers drawn from TOEFL iBT test raters were able to score responses consistently with over 98% exact or adjacent interrater agreement on a 6-point scale, and scores on the pilot version of the EIT were highly reliable (Cronbach's [alpha] = 0.93 on the 15-item pilot version). Correlations between EIT scores and other measures were generally as expected: Correlations with other speaking tasks were high (0.78-0.84) and slightly to somewhat lower for other language measures (0.73 for writing, 0.68 for listening, and 0.57 for reading). Correlation with an independent measure of holistic language proficiency (C-test) was moderately high (0.69), as expected. We discuss the study findings in terms of the TOEFL Essentials test validity argument and point out limitations to the current results along with future research needs. Overall, we believe that the findings provide initial support to warrant the use of the EIT as operationalized in the TOEFL Essentials test.

This exploratory study examines the role of foreign and second language contexts in the acquisition of conventional expressions. A group of 21 ESL learners was compared to 25 EFL learners randomly selected from a larger pool. Both groups completed an aural multiple-choice discourse completion task (MC-DCT), which was developed from a previously validated oral DCT. The aural MC-DCT consisted of 21 items with learner-generated options delivered aurally. A total of 91 native speakers also completed the task as a control group. The results showed an effect of learning environment on learners' selection of conventional expressions. The ESL group selected the conventional expressions in more items than the EFL group on the aural MC-DCT; the differences in the selections by the two groups were item-specific. The observed effect of learning context is discussed as related to individual items and type and modality of the task. The paper also discusses the special make-up of the ESL group due to the pandemic and expansion of the group for future research.

Individual differences in understanding other people's emotions have typically been studied with recognition tests using prototypical emotional expressions. These tests have been criticized for the use of posed, prototypical displays, raising the question of whether such tests tell us anything about the ability to understand spontaneous, non-prototypical emotional expressions. Here, we employ the Emotional Accuracy Test (EAT), which uses natural emotional expressions and defines the recognition as the match between the emotion ratings of a target and a perceiver. In two preregistered studies (N[subscript total] = 231), we compared the performance on the EAT with two well-established tests of emotion recognition ability: the Geneva Emotion Recognition Test (GERT) and the Reading the Mind in the Eyes Test (RMET). We found significant overlap (r > 0.20) between individuals' performance in recognizing spontaneous emotions in naturalistic settings (EAT) and posed (or enacted) non-verbal measures of emotion recognition (GERT, RMET), even when controlling for individual differences in verbal IQ. On average, however, participants reported enjoying the EAT more than the other tasks. Thus, the current research provides a proof-of-concept validation of the EAT as a useful measure for testing the understanding of others' emotions, a crucial feature of emotional intelligence. Further, our findings indicate that emotion recognition tests using prototypical expressions are valid proxies for measuring the understanding of others' emotions in more realistic everyday contexts.

We developed and validated the Family Reminiscence Scale (FARS) in which adults rate their frequency of reminiscing with their parents about childhood experiences. In three studies, we characterized how FARS was related to adults' recollections of their earliest memories in different cultural contexts. First, we examined the factorial structure of FARS and obtained two factors of reminiscing: first-time events and general-recurrent events. In the second study, confirmatory factor analyses were conducted, in which we established measurement invariance across gender and age groups. In Study 3, we tested the factorial structure of FARS in an American sample to ensure cross-cultural invariance. We also showed that the two factors were differentially related to the phenomenology of earliest memories in samples from Turkey and United States (Study 2 & Study 3). Overall, FARS was found to be reliable and valid to measure for adult samples to assess the quality of the linguistic input during childhood. Predictive value of FARS has been shown across different gender, age, and culture groups, underlining the organizational role of the early communicative context in the phenomenology and linguistic style of adults' early memories.

It is now well accepted that working in research teams that span universities, jurisdictions and countries can be rewarding and economically prudent. To this end, investigators collaborate in the pursuit of knowledge to address human and societal problems and translate results into local and global contexts. This implies that investigators need to develop study instruments that are fit for purpose and strategically manage issues arising from geographical, linguistic and cultural diversity. A proven method is cognitive interviewing to pre-test the study materials to ensure clarity and relevance in the study population. This paper describes the steps taken to increase the methodological reliability of study instruments through the use of cognitive interviewing and argues this technique should be a standard step in instrument development.

This report introduces the "WorkFORCE"™ Assessment for Job Fit, a personality assessment utilizing the "FACETS"™ core capability, which is based on innovations in forced-choice assessment and computer adaptive testing. The instrument is derived from the fivefactor model (FFM) of personality and encompasses a broad spectrum of personality assessment. This document provides an overview of the assessment, beginning with detailing evidence-based practices for personality measurement and modeling its relationship to workplace outcomes.We address the validity and fairness of this assessment, the creation of composite scores, and the generalizability of the assessment across languages and job types. We conclude with recommendations on the use of this capability for workforce applications and guidelines for future research.

One fundamental way to determine the validity of standardized English-language test scores is to investigate the extent to which they reflect anticipated learning effects in different English-language programs. In this study, we investigated the extent to which the "TOEFL iBT"® practice test reflects the learning effects of students at intensive English programs in the United States and China, as well as extracurricular English-learning activities that may be associated with the expected learning effects. A total of 607 students at the high school level or beyond participated in the United States and China, including 111 students who took 2 forms of the practice test under a pretest and posttest design. The results showed moderate to substantial levels of improvement on each of the TOEFL iBT sections, with different score gain patterns for students in the United States and China. We concluded that students who study at English programs similar to those included in this study can improve their English-language proficiency levels at least moderately over 6 months or longer, as indicated by changes in their scores on the TOEFL iBT practice test. This improvement is consistent with an interpretation of TOEFL iBT scores as indicators of English-language proficiency.

The topic of "teacher professionalism" is one of the most crucial ones in quality education research. It has a potential to generate results that could inform and hence enhance the practice in classrooms. Thus, research in this field needs reliable instruments to measure the professional knowledge of our teachers to be able to generate reliable results for our research problems. Not many instruments have been developed with regard to this topic. At the same time, an adequate validation of the instrument developed is often missing (Schilling & Hill, 2007). Hence, in a bigger project "ProwiN" (German acronym for professional knowledge of science teachers), test instruments for measuring science teachers' pedagogical, pedagogical content and content knowledge (PK, PCK, and CK) were developed for the subjects biology, chemistry and physics. The present study tested the validity of some of these items which were used to measure the pedagogical content knowledge (PCK) of biology teachers. These items focused on measuring teachers' professional knowledge by analyzing 1) teachers' "knowledge about student understanding" (or lack of understanding) of several topics in biology and 2) "knowledge about instructional strategies" like the use of models or experiments. The content validity of these instruments was examined by think-aloud interviews with American and German Biology teachers (N = 11). This study shows a high content validity for these items. Furthermore, this paper demonstrates the scope for adapting the conceptual framework of these items to measure biology teachers' PCK in other countries.

In physics education research (PER), concept inventories (CIs) have become standard instruments for assessing students' learning throughout instruction. To promote widespread use of concept inventories, previous studies have developed an approach to split a full length CI into short versions of CIs. This research extends the existing method to fully utilize the item response theory framework in equating and linking between the short CIs and the full length CIs. Three quantitative studies have been conducted: First, the extended algorithm is applied to divide the Brief Electricity and Magnetism Assessment (BEMA) into two half-length BEMAs (HBEMAs). Through a series of test-equating and validation analysis, the HBEMAs are confirmed to measure the same latent constructs of student understanding of electricity and magnetism to that of the original BEMA at a similar level of reliability. The second study establishes equivalent score conversions among the three versions of BEMA using the Stocking-Lord method, which has the best performance on equating error reduction among several methods explored. It is also confirmed that the equivalent statistical characteristics of the three versions of BEMA are equity and population invariant. In the third study, the extended algorithm is applied to link and compare the BEMA and the Conceptual Survey of Electricity and Magnetism (CSEM). After linking the BEMA and CSEM assessment scales, it becomes possible to directly convert and compare students' performances on the two CIs. It is found that the scales of BEMA and CSEM are almost identical after scale transformation. Based on these studies, it can be suggested that all short and long versions of BEMA and CSEM can be used interchangeably after scale transformation.

The phrase "no child left behind" has become a familiar expression in American education circles and in popular culture. The sentiment implied by these four words is noble. However, the effects of the top-down implementation of the high-stakes testing provisions of the law have been anything but salutary for public school children, teachers, and administrators. This claim is supported by data describing many of the ways in which well-intentioned but desperate educators, from the statehouse to the schoolhouse, have been driven to game the system in ironic defense of the children, teachers, and administrators least equipped to defend themselves. It is argued herein that, instead of reauthorizing the stronger accountability tenet of NCLB, it might do very well to let it fade away. (Contains 1 figure.)

Introduction: School readiness predicts both school and life success, so measuring it effectively is extremely important. Current school readiness tests focus on pre-academic skills; however, mastery motivation (MM: persistent, focus on trying to do a task) and executive functions (EF: planful self-control) are also crucial. Method: The purpose of the paper is to give an overview of a new, computer-based assessment of MM and EF. Results: We have developed a game-like, computer-based assessment for 3 to 8 year-old children, of MM, EF, and recognition of numbers and letters. The new measures are appropriate for both Hungarian and American cultures. They were engaging for children of this age, and preliminary evidence suggests that they are reliable and valid. Conclusion: The new tasks can be part of assessments of school readiness, and would be useful for school practice as well as researchers. The tasks ascertain the extent to which observed deficits in pre-academic domains are due to MM or EF difficulties. The results will contribute to the development of individualized intervention.

This study examined the psychometric properties of a social-networking site (SNS) activities scale (SNSAS) using Rasch Analysis. Items were also examined with Rasch Principal Components Analysis (PCA) and Differential Item Functioning (DIF) across groups of university students (i.e., males and females from the United States [US] and Europe; N = 840). Results from this study found psychometric support for the SNSAS to measure endorsement of activities in which university students engage. European males and females were more likely to endorse actively using SNSs compared to their more passive US counterparts. The SNSAS is a reliable and valid tool in a growing research area where few exist. University administrators worldwide can benefit from understanding the patterns of SNS behavior that may impact students' socialization, relationship formation, and academic outcomes.

In the United States, tests are held to high standards of quality. In developing countries such as Malawi, psychometricians must deal with these same high standards as well as several additional pressures such as widespread cheating, test administration difficulties due to challenging landscapes and poor resources, difficulties in reliably scoring performance assessments, and extreme scrutiny from political parties and the popular press. The purposes of this paper are to (a) familiarize the measurement community in the US about Malawi's assessment programs, (b) discuss some of the unique challenges inherent in such a program, (c) compare testing conditions and test administration formats between Malawi and the US, and (d) provide suggestions for improving large-scale testing in countries such as the US and Malawi. By learning how a small country instituted and supports its current testing programs, a broader perspective on resolving current measurement problems throughout the world will emerge. (Contains 4 tables and 3 notes.)

Purpose: How to build and enhance the strategic learning capability (SLC) of an organization becomes crucial to both research and practice. This study was designed with the purpose to conceptualize SLC by translating and interpreting the related literature to develop empirical dimensions that could be tested and used in a survey instrument. Design/methodology/approach: An instrument was developed to identify empirical dimensions of SLC. The reliability and validity of the instrument were tested. Findings: The resulting survey instrument included 59 items, and 49 remained after empirical test. Based on responses on a five-point performance scale, SLC items were identified and prioritized, and seven dimensions were discovered: external focus, strategic dialogue, strategic engagement, customer-centric strategy, disciplined imagination, experiential learning and reflective responsiveness. Originality/value: The findings of this study extend the knowledge base of multi-disciplines, including strategy management, organizational learning and strategic human resource development (HRD). This study highlights the conceptualization of SLC and importance of the SLC framework in the field of HRD.

Researchers, administrators, and policy makers need valid and reliable information about teaching practices. The Postsecondary Instructional Practices Survey (PIPS) is designed to measure the instructional practices of postsecondary instructors from any discipline. The PIPS has 24 instructional practice statements and nine demographic questions. Users calculate PIPS scores by an intuitive proportion-based scoring convention. Factor analyses from 72 departments at four institutions (N = 891) support a 2- or 5-factor solution for the PIPS; both models include all 24 instructional practice items and have good model fit statistics. Factors in the 2-factor model include (a) instructor-centered practices, nine items; and (b) student-centered practices, 13 items. Factors in the 5-factor model include (a) student--student interactions, six items; (b) content delivery, four items; (c) formative assessment, five items; (d) student-content engagement, five items; and (e) summative assessment, four items. In this article, we describe our development and validation processes, provide scoring conventions and outputs for results, and describe wider applications of the instrument.

Students' attitudes and approaches to physics problem solving can impact how well they learn physics and how successful they are in solving physics problems. Prior research in the U.S. using a validated Attitude and Approaches to Problem Solving (AAPS) survey suggests that there are major differences between students in introductory physics and astronomy courses and physics experts in terms of their attitudes and approaches to physics problem solving. Here we discuss the validation, administration, and analysis of data for the Turkish version of the AAPS survey for high school and university students in Turkey. After the validation and administration of the Turkish version of the survey, the analysis of the data was conducted by grouping the data by grade level, school type, and gender. While there are no statistically significant differences between the averages of various groups on the survey, overall, the university students in Turkey were more expertlike than vocational high school students. On an item by item basis, there are statistically differences between the averages of the groups on many items. For example, on average, the university students demonstrated less expertlike attitudes about the role of equations and formulas in problem solving, in solving difficult problems, and in knowing when the solution is not correct, whereas they displayed more expertlike attitudes and approaches on items related to metacognition in physics problem solving. A principal component analysis on the data yields item clusters into which the student responses on various survey items can be grouped. A comparison of the responses of the Turkish and American university students enrolled in algebra-based introductory physics courses shows that on more than half of the items, the responses of these two groups were statistically significantly different, with the U.S. students on average responding to the items in a more expertlike manner.

The OECD and its U.S. administrator, McGraw-Hill Education CTB, have recently concluded the first cycle of the OECD-Test for Schools in the U.S. This test is being marketed to local schools and is designed to compare 15-year-olds from individual participating schools against peers nationally and internationally using the OECD's PISA test as its foundation. This article discusses what PISA can and can't tell the United States about its education system and describes important aspects of the PISA design that educators should consider before participating in the school-based version of this international assessment.

This study assesses the Japanese translation of the Force and Motion Conceptual Evaluation (FMCE). Researchers are often interested in comparing the conceptual ideas of students with different cultural backgrounds. The FMCE has been useful in identifying the concepts of English-speaking students from different backgrounds. To identify effectively the conceptual ideas of Japanese students and to compare them to those of their English-speaking counterparts, more work is required. Because of differences between the Japanese and English languages, and between the Japanese and American educational systems, it is important to assess the Japanese translation of the FMCE, a conceptual evaluation originally developed in English for American students. To assess its appropriateness, we examined the performance of a large sample of students on the translated version of the FMCE and then compared the results to those of English-speaking students. The data comprise the pretest results of 1095 students, most of whom were first-year students at a midlevel engineering school between 2003 and 2012. Basic statistics and the classical test theory indices of the translated FMCE indicate that its reliability and discrimination are appropriate to assess Japanese students' concepts about force and motion. In general, the preconcepts of Japanese students assessed with the Japanese translation of the FMCE are quite similar to those of American students assessed with the FMCE, thereby supporting the validity of the translated version. However, our findings do show (1) that only a small percentage of Japanese students grasped Newtonian concepts and (2) that the percentage of Japanese students who used two different concept models together to answer some questions seems to be higher than that of American students.

The English Language Proficiency Assessment (ELPA) is used in the state of Michigan in the United States to fulfill government-mandated No Child Left Behind (NCLB) requirements. The test is used to promote and monitor achievement in English language learning in schools that receive federal NCLB funding. The goal of the project discussed here was to evaluate the perceived effectiveness of the ELPA and to see if those perceptions could meaningfully contribute to a broad concept of the test's validity. This was done by asking teachers and test administrators their views on the ELPA directly after its administration. Two hundred and sixty-seven administrators took a survey with closed and open-ended questions that aimed to tap into the consequential dimensions of test validity. An exploratory factor analysis identified five factors relating to the participants' perceptions of the ELPA. Analysis of variance results revealed that educators at schools with lower concentrations of English language learners reported significantly more problems in administering the ELPA. Three themes (the test's appropriateness, impacts, and administration) emerged from an analysis of qualitative data. This article discusses these results not only as a means of better understanding the ELPA, but also to contribute to larger-scale discussions about consequential validity and standardized tests of English language proficiency. It recommends that broadly defined validity data be used to improve large-scale assessment programs such as those mandated by NCLB. (Contains 5 figures, 5 tables, and 5 footnotes.)

Background: Readability tests are indicators that measure how easy a document can be read and understood. Simple, but very often ignored, readability statistics cannot only provide information about the level of difficulty of the readability of particular documents but also can increase an evaluator's credibility. Purpose: The purpose of this article is two-fold: (1) to provide readers with logical reasons for using readability tests and (2) how to choose the right test for a project. Setting: United States. Research Design: A comparative framework is used to present the need for readability testing. (Contains 4 tables and 2 figures.)

In this article we present results of a comprehensive review of principal leadership assessment practices in the United States. Our analyses of both the general content and the usage of 65 instruments, 56 at the district level and 9 at the state level, provided an in-depth look at what and how districts and states evaluate principals. Using the learning-centered leadership framework, we focused on identifying the congruence (or lack thereof) between documented assessment practices and the research-based criteria for effective leadership that are associated with improved school performance. Using an iterative and deductive process for instrument content analysis, we found that states and districts focused on a variety of performance areas (e.g., management, external environment, or personal traits) when evaluating their principals, with different formats at various levels of specificity. We also found very limited coverage of leadership behaviors that ensure rigorous curriculum and quality instruction, which are linked with schoolwide improvement for the ultimate purpose of enhanced student learning. In seeking information on how principals are evaluated, we found that in most cases, the practices of leadership assessment lacked justification and documentation in terms of the utility, psychometric properties, and accuracy of the instruments. (Contains 4 figures, 2 tables and 1 note.)

The purposes of this study were to identify the essential dimensions of intercultural communicative competence (ICC) and to establish a framework for assessing the ICC level of high school students that included a self-report inventory and scoring rubrics for online interaction in intercultural contexts. A total of 472 high school students from Taiwan and the United States completed the "Intercultural Communicative Competence Inventory" (ICCI), adapted from Fantini's YOGA inventory (2000, 2001), and 72 students were selected to join the "Learning of Intercultural Language over the Net" (LILON) project where they participated in structured intercultural learning activities facilitated through telecommunications technology. Statistical analyses confirmed that ICC is a composite of attitudes, skills, knowledge, awareness, and meta-awareness, which supported our proposed framework for online intercultural interactions. (Contains 5 figures and 10 tables.)

Measuring student intelligence has been problematic in the United States since standardized testing first began in the early 1900s. The omnipresence of standardized testing in student populations is illustrated by the most popular contemporary tests which are used by some scholars to advance the notion that intelligence differences between whites and racial minorities may be based in actual genetic or hereditary differences rather than cultural or environmental factors or in the testing construct itself. For academicians interested in broaching this issue with students in the classroom, it is important to demonstrate those elusive factors that play a role in divergent scores across racial groups and income levels. One way to demonstrate these effects is a tangible classroom activity that provides greater insight into such bias in testing. This paper offers a modernized version of a widely used mock questionnaire--Dove's "Black IQ Test"--intended to draw out factors related to test bias for classroom discussion which they call "You Think You Know Ghetto?"

Large-scale assessment programs are becoming increasingly common throughout Canada and the United States. Given the emphasis on inclusive education in North America, special education students are largely expected to participate in these programs. However, several challenges exist for educators, policymakers, and psychometricians with respect to including students with disabilities in large-scale assessments. This article is a critical interpretive review of the academic literature in this area intended to identify and examine issues pertinent to inclusive practice. In particular, attention is given to consequences (both positive and negative) of including students with disabilities in large-scale assessments, validity of assessment results, provisions for accommodations, and research limitations. Areas for continued research are also considered.

The idea that America was being harmed because its schools were not keeping up with those in other advanced nations emerged after Sputnik in 1957, took a firm hold on education policy when "A Nation at Risk" appeared in 1983, and continues today. Policy makers justify this concern by pointing to evidence showing that, for individuals within the U.S., higher test scores predict a number of important life advantages, such as going on to college and making more money as an adult. Since Sputnik, the evidence driving worries about the performance of U.S. schools have come primarily from a series of international achievement testing programs that started in 1964 with the First International Mathematics Study (FIMS). This was followed by the Second International Mathematics Study (SIMS), the Third International Mathematics and Science Study (TIMSS), and, most recently, the Programme for International Student Assessment (PISA). In this article, the author shows that for the U.S. and for the top dozen or so most-advanced nations in the world, standings in the league tables of international tests are worthless. There is no association between test scores and national success, and, contrary to one of the major beliefs driving U.S. education policy for nearly half a century, international test scores are nothing to be concerned about. America's schools, he asserts, are doing just fine on the world scene. (Contains 4 endnotes.)

The research for this study was undertaken to analyze the New York State 11th grade United States History Regents exams through conducting a content analysis of the types of multiple-choice questions asked in Part I of the tests with a particular emphasis on the variety of questions asked regarding women and race. Because these tests stand at the pinnacle of social studies education in the state of New York and are required of all students, it stands to reason that the questions indicate the type of social knowledge officially sanctioned by its citizens. This study demonstrates that very few questions on the New York State United States History Regents Exam deal with race or minority issues, and that of those that do, similar themes reappear from year to year, with only slight variations. The same is true about women's history questions. So, despite claims to the contrary, the Regents exams show that little knowledge of race and women's issues is required and therefore likely not being taught. This has important possible implications for society at large. (Contains 2 figures and 37 notes.)

This paper extends and further validates the quality-of-life/community-healthcare model and measures developed by Rahtz and Sirgy (2000). The quality of life (QOL) model is based on the theoretical notion that community residents' satisfaction with healthcare services available within their community affects community quality of life and life satisfaction. This study extends the model and, in keeping with past research, posits that the bottom-up spillover from community QOL to life satisfaction is greater for those individuals with low personal health satisfaction, lower income, and old age. Data were collected from 1094 community residents in the East-Coast area of the United States. The results provide good support of the new model. Managerial implications and applications for the new model are discussed.