Search

Online metacognitive skills are the real-time awareness of cognition, which can effectively promote science learning and improve performance in solving scientific problems. Therefore, it is important to enhance and diagnose students' online metacognitive skills in science education. This study aimed to evaluate ninth-grade students' online metacognitive skills while processing chemistry problems. To achieve this goal, this study constructed a framework for guiding the development of an instrument comprising 12 two-tier items. A total of 258 ninth graders took part in the field testing in Jiangsu, China. A partial credit Rasch model analysis was employed to inform instrument development and evaluation. The results revealed that this instrument was valid and reliable for assessing students' online metacognitive skills. Nearly 60% of the ninth-grade students in this sample were able to monitor their own thought processes or evaluate their own cognitive performance in processing chemistry problems. About one-third of the students could regulate their thought processes. However, less than 4% of the students could make attributions about their cognitive performance.

We report the validity of a test instrument that assesses the arithmetic ability of primary students by (a) describing the theoretical model of arithmetic ability assessment using Wilson's (2004) four building blocks of constructing measures and (b) providing empirical evidence for the validation study. The instrument consists of 21 multiple-choice questions that hierarchically evaluate arithmetic intended learning outcomes (ILOs) on arithmetic ability, hierarchically, based on Bloom's cognitive taxonomy for 138 primary three grade students. The theoretical model describes students' arithmetic ability on three distinct levels: solid, developing, and basic. At each level, the model describes the characteristics of the tasks that the students can answer correctly. The analysis shows that the difficulty of the items followed the expected order in the theoretical construct map, where the difficulty of each designed item aligned with the cognitive level of the student, the item difficulty distribution aligned with the structure of the person construct map, and word problems required higher cognitive abilities than the calculation problems did. The findings, however, pointed out that more difficult items can be added to better differentiate students with different ability levels, and an item should be revised to enhance the reliability and validity of the research. We conclude that the conceptualizations of such formative assessments provide meaningful information for teachers to support learning and tailoring instruction.

There is growing literature on computerized dynamic assessment (C-DA) wherein individual items are accompanied by mediating prompts, but its effectiveness at fine-grained levels across time has not been explored sufficiently. This study constructed a computerized listening dynamic assessment (CLDA) system, where mediation was informed by an attribute-based mediation model (AMM) that established the relationship between the listening items and their underlying cognitive attributes. One hundred and twelve low-level university learners participated in the study, with the experimental group using the AMM-informed CLDA system (hereafter the CLDA group) and the control group (CG) using a non-dynamic assessment. Results indicated that the CLDA group significantly outperformed the CG in the post- and transfer- tests at both the test and attribute levels, and mediation was more effective for items of low and medium difficulty levels than those of high difficulty levels. Questionnaire and interview data indicated that most students perceived the CLDA system positively. The study demonstrates the advantages of AMM-informed C-DA in fine-grained diagnosis and tailored mediation. At the same time, it helps advance the validation pursuit of future mediation development.

Pearson Test of English Academic (PTE Academic), a high-stakes English language proficiency test, underwent substantial revisions in 2021. The test duration was reduced from 3 h to 2 h by reducing specific task numbers and sections. This study investigates the impact of these changes on teachers' perceptions and teaching practices, areas previously underexplored in language assessment literature. It focuses on the implications of the test's modifications, examining the face validity and washback effects through the lens of teachers in mainland China. Semi-structured interviews with four experienced PTE Academic test trainers, who were familiar with both the original and revised test formats, reveals that the revised PTE Academic is perceived to maintain strong face validity, particularly noted in its academic authenticity, balanced skill structure, and perceived result accuracy. Additionally, most teachers perceived an increase in test difficulty. A mixed washback effect was observed: while improvements in students' language competence--a positive outcome--were noted, teachers also reported a continued reliance on teaching test-oriented strategies for higher-scoring tasks, indicating negative washback. This study highlights the significant implications of reducing PTE Academic's duration and offers targeted recommendations for its future improvement. These suggestions aim to enhance students' academic language skills, thereby better aligning PTE Academic with the practical language abilities required in university settings.

Based on the concept of sustainability and the aim of addressing China?s great need for talent with green skills, a green skills scale was designed with the help of Chinese university students. Students from four institutions of higher education who had agreed to the Principles for Responsible Management Education (PRME) participated as research subjects. With reference to the reviewed literature, a green skills scale was developed for Chinese university students. The scale was reviewed by experts and scholars and subjected to a reliability and validity analysis, and 900 students were selected to be included in the sample. The green skills of university students are divided into four dimensions: cognition of green knowledge, mastery of green skills, application of green skills, and green skills development, resulting in a total of 16 questions. The overall reliability of the scale is 0.845, the reliability of the dimensions lies between 0.813 and 0.880, the correlation coefficients between the dimensions lie between 0.185 and 0.497, and the root-mean-square values of the AVE of the corresponding coefficients are all larger than the correlation coefficients of the corresponding row and column variables, demonstrating that the green skills scale for university students has good reliability and validity.

The calibration of test materials generally involves the interaction between empirical analysis and expert judgement. This paper explores the extent to which scale familiarity might affect expert judgement as a component of test validation in the calibration process. It forms part of a larger study that investigates the alignment of the LanguageCert suite of tests, Common European Framework of Reference (CEFR), the China Standards of English (CSE) and China's College English Test (CET). In the larger study, Year 1 students at a prestigious university in China were administered two tests--one with items based on China's College English Test (CET), and the other a CEFR-aligned test developed by LanguageCert--the LanguageCert Test of English (LTE). Comparable sections of the CET and the LTE involved sets of discrete items targeting lexico-grammatical competence. In order to ascertain whether expert judges were equally comfortable placing test items on either scale (CET or CEFR), a group of professors from the university in China who set the CET-based test, were asked to expert judge the CET items against the nine CSE levels with which they were very familiar. They were then asked to judge the LTE items against the six CEFR levels, with which they were less familiar. Both sets of expert ratings and the test taker responses on both tests were then calibrated within a single frame of reference and located on the LanguageCert scale. In the analysis of the expert ratings, the CSE-familiar raters exhibited higher levels of agreement with the empirically-derived score levels for the CET items than they did with the equivalent LTE items. This supports the proposition that expert judgement may be used in the calibration process where the experts in question have a strong knowledge of both the test material and the standards against which the test material is to be judged.

Student understanding of chemical equilibrium in aqueous solutions (CEAS) plays a vital role in their upper secondary school chemistry learning and everyday life. Diagnosis of students' alternative conceptions (ACs) of the CEAS will provide teachers with valuable information to make instructional decisions on student learning. This study aims to develop and validate an instrument to diagnose students' ACs about the CEAS, including ionization equilibrium, water self-ionization equilibrium, the equilibrium of salt hydrolysis, and precipitation and dissolution equilibrium. Using Treagust (1998)'s development framework, we have developed 25 two-tier multiple-choice items for the CEAS diagnostic test. After completing the corresponding courses, 750 Grade 11 students from five public schools responded to the CEAS diagnostic test. Rasch modeling approach was employed to provide psychometric properties of the CEAS diagnostic test consisting of one-dimensionality, reliability, and validity. This study identified 15 ACs toward the CEAS. This study found that most students performed better on concept tiers rather than reasoning tiers. In addition, students have difficulties in connecting acidity, solubility, ionization, and chemical reaction and in using mathematical thinking to do transformation between concentration, equilibrium constant, and pH value.

Problem solving is considered one of the most critical competencies for life. However, many students may perform well in school yet cannot transfer the skills they learned in school to solve real-world problems, especially in science education. This study discusses the characteristics of the problem from the physics education perspective, based on Problem Representation Theory, and proposes a new type of problem: The Primitive Physics Problem. Moreover, the significance and contribution in promoting students' physics problem-solving ability, both theoretical and empirical, are described. Then the study develops a set of instruments that was tested on 811 students in 12th grade from 10 upper-secondary schools in China. The results showed that the new instrument had good reliability and validity. The promising application in the instruction of the Primitive Physics Problem and in measuring upper-secondary school students' problem-solving ability of the instrument is discussed.

Teachers' practical knowledge is closely related to teaching practice. The purpose of this research was to develop an instrument for assessing upper-secondary school science teachers' perceived practical knowledge about practical work. The development of this instrument was based on five components in the conceptual framework of PCK about practical work: orientations of practical work, knowledge of curriculum materials in practical work, knowledge of students in practical work, knowledge of educational strategies for practical work, and knowledge of assessment in practical work. After the questionnaire was developed, 636 Chinese upper-secondary school science teachers participated in this research voluntarily. The results from exploratory factor analysis, correlation analysis and confirmatory factor analysis provided sufficient evidence for the construct validity of instrument. The high Cronbach coefficient indicated that the instrument had good internal consistency reliability. Finally, an instrument with six factors and 25 items was documented. The instrument would benefit science teacher educators and researchers.

The elicited imitation task (EIT), in which language learners listen to a series of spoken sentences and repeat each one verbatim, is a commonly used measure of language proficiency in second language acquisition research. The "TOEFL® Essentials"™ test includes an EIT as a holistic measure of speaking proficiency, referred to as the "Listen and Repeat" task type. In this report, we describe the design considerations that informed the development of the EIT for TOEFL Essentials. We also report the results of a series of investigations conducted during the prototyping and pilot phases of test development, which were undertaken with the goal of confirming task design specifications, evaluating scoring performance, and obtaining initial validity evidence to support score interpretation and use of the EIT in the TOEFL Essentials test. We found that task design variables generally performed as expected. The length of input sentence was strongly associated with performance (Pearson r = 0.88), consistent with the construct measured by the EIT, while other task variables not directly related to the EIT construct did not impact performance (e.g., graphics, speaker accent, and response time). Scorers drawn from TOEFL iBT test raters were able to score responses consistently with over 98% exact or adjacent interrater agreement on a 6-point scale, and scores on the pilot version of the EIT were highly reliable (Cronbach's [alpha] = 0.93 on the 15-item pilot version). Correlations between EIT scores and other measures were generally as expected: Correlations with other speaking tasks were high (0.78-0.84) and slightly to somewhat lower for other language measures (0.73 for writing, 0.68 for listening, and 0.57 for reading). Correlation with an independent measure of holistic language proficiency (C-test) was moderately high (0.69), as expected. We discuss the study findings in terms of the TOEFL Essentials test validity argument and point out limitations to the current results along with future research needs. Overall, we believe that the findings provide initial support to warrant the use of the EIT as operationalized in the TOEFL Essentials test.

This exploratory study examines the role of foreign and second language contexts in the acquisition of conventional expressions. A group of 21 ESL learners was compared to 25 EFL learners randomly selected from a larger pool. Both groups completed an aural multiple-choice discourse completion task (MC-DCT), which was developed from a previously validated oral DCT. The aural MC-DCT consisted of 21 items with learner-generated options delivered aurally. A total of 91 native speakers also completed the task as a control group. The results showed an effect of learning environment on learners' selection of conventional expressions. The ESL group selected the conventional expressions in more items than the EFL group on the aural MC-DCT; the differences in the selections by the two groups were item-specific. The observed effect of learning context is discussed as related to individual items and type and modality of the task. The paper also discusses the special make-up of the ESL group due to the pandemic and expansion of the group for future research.

The early phase is the decisive period for the teachers' career. Intellectual styles (particularly thinking styles) play a vital role in teachers' professional development. With the purpose of measuring thinking styles of teachers at the early phase, this study was designed to validate a revised inventory of thinking styles (TSI-R2) for pre-service and early career teachers. A sample of preservice teachers (n = 248) were invited to join the pilot study and a sample of teachers with one year work experience (n = 252) participated in the main study. Exploratory factor analysis, confirmatory factor analysis and reliability analysis were performed. The results suggested that with slight modification, TSI-R2 was a reliable and valid instrument to measure thinking styles of pre-service teachers and early career teachers, which has some implications to enhancing teacher education and development.

Previous studies on students' conceptions of learning science (COLS) have focused primarily on formal environments. In these studies, students' COLS were positively associated with their approaches to and outcomes for science learning. However, little research has been conducted to explore students' conceptions of learning science in informal environments (COLSIE), despite its importance to students' academic achievement. To fill this research gap, this study qualitatively and quantitatively explored Chinese primary school students' COLSIE. First, in Study I, interview data gathered from a group of 80 students were analysed using the phenomenographic method, and ten hierarchical categories of COLSIE emerged (e.g., communicating and explaining). Based on these categories, a survey was developed and distributed to another group of 414 students in Study II. Exploratory factor analysis was conducted to validate the survey, which revealed nine factors matching the ten categories except for the initial categories of applying and understanding. This study also revealed the commonalities and uniqueness of COLSIE in comparison with students' COLS in formal environments. The findings suggested that informal science learning experiences may strengthen students' impressions of science practices. Science educators are encouraged to provide their students with opportunities to engage with science practices in informal environments.

Science educators have highlighted the need to develop students to integrate knowledge across science disciplines to address real-world issues. However, there has been little research about the development of interdisciplinary assessment instruments. In this research, the instrument that measures the level of upper-secondary school students' interdisciplinary understanding of environmental issues was developed and validated based upon Wilson's Construct Modeling framework. After a pilot testing, the revised assessment instrument of interdisciplinary understanding covering five typical environmental problems comprised 14 multiple-choice questions and four constructed-response questions. Five hundred twenty-three eleventh graders, including 279 boys and 244 girls from mainland China, made up the research sample. The partial credit Rasch analysis has verified the reliability and validity of the interdisciplinary understanding instrument. In addition, the results of cluster analysis revealed that over half of the students could use some partially accurate scientific concepts and principles from two or more disciplinary perspectives to deal with a specific environmental issue. The validated instrument can provide insights for assessing and developing upper-secondary school students' interdisciplinary understanding in science education.

Purpose: Since the onset of the COVID-19 pandemic in China, student engagement in online learning has been a critical issue for all educational institutions. The university student engagement inventory (USEI) is the most used scale for assessing the construct of student engagement. The present study aimed to evaluate the psychometric properties of the USEI among 1504 Chinese university students who completed a survey through an online platform between December 2020 and January 2021. Design/methodology/approach: In this cross-sectional study, content validity, construct validity and reliability of the scale were assessed. Findings: The results supported the three-factor model with acceptable goodness of fit (X[superscript 2] (71) = 369.717, p = 0.13, X[superscript 2]/df = 5.207, comparative fit index (CFI) = 0.967, normed fit index (NFI) = 0.960, Tucker-Lewis index (TLI) = 0.958, standardized root mean square residual (SRMR) = 0.030, root mean square error of approximation (RMSEA) (90% CI) = 0.053 [0.049, 0.057]), good internal consistency and construct reliability (Cronbach's alpha and omega coefficient >0.70) and strong convergent validity. Also, the measurement invariance was confirmed across gender. Originality/value: This study showed that the 3-factor structure of USEI with Chinese university students had good construct validity, internal consistency and reliability. It could help measure student engagement in online learning in China.

Many Chinese universities are implementing an educational reform to transform their general English courses into academic English courses. Accordingly, how to assess students' English ability should also be reformed. In this test review, we introduce a school-based general academic English summative test developed by English instructors. An argument-based approach was adopted to analyze the test validity by obtaining students' test data and their reflective responses to the test. This review can provide a practical reference for the development of a valid general academic English summative test by including practices of course instructors and voices of students, two important test stakeholders, in test design.

This study developed and validated an instrument to investigate senior school students' understanding of electrostatics and provide a cognitive diagnostic assessment of their strengths and weaknesses on the related concepts (e.g., electric charge). The instrument included 20 four-tier multiple-choice items and the development process is organized around two activities: the development of the instrument and its validation. The development step defined the secondary concepts and designed the items using the misconceptions related to them. In the validation step, the instrument was applied to 1850 senior high school students from nine schools in two provinces in China, and the collected data were analyzed using the CDM package in R language. This step ensures that the diagnostic reports represent students' conceptual understanding reliably and validly by selecting the best model, analyzing item quality, overall test reliability, and the instrument's structure. The instrument can provide the percentage of students in the test population who possess certain combinations of concepts, the percentage of students in the test population possessing individual concepts, and the fine-grained size of concept proficiency information, which can be integrated as one completed report to issue to students, teachers, and parents to demonstrate students' status of conceptual understanding related to electrostatics. In addition, the construct induced from the diagnostic results can also be aggregated to the classroom, schools for instruction planning or low-stakes decision making, or infer a learning sequence.

The present study was executed with the purpose of validating ELT Certificate Lesson Observation and Report Task (ELTC-LORT), which was developed by China Language Assessment to certify China's EFL teachers by performance-based testing. The ELT Certificate has high-stakes considering its impacts on candidates' recruitment, ELT in China and quality of education, so it is crucially important for its validation so as to guarantee fairness and justice. The validity of task construct and rating rubric went through a process suited for many-facet Rasch measurement supplemented with qualitative interviews. Participants (N = 40) were provided with a video excerpt from a real EFL lesson, and required to deliver a report on the teacher's performance. Two raters graded the records of the candidates' reports using rating scales developed to measure EFL teacher candidates' oral English proficiency and ability to analyze and evaluate teaching. Many-facet Rasch analysis demonstrated a successful estimation, with a noticeable spread among the participants and their traits, proving the task functioned well in measuring candidates' performance and reflecting the difference of their ability. The raters were found to have good internal self-consistency, but not the same leniency. The rating scales worked well, with the average measures advancing largely in line with Rasch expectations. Semi-structured interviews as well as focus group interviews were executed to provide knowledge regarding the raters' performance levels and the functionalities of the rating scale items. The findings provide implications for further research and practice of the Certificate.

The past decade has seen an emerging interest in mapping (aligning or linking) test scores to language proficiency levels of external performance scales or frameworks, such as the Common European Framework of Reference (CEFR), as well as locally developed frameworks, such as China's Standards of English Language Ability (CSE). Such alignment is ultimately a claim about the interpretation of test scores in relation to external levels of language proficiency. To support such a claim, established procedures should be carefully implemented and multiple sources of evidence should be collected. In this research report, we demonstrate the application of a series of steps in building an argument for aligning the scores of the "TOEFL iBT"® test, an international, large-scale language proficiency test of English as a foreign language (EFL), to the levels of the CSE. The alignment process comprised the following steps: (a) establishing construct congruence between the TOEFL iBT test and the CSE; (b) establishing recommended minimum test scores (cut scores), set by local experts, to classify language learners into the local proficiency levels; (c) collection of scores by test takers (N = 1,326) and evaluations of the test takers' proficiency levels by their teachers, based on the local framework; and (d) consideration of the results of other alignment studies in the local context as well as the link between the CEFR and the CSE levels. We conclude with a discussion of the contextual issues that should be considered when interpreting test scores in relation to external proficiency levels. These contextual issues are important considerations because they have the potential to impact score-based decisions on individuals and institutions. We also discuss the implications for similar alignment research.

Based on the articles written by mainland Chinese scholars published in the most influential Chinese and international journals, the present article analyzed the language testing research, compared the tendencies of seven categories between 2000-2009 and 2010-2019, and put forward future research directions by referring to international hot topics. Of all the seven categories of research topics, validity, performance test and China's Standards of English Language Ability were three most popular themes, while classroom assessment, technology, rater/test taker differences and professionalization were much less popular. Except for research on performance test and technology, the other five aspects showed an increase in the second decade, with that of China's Standards of English Language Ability rising the most dramatically. Referring to international research trends, the research predicted that validity, classroom assessment, China's Standards of English Language Ability and professionalization, especially the ethics and social justice, might be the promising research topics for language testers.

Given the importance of student engagement for educational success, it is essential to explore how teachers can stimulate student engagement in online learning environment. However, relatively little research drawing from self-determination theory has examined the links between teaching motivations and student engagement. To this end, this study was conducted to survey 414 Chinese college students' perceptions of teaching motivations, their own intrinsic and extrinsic motivations during the learning process, as well as their engagement with online learning. The findings indicated that the survey had satisfactory validity and internal consistency. Structural equation model revealed the interrelationships between autonomy-supportive teaching motivations, controlling teaching motivations, student intrinsic and extrinsic motivations, and student engagement. The results showed that in online learning environments, autonomy-supportive teaching motivations and student intrinsic motivation were positively related to student engagement. Unexpectedly, controlling teaching motivations and student extrinsic motivation had no significant effect on student engagement. Moreover, the mediating effects of student intrinsic motivation including perceived autonomy, competence, and relatedness offered a deeper understanding of the association between autonomy-supportive teaching motivations and student engagement. The main findings and practical implications together are discussed in depth.

Objectives: The aim of this study was to develop a scale measuring inclusive school quality from parents' perceptions (Parents' perceptions of Inclusive School Quality, PISQ) and investigate the perceptions of Chinese parents of children with disabilities of inclusive school quality using the PISQ scale. Method: The data for 78 parents of children with disabilities were used to measure the reliability and analyse the items of the PISQ scale. The data for 255 parents of children with disabilities were used to verify the construct validity and internal consistency of the PISQ scale and investigate parents' perceptions of inclusive school quality in China. SPSS 21.0 and AMOS 24.0 were used for the statistical analyses. Results: The newly developed PISQ scale was found to have good construct validity and reliability. The investigation of Chinese parents of children with disabilities indicated that parents' perception of inclusive school quality in China was at a middle-high level and that the parents of children with ASD showed the lowest perception of inclusive school quality. Conclusion: This study reports the development process of the PISQ scale and presents a valid scale for measuring inclusive school quality in the future. Additionally, this study provides implications for improving the inclusive practices of regular Chinese schools.

This article examines the changes taken place over the decades after a test-based accountability system was in place and proposes an agenda for consequential research of the testing program. The article starts with a review of theories of test-based accountability (Supovitz, Journal of Educational Change 10:211-227, 2009) and a socially grounded, integrated approach to test validation (Chalhoub-Deville, Language Testing 33:453-472, 2016; Chalhoub-Deville and O'Sullivan, Validity: Theoretical development and integrated arguments, 2020). Following Supovitz's frameworks of accountability testing, the case of the College English Test (CET), a national English language testing program in China, is analyzed to explain the nature of the consequences of accountability testing and show the motivational effects, alignment effects, the informational role and the symbolic values of the accountability testing program. The case analysis highlights the importance of investigating consequences of an accountability test at the aggregate and the system levels. Guided by Chalhoub-Deville and O'Sullivan's (Validity: Theoretical development and integrated arguments, 2020) integrated validity framework, the article delineates the claims for the theory of action argument and the communication engagement argument for future investigation of the consequences of the CET.

Background: In many countries and regions, such as the United States, Europe and China, a trend has emerged in which students' enthusiasm for STEM is declining. This decline may be related to students' lack of science self-efficacy. An accurate examination of students' science self-efficacy can provide a research foundation for how to cultivate it. This paper used mixed methods to develop a valid science self-efficacy scale for high school students, focusing on the perceived competence dimension. A cross-sectional analysis exploring and interpreting differences across grades and genders in science self-efficacy among high school students was conducted. Subsequently, a 1-year longitudinal study was conducted on the development of science self-efficacy in China. Results: This study developed a 24-item science self-efficacy instrument based on the Rasch model, and the validity of the instrument was assessed through multiple aspects, including face, content, construct, and predictive validity. This instrument was used to divide students' science self-efficacy into four different levels. A cross-sectional study examining 1564 high school students in 10th--12th grades revealed that students' science self-efficacy exhibited a complex process of decreasing and then increasing by grade. Most girls' science self-efficacy was higher than that of boys for Levels 1 and 4, while for the intermediate levels, i.e., Levels 2 and 3, most boys had higher science self-efficacy than girls. The quantitative and qualitative results of the longitudinal study through a 1-year follow-up of 233 high school students indicated that students' science self-efficacy significantly improved. We revealed inconsistencies between cross-sectional and longitudinal studies of the change in science self-efficacy from 10 to 11th grade. Conclusion: This study makes many contributions. First, we developed a science self-efficacy measurement instrument for high school students with high reliability and validity based on the Rasch model and characterized four different levels of student science self-efficacy. Second, the gender differences in science self-efficacy and the complex changes among grades were explained from the perspective of science self-efficacy level. Finally, students' science self-efficacy significantly improved in the longitudinal study, which was explained by self-efficacy theory and the Chinese core competency-oriented science curriculum.

Language test preparation has often been studied within the consequential validity framework in relation to ethics, equity, fairness, and washback of assessment. The use of independent and integrated speaking tasks in the "TOEFL iBT"® test represents a significant development and innovation in assessing speaking ability in academic contexts. Integrated tasks that involve synthesizing and summarizing information presented in reading and listening materials have the potential to generate new test preparation strategies. This study investigated the experiences of over 1,500 Chinese test takers and 23 teachers who were preparing for the TOEFL iBT speaking tasks. It examined the frequency of use of a number of different test preparation activities and materials, reasons, and expectations for taking preparation courses and the features of preparation courses. In addition, we examined the usefulness of test preparation from two perspectives: students' and teachers' perceptions as well as the relationship between test preparation and performance. Data were collected via questionnaires, focus group discussions, interviews with test takers and teachers, and classroom observations. The data showed that (a) test preparation was a hugely complex, multiple-components construct, and teaching and learning test-taking strategies compose the most prominent feature of intensive preparation courses; (b) there were significant age-related differences in students' preparation activities and focuses, although with small effect sizes; (c) there was a high agreement between teachers and students in their views on the usefulness of test preparation activities; and (d) there existed only a weak relationship between test preparation and performance. The only significant predictor of students' test performance was the frequency of their use of the "TOEFL Practice Online TPO"® practice tests. The findings of the study can enhance our understanding of the pedagogical practices that characterize test preparation programs and contribute to the ongoing validity argument for the TOEFL iBT Speaking test. The implications of the findings for test publishers, test takers, teachers, and test preparation schools are discussed with reference to the instructional, learning, and affective aspects of the multifaceted construct of test preparation.

This study aims to develop and validate a new instrument for measuring chemistry teachers' perceptions of Pedagogical Content Knowledge for teaching Chemistry Core Competencies (PCK_CCC) in the context of new Chinese chemistry curriculum reform. The five constructs and the initial 17 items in the new instrument were contextualized by the PCK pentagon model (Park S. and Oliver J. S., (2008), J. Res. Sci. Teach., 45(7), 812-834.) with the notions of the Senior High School Chemistry Curriculum Standards (Ministry of Education, P. R. China, 2017). 210 chemistry teachers from a University-Government-School initiative voluntarily participated in this study. The findings from item analysis, confirmatory factor analysis and correlation analysis provide sufficient empirical evidence to support the convergent and discriminant validity of the instrument. The concurrent validity of the instrument was confirmed by testing mean differences among teacher demographic groups. The high Cronbach's coefficient alpha values show good internal consistency reliability of the instrument. Integrating the evidence from theory and data, we documented a valid and reliable PCK_CCC instrument with five constructs consisting of 16 items. This study provides a thorough process for developing and validating instruments that address teacher perceptions of their PCK in a particular subject domain. The valid and reliable PCK_CCC instrument would be beneficial for teacher education researchers and teacher professional programs.

The past three decades have witnessed a boost of interest in vocabulary learning in EFL contexts since Meara (1980) identified it as 'a neglected aspect of language learning' (p. 221). A mushrooming amount of literature has emerged in various aspects of vocabulary and its acquisition (e.g., Carter, 1998; Coady & Huckin, 1997; Manyak, 2010; Meara, 1995, 2005; Nation, 1990, 2006; Read, 2000; Schmitt, 2000; Schmitt & McCarthy, 1997). With a movement from teaching-orientedness to learner-certeredness and learner autonomy, vocabulary learning strategies seem to have gained its legitimacy as one auxiliary approach to vocabulary learning. Despite this, there appears no satisfactory instrument particularly for assessing vocabulary learning strategy use in an EFL context, although a few researchers have tried to do so (e.g., Gu & Johnson, 1996; Schmitt, 1997). To this aim, a new inventory for vocabulary learning, the Strategies Inventory for Vocabulary Learning (SIVL) was proposed for Chinese EFL university learners. To validate the instrument, confirmatory and exploratory factor analyses were employed to assess its psychometric properties. Results showed that the hypothesized theoretical model proved to be a good representation of the sample data, and that the SIVL exhibited satisfactory psychometric features. This positive evidence indicates that the SIVL can serve as a reliable and valid research instrument for assessing Chinese EFL university learners' vocabulary learning strategy use. It is suggested that the SIVL can be a valuable resource for EFL learners and practitioners in that it can raise their awareness of strategy use and strategy training by employing this instrument, leading to more successful vocabulary teaching and learning.

Chinese higher education is experiencing rapid development and growth. With tremendous resources invested in higher education, policy makers have requested more direct evidence of student learning. However, assessment tools that can be used to measure college-level learning are scarce in China. To mitigate this situation, we translated the critical thinking test from the ETS® Proficiency Profile (EPP) into Chinese. EPP has been widely used in the United States to assess general college learning outcomes. We pilot tested the EPP--Chinese test with students from a university in China. Results suggest that (a) the test is unidimensional and therefore is sufficient to report a total score from a practical standpoint; (b) the total score reliability is satisfactory; (c) most items showed moderate correlations with the total score, but the translation of one item needs additional revision; (d) the test is correlated with related constructs (e.g., the Chinese college entrance examination and a national English test); and (e) no item showed differential item functioning or was found to be biased toward any subgroup. In summary, the Chinese version of the critical thinking test showed potential as a suitable assessment tool for Chinese college students.

It is important to identify suitable parameters to measure and assess the quality of physical education. This paper aims to understand such measuring tool by focusing on professional perceptions and areas of concern of quality physical education (QPE). The research survey includes data gathered from 799 professionals in 11 cities in Asia. These professionals had experience working at primary schools (n = 155), secondary schools (n = 249), and universities (n = 395). To examine professional perceptions of quality physical education, at the preliminary stage of this study, 65 potential items were identified; of these, 48 items on 8 dimensions were retained, following an exploratory factor analysis. The 8 dimensions covered "skill development and bodily awareness" ([alpha] =0.935), "facilities and norms in PE" ([alpha] =0.932), "quality teaching of PE" ([alpha] =0.923), "plans for feasibility and accessibility of PE" ([alpha] =0.824), "social norms and cultural practice" ([alpha] =0.825), "governmental input for PE" ([alpha] =0.859), "cognitive skills development" ([alpha] =0.920), and "habituated behaviour in physical activities" ([alpha] =0.933). These factors indicated excellent internal consistency and model fit, enabling the construction of highly relevant statements to measure professional perceptions of QPE.

China has witnessed a considerable expansion of private higher education institutions (HEIs) over the last two decades, and research has shown that teacher leadership (TL) is an essential aspect of providing quality higher education. This study proposed a model to explain TL and the factors that contribute to it in private HEIs. A sample of 4,152 participants responded to an 11-item questionnaire using a 5-point scale designed to measure three variables: TL, teacher self-efficacy (TSE), and teacher competence (TC). The results showed that the three variables were valid in explaining TL and the factors that contribute to it. Hypothesis tests revealed that both hypotheses were supported. Finally, the results revealed that TSE and TC are both significantly associated with TL. The practical and theoretical implications of these findings and the scope for future research are discussed.

Performing routines in interlanguage is vitally important for EFL learners since it can cause embarrassment between speakers from different cultures. The present study aims to 1) investigate the reliability and validity of an interlanguge pragmatic competence test on routines in a Chinese EFL context with multiple choice discourse completion task (MDCT), 2) examine the statistical characteristics of the test, and 3) explore the test takers' strategies used in the process of taking the test. Altogether 390 students from China took part in the study. The data were collected with the interlanguage pragmatic competence test on routines and think aloud protocol. Data analyses methods included the split-half method, item analysis theory, descriptive statistics, an independent T-test and content analysis. The results indicate that 1) the interlanguage pragmatic competence test on routines showed strong reliability and validity in the Chinese EFL context, 2) interlanguage pragmatic routines were not easy for the Chinese EFL learners, and situational routines were easier than functional routines, and 3) three strategies: repeated reading, interpreting or translating, and weighing options or justifying responses, were used in the process of taking the test by the Chinese EFL learners.

Speech acts are the major concern of interlanguage pragmatists. The present study aimed to 1) examine the reliability and validity of an interlanguage pragmatic (ILP) competence test on speech acts in a Chinese EFL context, and 2) investigate EFL learners' variations of ILP competence by language proficiency. Altogether 390 students participated in the present study. The students were divided into three groups based on their language proficiency. The data were collected with an ILP competence test and semi-structured interviews. The ILP competence test was in the form of a written discourse completion task (WDCT), including ten speech acts and 30 situations. Data analysis methods included the Many Facets Rasch Model (MFRM), one-way ANOVA, post-hoc Scheffe test and content analysis. The results indicated that the ILP competence test was with high reliability and validity, and variations existed in four aspects of conducting speech acts: 1) use of correct speech act, 2) typical expressions, 3) amount of speech and information, and 4) degrees of formality, directness and politeness, according to the level of language proficiency. Overall, the students with higher language proficiency performed better than the ones with lower language proficiency.

In this study, data collected from 875 college freshman and sophomore students enrolled in a 4-year university in central China are used to examine the applicability and validity of a Chinese version of Holland's Self-Directed Search (SDS) that was adapted in the 1990s. The total sample was randomly divided into two groups. Data from the first half of the sample (n[subscript 1] = 438) were used to analyze the reliability and validity of the adapted SDS, and 52 items were identified for removal due to lack of cross-cultural fit, lack of validity to measure the intended construct, and/or being obsolete due to progress in societal and occupational trends. The SDS before and after item removal were further compared using confirmatory factor analysis with the second half of the sample (n[subscript 2] = 437). Practical implications of this empirical modification approach are discussed. This study contributes to the literature by extending the current discussion about cross-cultural adaptation of Holland's inventories and by suggesting a methodological approach to improve the reliability and validity of adapted assessment tools.

The current study aims to explore the cognitive validity of the iBT TOEFL reading test by investigating test takers' eye movements on individual items. It is assumed that successful test takers would adopt the intended reading processes, the same types and levels of cognitive processes that they would use for real-world reading tasks. Forty-seven Chinese ESL students participated in the study, in which they took the TOEFL reading practice testlet on a computer, completed comprehension subskill tasks, and had stimulated recall interviews. Results showed that test takers tend to rely heavily on careful reading skills, while expeditious reading skills were rarely activated. The scope of reading was often restricted to the local level; learners hardly read more than a paragraph to answer questions. In some factual question items, successful readers were more efficient in reading and locating key information, whereas such group differences were not found in other items. Lastly, the gaze plots suggest that learners' eye movements manifest various interactions between comprehension subskills, primarily subject to bottom-up linguistic processing. The limitations and implications of learners' eye-tracking data for test validation will be further discussed.

With technology-enhanced learning playing an increasingly important role in higher education, university students are enjoying unprecedented benefits while likely experiencing technostress due to increased requirements and different academic expectations of technology-enhanced learning. This study aimed to develop a psychometric scale to measure university students' levels of technostress in technology-enhanced learning. An initial technostress scale was developed drawing on the person-environment fit theory and prior research on technostress. In total 620 students were recruited from two public universities in China. Three rounds of study were performed to validate the initial technostress scale using factor analysis, the Rasch model, and multilevel linear regression. The finalised technostress scale with 8 items demonstrated robust psychometric properties, including high internal consistency, validity, uni-dimensionality, and measurement invariance across students of different demographics. The development of the technostress scale is an important step in identifying maladapted students and preserving their wellbeing so as to increase their constructive and active participation in technology-enhanced learning.

ELTeach is an online professional development program developed by Educational Testing Service (ETS) in collaboration with National Geographic Learning. The ELTeach program consists of two courses: English-for-Teaching and Professional Knowledge for English Language Teaching (ELT). Each course includes a coordinated assessment leading to a score report and certificate for teachers of English as a foreign language (EFL).The Test of English-for-Teaching ("TEFT"™), the assessment component of the English-for-Teaching course, measures EFL teachers' command of English for teaching English in classroom settings, as presented in the course. In this study, we examined the internal structure of the TEFT assessment. Results of the analyses demonstrated the role of both skill and content in representing the test's internal structure. The final parcel model had a higher-order general factor and four first-order factors corresponding to reading, writing, listening, and speaking. The findings support the current score reporting practice, that is, to report a total scaled score along with score information on language skills and on language use in specific content areas.

In this study, based on the analysis of existing definitions of emotional labor, operational definition of teachers' emotional labor is given and questionnaire on emotional labor among primary and secondary school teachers is developed. Research results: exploratory factor analysis shows that teacher's emotional labor involves three dimensions including surface acting, active deep acting and passive deep acting; the questionnaire has good reliability and validity; confirmatory factor analysis shows that emotional labor questionnaire involving three factors is established, which further verifies the validity of scale on emotional labor among primary and secondary school teachers.

Since its first pilot study was launched in 2003, China Accreditation Test for Translators and Interpreters (CATTI) has developed into the most authoritative translation and interpretation proficiency qualification accreditation test in China and played an important role in assessing and cultivating translators and interpreters. Based on the analysis of its form and content, the current study appraises CATTI in terms of reliability, validity, practicality, fairness and washback. The results suggest that CATTI has a satisfying validity and reliability, but the amount and types of test items, test content, evaluation rubrics and transparency are expected to be improved.

One fundamental way to determine the validity of standardized English-language test scores is to investigate the extent to which they reflect anticipated learning effects in different English-language programs. In this study, we investigated the extent to which the "TOEFL iBT"® practice test reflects the learning effects of students at intensive English programs in the United States and China, as well as extracurricular English-learning activities that may be associated with the expected learning effects. A total of 607 students at the high school level or beyond participated in the United States and China, including 111 students who took 2 forms of the practice test under a pretest and posttest design. The results showed moderate to substantial levels of improvement on each of the TOEFL iBT sections, with different score gain patterns for students in the United States and China. We concluded that students who study at English programs similar to those included in this study can improve their English-language proficiency levels at least moderately over 6 months or longer, as indicated by changes in their scores on the TOEFL iBT practice test. This improvement is consistent with an interpretation of TOEFL iBT scores as indicators of English-language proficiency.

Traditional Chinese medicine (TCM) is an important component of China's medical system. How to educate TCM practitioners in China, therefore, has become a crucial issue. To contribute to this issue, the current research identified the competency model of TCM practitioners in China and developed an evaluation for TCM students. We combined Bloom's taxonomy with our model construction to examine the current TCM practitioners' requirements. Following the protocol of developing a measurement, we interviewed 20 TCM practitioners, conducted two rounds of surveys in China (n = 220/n = 489), ran exploratory factor analysis (EFA) and confirmatory factor analysis (CFA), and examined the correlation between the measurement and self-rated outcomes. The evaluation tool of TCM students consists of five factors: "modern medical knowledge and skills," "TCM medical knowledge and skills," "professional ethics," "communication and cooperation skills," and "clinical practices." The reliability and validity of our evaluation tool were testified. Our work may contribute to TCM education in China.

Background: This study aims to empirically answer the question of whether the role of sub-reading skills changes depending on the test format (e.g., multiple-choice vs. open-ended reading questions). The test format effect also addresses the issue of test validity--whether the reading test properly elicits construct-relevant reading skills or ability. The research questions guiding the study are as follows: (1) Do test scores differ systematically depending on the test format? (2) Do the predictors of test scores differ systematically depending on the test format? Methods: Ninety Chinese ESL students participated in the study at the post-secondary level and took two TOEFL practice testlets, one with multiple-choice (MC) questions and the other with stem-equivalent open-ended (OE) questions. In addition to the reading comprehension test, the participants completed a vocabulary test, grammar test, word recognition task, sentence processing task, working memory test, and strategy questionnaires (reading and test-taking strategies). Results: The participants performed better on the MC questions than the corresponding OE questions, regardless of the text effect. More importantly, an L2 reading test in a different format involved different sub-reading components; vocabulary knowledge was the only significant predictor of MC test scores, whereas for the OE reading test, grammar knowledge, word recognition skills, and possibly inferencing strategies were found to be significant predictors. Conclusion: Despite a number of limitations, the value of this study lies in the effort to empirically test format effects by taking a componential approach to reading. The findings suggest the possibility that differently formatted reading questions may tap into different sub-reading component skills. To accurately reveal the underlying structure of the reading construct being tested in MC and OE tests, however, we call for a larger scale data collection with mixed research methods employed.

In physics education research (PER), concept inventories (CIs) have become standard instruments for assessing students' learning throughout instruction. To promote widespread use of concept inventories, previous studies have developed an approach to split a full length CI into short versions of CIs. This research extends the existing method to fully utilize the item response theory framework in equating and linking between the short CIs and the full length CIs. Three quantitative studies have been conducted: First, the extended algorithm is applied to divide the Brief Electricity and Magnetism Assessment (BEMA) into two half-length BEMAs (HBEMAs). Through a series of test-equating and validation analysis, the HBEMAs are confirmed to measure the same latent constructs of student understanding of electricity and magnetism to that of the original BEMA at a similar level of reliability. The second study establishes equivalent score conversions among the three versions of BEMA using the Stocking-Lord method, which has the best performance on equating error reduction among several methods explored. It is also confirmed that the equivalent statistical characteristics of the three versions of BEMA are equity and population invariant. In the third study, the extended algorithm is applied to link and compare the BEMA and the Conceptual Survey of Electricity and Magnetism (CSEM). After linking the BEMA and CSEM assessment scales, it becomes possible to directly convert and compare students' performances on the two CIs. It is found that the scales of BEMA and CSEM are almost identical after scale transformation. Based on these studies, it can be suggested that all short and long versions of BEMA and CSEM can be used interchangeably after scale transformation.

The maximal reliability of a congeneric measure is achieved by weighting item scores to form the optimal linear combination as the total score; it is never lower than the composite reliability of the measure when measurement errors are uncorrelated. The statistical method that renders maximal reliability would also lead to maximal criterion validity. Using a career satisfaction measure as an example, the present article calculated the maximal reliability and maximal criterion validity and compared them with the composite reliability and the scale criterion validity, respectively. The improvement of reliability and validity indicated that the optimal linear combination is preferred when forming a total score of a measure. The Mplus codes for analyzing maximal reliability, maximal criterion validity, and related parameters are provided.

The authors question the answer the national Common Core State Standards (CCSS, 2010) claims. The questions center on the validity of the new standardized tests based on the CCSS and teachers' evaluations being tied to student test scores on flawed tests. The proposed tests on the CCSS will position children as deficient, and will not recognize the Funds of Knowledge children and their families bring to the educational transaction. The developers of the proposed new tests seem particularly uninformed on much research of how children learn new vocabulary. The authors question the literary theory (New Criticism) and learning theory (information processing) that undergird the CCSS, which exclude theories and research (transactional theory, critical literacy, Funds of Knowledge, arts-based research) that could be beneficial, especially to children of color and the poor. The authors question the validity of dictating percentages of informational and literary texts, and the lack of emphasis on the emotional lives of children (the word "analysis" appears 94 times in the CCSS). They question whether the CCSS are truly internationally benchmarked when children are not biologically nor developmentally capable of some of the demands in the early grades, and impose standards not shared by Finland and China that may make American students less competitive, not more.

This paper aims at developing a procedural framework for the development and validation of diagnostic speaking tests. The researcher reviews the current available models of speaking performance, analyzes the distinctive features and then points out the implications for the development of a procedural framework for diagnostic speaking tests. On basis of the overview and the experience gained from the development of Diagnostic College English Speaking Test, The researcher elaborates on the process of the development of the procedural framework. The framework is composed of four major phases: phase 1 needs analysis, phase 2 test design, phase 3 test piloting, administration and validation, and phase 4 test impact. Each phase is further divided into several steps, each having concrete aims and objectives. Literatures relevant to each component of the procedural framework are also recommended by the researcher to help teachers to develop their own diagnostic tests. The paper concludes with suggestions for further research on the development of computerized diagnostic assessment systems.

The purpose of this study was to develop and evaluate a History Interest Inventory (HII) to measure Chinese K-12 students' history interest. In this study, data were drawn from two independent samples of a city in Eastern China, and exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were conducted to reduce instrument size and evaluate the construct validity associated with this HII. Furthermore, multiple regression analysis was run to evaluate the weights of each component in predicting students' achievement. The indices of construct validity and reliability indicated that the History Interest Inventory (HII) was an effective and reliable instrument to measure Chinese students' history interest. To apply this instrument to students from other countries, it is suggested that a CFA be conducted with the samples from target countries to check its validity and reliability.