Search

The United States continues to be a top destination for international students pursuing an advanced degree. Some information about the characteristics of international students applying to graduate programs in the United States is available, but little is known about how these characteristics are related to test taker performance on graduate admissions tests and how performance may be related to graduate program characteristics. The purpose of this study was to investigate different patterns of performance of international test takers from four cultural regions and two large countries (China and India) on both the "GRE"® test and the "TOEFL"® test and the relationship with demographic and graduate program characteristics. Using finite mixture modeling, we investigated the most common score profiles using GRE and TOEFL for international students intending to pursue a graduate program within the United States; evaluated the demographic and college-level factors related to the profiles; and evaluated whether the profiles were differentially associated with gender, intended field of study, and intended degree level. Results showed the following broad patterns of results: (a) Most countries and cultural regions, except for the Middle East, had three or four latent profiles representing low, medium, and high scores on the GRE and TOEFL sections; (b) two high-performing profiles were found in Confucian Asia, one with higher GRE Quantitative Reasoning scores and the other with higher scores on GRE Verbal and TOEFL; (c) regardless of profile, test takers from China performed highest on the GRE Quantitative Reasoning section as compared to other GRE and TOEFL section scores; (d) in general, there was a relationship with students in the lower performing profiles taking the TOEFL and GRE multiple times; (e) regardless of country or cultural region, men were represented more than women overall and across most of the profiles; and (f) test takers showed a preference for science-, technology-, engineering-, and mathematics-based fields and master's degrees, but this varied across country and cultural region. Implications for future research are discussed.

This proof-of-concept study examined the feasibility of a new scoring procedure designed to reduce the time of scoring a video-based public speaking assessment task. Instead of scoring the video in its entirety, the performance was evaluated based on content-related (e.g., speech organization, word choice) and delivery-related (e.g., vocal expression, nonverbal behaviors) dimensions. Content-related dimensions were scored based on speech transcripts, while delivery dimensions were scored using a video thin-slicing technique, where scores were assigned based on brief vignettes of a video rather than the complete performance. Initial feasibility data were collected from 4 novice raters scoring 10 video performances. Results indicated that for delivery dimensions, four 10 second thin slices yielded interrater consistency reliability somewhat similar to that of full-video scoring, and additional slices resulted in only small improvements. For transcription-based scoring, while raters were consistent, their scores had low correlations with criterion scores, which was likely due to differences in the modality (video vs. text). Video thin-slicing appears to be a promising scoring technique for relevant constructs. Further testing of a combination of audio and transcript is recommended for scoring content-related constructs.

Economic globalization and interdisciplinary advancements have increased the demand for college graduates to possess transferable skills that would allow them to contribute effectively to the modern workforce. In particular, transferable competencies such as civic competency and intercultural competency are critical for colleges to prepare responsible citizens and productive workers. Despite the recognized importance, the choice and quality of assessment for such competencies have been fairly limited due to the challenges in defining such complex, multidimensional constructs and identifying item types that can adequately assess them. In this report, we describe the principles we followed to operationalize definitions for civic competency and intercultural competency and the process we followed to design assessments for these 2 competencies. Findings from a large-scale pilot test are reported. Results showed that these multidimensional constructs can be adequately assessed and that there is room for students to improve in these areas. Implications for higher education institutions on how to promote these critical competencies are discussed.

Oral communication has been identified as an important skill by higher education institutions and by the workforce community. Despite its importance, minimal research has been conducted around the development of tasks to measure oral communication skills and behaviors. The purpose of this preliminary study is to evaluate the different factors related to the design and scoring of oral communication tasks. For this study, oral communication refers specifically to video-based oral presentations using a webcam. The study administered 2 informative and 2 persuasive speech tasks of varying length (2 or 4 minutes) to 8 college students. Performance was scored holistically on overall performance, and on content and delivery dimensions by 2 raters. Results show that (a) interrater reliability estimates may be impacted by task type and length, (b) there is no variation in the three scoring criteria, (c) a minimum of 2 tasks is necessary to achieve satisfactory reliability on an oral communication measure, and (d) supporting materials may impact student performance. Results of this preliminary study provide guidance on how to further this foundational research. Future research and next steps are discussed.

Critical thinking has become an essential skill required by both higher education and the workforce. Research to date has reported moderate cross-sectional learning differences in critical thinking as students progress through a 4-year college. However, retention, differential participation, and students' low test-taking motivation possibly confounded conclusions from prior studies. Controlling for such factors, we found a cross-sectional difference of 0.24 SDs after 4 years in college (n = 2,381 students, 46 institutions), considerably smaller than what's reported in prior studies. Natural science majors performed the highest and business majors performed the lowest. Minority students achieved only half of the cross-sectional difference (0.18 SDs) of white students (0.39 SDs), and only 6% of them scored at the Advanced level, compared to 24% of white students.

The "ETS"® Proficiency Profile (EPP), a college-level assessment, has been widely used to evaluate general education student learning outcomes (SLOs) in college. The purpose of this study was to investigate validity evidence for the EPP by evaluating the relationship with outcomes such as student retention, cumulative grade point average (GPA), and degree attainment, and by investigating differential validity across subgroups and cross-sectional learning gains. Three main conclusions were drawn from this study: (a) Students made significant learning gains from freshman to senior year using EPP scores; (b) freshman scores showed modest relationships with cumulative GPA at various points in college and senior scores showed strong relations with final-year cumulative GPA; and (c) differential validity was found across gender, race, and college major when looking at the relationship between EPP scores and first-year and sophomore GPA. Implications of these results are discussed.

Civic learning is increasingly recognized as important by the higher education and workforce communities. The development of high-quality assessments that can be used to evaluate students' civic learning during the college years has become a priority. This paper presents a comprehensive review of existing frameworks, definitions, and assessments of civic-related constructs from approximately 30 projects relevant to higher education, and includes a discussion of the challenges related to assessment design and implementation. Synthesizing information from the review, we propose an assessment framework to guide the design of a next-generation assessment of individuals' civic learning that takes advantage of recent advances in assessment methods. The definition identifies 2 key domains within civic learning: civic competency and civic engagement. Civic competency encompasses 3 areas (civic knowledge; analytic skills; and participatory and involvement skills), and civic engagement also captures 3 areas (motivations, attitudes, and efficacy; democratic norms and values; and participation and activities). We discuss item formats and task types that would ensure fair and reliable scoring for the assessment. The review of definitions of civic learning and its components developed by organizations, the proposed assessment framework, and assessment considerations presented here have potential benefits for a range of higher education institutions. This includes institutions that currently have students engaged in relevant curricular or cocurricular activities and also institutions that would find assessments of civic competency and engagement helpful in program development or in evaluating students' accomplishments.

Objective: Using data from "HEIghten"® Critical Thinking, a student learning outcomes assessment, the purpose of this study was to evaluate what variables are associated with higher critical thinking performance for students enrolled in various community college programs and to evaluate performance differences across demographic and college-level subgroups as well as student perceptions. Method: With data from 1,307 students enrolled across 34 U.S. and Canadian higher education institutions (72% enrolled in 2-year institutions), we utilized a hierarchical regression to identify variables associated with critical thinking performance. Critical thinking performance differences were evaluated using analyses of variance (ANOVAs) and "t"-tests across student demographic and college experience subgroups and across student perceptions. Results: Results of this study showed (a) consistent significant predictors associated with higher critical thinking performance; (b) a positive relationship between critical thinking performance and the frequency of using critical thinking in college courses; (c) significant, but relatively small performance differences across demographic and college experience subgroups; and (d) positive relationships between student perceptions and critical thinking performance. Conclusion: This study added to the limited literature evaluating critical thinking skills for community college students. Overall, results suggest that institutions should focus attention to the frequency at which students are using critical thinking throughout their courses, which could increase student performance in this particular area, especially if critical thinking is an explicit outcome within the course. Results also suggested the need to emphasize critical thinking skills more across various community college programs and across non-STEM-focused programs. Suggestions for future research are discussed.

Quantitative literacy has been recognized as an important skill in the higher education and workforce communities, focusing on problem solving, reasoning, and real-world application. As a result, there is a need by various stakeholders in higher education and workforce communities to evaluate whether college students receive sufficient training on quantitative skills throughout their postsecondary education. To determine the key aspects of quantitative literacy, the first part of this report provides a comprehensive review of the existing frameworks and definitions by national and international organizations, higher education institutions, and other key stakeholders. It also examines existing assessments and discusses challenges in assessing quantitative literacy. The second part of this report proposes an approach for developing a next-generation quantitative literacy assessment in higher education with an operational definition and key assessment considerations. This report has important implications for higher education institutions currently using or planning to develop or adopt assessments of quantitative literacy.

Critical thinking is one of the most important skills deemed necessary for college graduates to become effective contributors in the global workforce. The first part of this article provides a comprehensive review of its definitions by major frameworks in higher education and the workforce, existing assessments and their psychometric qualities, and challenges surrounding the design, implementation, and use of critical thinking assessment. In the second part, we offer an operational definition that is aligned with the dimensions of critical thinking identified from the reviewed frameworks and discuss the key assessment considerations when designing a next-generation critical thinking assessment. This article has important implications for institutions that are currently using, planning to adopt, or designing an assessment of critical thinking.

Community colleges currently enroll about 44% of the undergraduate students in the United States and are rapidly expanding. It is of critical importance to obtain direct evidence of student learning to see if students receive adequate training at community colleges. This study investigated the 10-year trends of community college students' (n = 46,403) performance in reading, writing, mathematics, and critical thinking, as assessed by the ETS[TM] Proficiency Profile (EPP), an assessment of college-level learning outcomes. Results showed that community college students caught up with and significantly outperformed students from liberal arts colleges by the end of the 10-year period and made significant improvement in critical-thinking skills. An increasing gender gap was observed in mathematics at community colleges. Prevalent ethnic minority and English as a second language (ESL) gaps were noted but gaps between ESL and non-ESL students and between Hispanic and White students were decreasing. Additionally, Asian students at community colleges showed an overall decline in performance. Findings from this study provide significant implications for community college leaders, researchers, and policymakers.

Twenty-first century and higher-learning skills such as critical thinking are highly desired for new college graduates entering the workplace, especially with rapid changing technology and a more globalized economy. As a result, the importance of critical thinking development at higher education institutions has increased. The purpose of this study was to examine the effect of student and institutional variables on the difference in critical thinking scores between first-year and third/fourth-year students at higher education institutions. We examined the extent to which institutional variables moderate the relationship between student variables and the critical thinking score difference. We used a multi-level modeling approach to account for the clustering of students in institutions. Results of this study showed that: (a) institutional variability accounted for 15% of the variance in estimated critical thinking scores, (b) third/fourth-year students had higher critical thinking scores as compared to their first-year peers even when controlling for college admissions score, and (c) in the final model, the main effects of college admission scores, student-faculty ratio, and percent Black/African American were statistically significant; however, retention was the only significant moderator of the relationship between critical thinking scores and class level. Implications and future research are discussed.

Scientific argumentation is one of the core practices for teachers to implement in science classrooms. We developed a computer-based formative assessment to support students' construction and revision of scientific arguments. The assessment is built upon automated scoring of students' arguments and provides feedback to students and teachers. Preliminary validity evidence was collected in this study to support the use of automated scoring in this formative assessment. The results showed satisfactory psychometric properties related to this formative assessment. The automated scores showed satisfactory agreement with human scores, but small discrepancies still existed. Automated scores and feedback encouraged students to revise their answers. Students' scientific argumentation skills improved during the revision process. These findings provided preliminary evident to support the use of automated scoring in the formative assessment to diagnose and enhance students' argumentation skills in the context of climate change in secondary school science classrooms.

Test accommodations for English learners (ELs) are intended to reduce the language barrier and level the playing field, allowing ELs to better demonstrate their true proficiencies. Computer-based accommodations for ELs show promising results for leveling that field while also providing us with additional data to more closely investigate the validity and effectiveness of those accommodations. In this study, we evaluate differences across non-ELs and two EL groups in their decision to use either of two computer-based accommodations on high school history and math assessments. We also evaluate differences in response times across these groups. Results showed that ELs used accommodations more than non-ELs; however, many students did not use any accommodations, and use decreased as the assessment progressed. In addition, students had longer response time for items with accommodations in history but not mathematics. Recommendations for future research in accommodations for ELs are discussed.

This study examines learning gains of college students' performance in critical thinking, reading, writing, and mathematics as assessed by the ETS Proficiency Profile (EPP). In this study, students' college learning gain was estimated by calculating the score differences between their first and last test administrations. Results revealed that (a) after being in college for one or two years, students did not demonstrate significant learning gains, (b) after three or more years, students made small to moderate gains on the EPP total score, and reading and mathematics subscales, (c) after four or five years, students made small to moderate learning gains on EPP total score and all four subscales, and (d) among various demographic and college-level variables, college experience was the largest significant predictor of students' learning gain, followed by first-year GPA. Implications of these results are discussed.

Learning outcomes assessment has been widely used by higher education institutions both nationally and internationally. One of its popular uses is to document learning gains of students. Prior studies have recognized the potential imbalance between freshmen and seniors in terms of their background characteristics and their prior academic performance and have used linear regression adjustments for these differences, which some researchers have argued are not fully adequate. We explored an alternative adjustment via propensity score weighting to balance the samples on background variables including SAT score, gender, and ethnicity. Results involving a cross-sectional sample of freshmen and seniors from seven groups of majors within a large research university showed that students in most of the majors demonstrated significant learning gain. Additionally, there was a slight difference in learning gain rankings across major groupings when compared to multiple regression results.

Social network analysis (SNA), a methodological approach that enables the mathematical examination of interprofessional relationships, can be an important tool for understanding and leveraging the social relationships that support and restrain instructional innovation and the quality and pace of school reform initiatives. In this article, we explicate the conceptual underpinnings of SNA and summarize how it has been used in a range of preK-12 educational evaluation contexts. A school-based case study is presented in which school leaders examined formal and advice-seeking networks among teachers, staff, and administrators and used the findings to reduce teacher isolationism, promote efficient communication, and increase system capacity for instructional innovation. Challenges associated with SNA for educational evaluation and school improvement are also discussed.

Abstract: The ETS® Proficiency Profile (EPP), a college‐level assessment, has been widely used to evaluate general education student learning outcomes (SLOs) in college. The purpose of this study was to investigate validity evidence for the EPP by evaluating the relationship with outcomes such as student retention, cumulative grade point average (GPA), and degree attainment, and by investigating differential validity across subgroups and cross‐sectional learning gains. Three main conclusions were drawn from this study: (a) Students made significant learning gains from freshman to senior year using EPP scores; (b) freshman scores showed modest relationships with cumulative GPA at various points in college and senior scores showed strong relations with final‐year cumulative GPA; and (c) differential validity was found across gender, race, and college major when looking at the relationship between EPP scores and first‐year and sophomore GPA. Implications of these results are discussed. [ABSTRACT FROM AUTHOR]

Quantitative literacy has been identified as an important student learning outcome (SLO) by both the higher education and workforce communities. This paper aims to provide preliminary evidence of the psychometric quality of the pilot forms for HEIghten quantitative literacy, a next-generation SLO assessment for students in higher education. We evaluated the psychometric quality of the test items (e.g., item analyses), individual- and group-level reliability, the relationship with student performance and related variables (e.g., grade point average) as well as student perceptions, and differences across college-related and demographic subgroups. Our study used data from a pilot test administered to over 1,500 students at 23 higher education institutions in the United States. Results showed that (a) overall, items were functioning well, but a small portion of items should be dropped due to unsatisfactory performance; (b) correlations across sub-areas of the assessment were very high indicating that the assessment may be unidimensional; (c) reliability estimates similar to existing SLO assessments were found at both individual and group levels; (d) assessment scores correlated positively with high school and college GPA, number of math college courses, self-rated quantitative literacy skills, and college admissions scores; (e) students had positive perceptions about the assessment; and (f) performance differences were found across institution type, college majors, gender, racial/ethnic groups, and language groups, but not across credit-hour categories. Implications for operational test development and understanding of quantitative literacy performance are discussed. [ABSTRACT FROM AUTHOR]