Search

Background/Context: Social network analysis has been used to demonstrate an association between individuals' relationships and their engagement in areas such as public health (Perry & Pescosolido, 2015) and teacher professional development (Penuel, 2009). We extend and apply this approach to the higher education context. We operationalize network questionnaires by studying the personal networks of first-generation college students (FGCS) and the relationship between their network composition and their use of campus services. The FGCS population is an ideal population for this study given how little we know about how these students build relationships and utilize support networks to cope with the unique challenges they face in entering college without the resources associated with having college-educated parents (Pike & Kuh, 2005; Cataldi et al., 2018). Purpose/Objective/Research Question: In this paper, we describe how we developed an instrument to explore FGCS's social networks by modifying the General Social Survey (GSS), a commonly used social network questionnaire. We detail the cognitive interview process we utilized to inform our decisions about our survey language and parameters. By outlining this process, our paper provides a guide for cognitive interviewing that can be applied to other survey instruments. Our paper also provides feedback on the GSS and guidance on modifying this question for a higher education audience. Setting: Cognitive interviews and, subsequently, full-scale survey administration took place at two California Community College campuses and two California State University campuses. Each of these colleges is designated as a Hispanic-serving institution and also serves significant numbers of FGCSs. Population/Participants/Subjects: For cognitive interviews, we recruited FGCS who already had completed 30 or more credits, ensuring that those who participate in cognitive interviews would not be in our subsequent study focused on first-year students. We conducted a total of 17 cognitive interviews across our four partner colleges. Research Design: In order to develop and validate our social network questions, we used a three-phased set of cognitive interviews. Cognitive interviews were developed as a tool for identifying and correcting sources of errors within survey questions (Willis, 2004). As a method, they assist the researcher in learning whether a question as stated is asking what the researcher thinks is being asked and whether specific words and phrases will be interpreted by respondents in the way the researcher expects. We used a "think-aloud interview" style recommended by Desimone & Le Floch (2004). Students were asked to talk through aloud how they understood the question, any concerns they had about answering the question and whether any of the words or phrases were confusing. The interviews use scripted verbal probes to examine the wording, ordering, and format of the items. Data Collection: We collected data in three phases. In the first phase, we asked respondents for feedback on four name generator questions (each one a different modification of the GSS). In phase two, we edited the name generator questions based on feedback from phase one and piloted the edited versions along with name interpreter questions. In phase three, we tested the final versions of the name generators and modified versions of the name interpreter questions. We conducted the interviews online using a PowerPoint presentation. Students were asked to read the full item and respond to the questions using the Zoom chat feature. Data Analysis: We analyzed the data collected at the end of each phase. We met as a team and used interview notes, recordings, and reflection memos to look for themes within the students' responses to determine what language needed modification in order for the wording to reflect the intended question. Findings/Results: In phase one, we learned that students interpreted "important matters" (a commonly used phrase in the GSS) broadly as big events that could be personal, college-related or work related, making it difficult to answer. Instead, students advised making the question context-specific. Therefore, we eliminated the versions of the name generators that referred to important matters generally and focused on the two questions that addressed a college-related scenario. Some respondents indicated that they reached out to college staff for help and that they did not receive the help that they were looking for, but with the original wording ("from whom you have received help") they did not think they could include these individuals. Based on this feedback, we modified the language to "turn to", which encompasses individuals from whom respondents sought support, but who were not helpful. In phase two, we found that students were more likely to include the relevant college staff from the two name generators that were specific to student challenges in the context of college. We also learned that students were able to answer these questions more easily when the relevant challenges were made salient by reflecting on the types of challenges they faced in college prior to answering the network questions. Therefore, we finalized two name generator questions, one for who they turned to for academic challenges and another for who they turned to for non-academic challenges. Before answering each of those name generator questions, they answered a priming question about the challenges they faced that year during college. Respondents also provided feedback on name interpreters, recommending additional categories for types of individuals from whom they sought support. We finalized the name generator and interpreter questions after a final assessment in phase three. Conclusions: In his book on using the GSS for social network analysis, Small (2017) found that using the GSS to elicit graduate students' core discussion networks often did not elicit the people that they spoke to recently about school or personal challenges. Our experience modifying the GSS for first-generation college students builds on Small's feedback on the GSS offering specific modifications to this commonly used instrument to visualize support networks in a higher education context. Moreover, the student feedback that we share can inform other studies exploring academic and non-academic support systems, such as demonstrating the importance of priming students, especially in an increasingly remote educational environment, to think about their college student identity before answering questions about the college context.

This report in brief shares findings from the RAND Corporation's five-year study of The Wallace Foundation's University Principal Preparation Initiative (UPPI). The job of the school principal has become much more complex and demanding over the past several decades. Many university-based principal preparation programs (PPPs)--which prepare the majority of school principals--have struggled with how to make the fundamental changes needed to prepare principals for today's schools. To test a path forward, The Wallace Foundation provided grants to seven universities and their partners to redesign their PPPs in line with research-supported practices. Key findings, which suggest that significant change around research-based practices is possible, are presented in this report. This report illustrates that it is feasible for universities -- in partnership with high-need districts, state agencies, and with the support of mentor programs that have engaged in successful redesign -- to improve principal preparation programs to reflect the best available evidence. [For the full report, see ED620465.]

The job of the school principal has become much more complex and demanding over the past several decades. Many university-based principal preparation programs -- which prepare the majority of school principals -- have struggled with how to make the fundamental changes needed to prepare principals for today's schools. This report shares lessons learned from redesigning university principal preparation programs. From 2016 to 2021, seven university principal preparation programs, with their district and state partners, fundamentally reshaped their principal preparation programs under The Wallace Foundation's University Principal Preparation Initiative (UPPI). This report describes the methods of and findings from the RAND Corporation's five-year study of UPPI. It is primarily intended as a secondary resource for readers who would like more detail about the study's findings and methods. Key findings are highlighted in the report in brief and three upcoming targeted reports: one for principal preparation programs, one for school districts, and one for state education organizations. [For the report in brief, see ED620468. For Volume 2, see ED609667.]

With funding from the Bill and Melinda Gates Foundation, CRESST conducted a multi-year evaluation of a major school reform project at Alain Leroy Locke High School, historically one of California's lowest performing secondary schools. Beginning in 2007, Locke High School transitioned into a set of smaller, Green Dot Charter High Schools, subsequently referred to as Green Dot Locke (GDL) in this supplemental report. This report extended the previous report, which tracked the first and second cohorts of 9th-graders who entered GDL in fall 2007 and 2008 respectively thru the 2010-11 school year, by bringing the second cohort of students to graduation. The CRESST evaluation, employing a rigorous quasi-experimental design with propensity score matching, found statistically significant, positive effects for the GDL transformation including improved achievement, school persistence, graduation, and completion of college preparatory courses. Classification of key college-prep courses based on Green Dot and LAUSD course titles is appended. (Contains 7 tables, 8 figures and 10 footnotes.)

With funding from the Bill and Melinda Gates Foundation, CRESST researchers conducted a multi-year evaluation of a major school reform project at Alain Leroy Locke High School, historically one of California's lowest performing secondary schools. We found the one-year teacher retention rate at GDL in 2010-2011 was 79%; Green Dot Locke teachers, overall, were more highly educated than their LAUSD counterpart (including their peers at neighboring high schools) in 2010-2011; and GDL teachers were more likely to be non-White and teacher staff were more racially diverse than LAUSD teachers. Comparing the teachers who stayed (i.e., "stayers") and those who left ("movers") as of 2010-2011 school year, we did not detect any significant differences in age, gender, years of teaching at GDPS, student growth percentile scores or student course pass rates between movers and stayers. We did find a difference in the percentage of teachers meeting or exceeding the evaluation standards, in favor of stayers, and White and English teachers were more likely to be movers. Exploring the relationship between available multiple measures of teacher quality, we found small correlations, between three evaluation scores and student growth percentile (SGP) scores. We also found small to medium positive relationships between teacher evaluation scores with Tripod measures, and the majority of these were statistically significant. While these correlations are not surprising, they are also not robust. Additionally, there was considerable variation across academies on numerous measures (e.g., teacher survey results, interview accounts). While the purpose of this report was not to investigate or compare professional capacity across academies, this appears to be an important part of the Green Dot Locke story to be examined more closely in the future. Appended are: (1) Teacher Interview Protocol; (2) Principal Interview Protocol; (3) 2010-2011 End Of Year Survey Results; and (4) Interview Comments on Green Dot Core Tenets. (Contains 22 tables, 7 figures and 33 footnotes.)

This report describes findings from an efficacy study examining the effects of professional development focused on integrating academic literacy instruction with content area coursework on teacher instructional practices and student achievement in reading and high school biology and US history. The research builds on a well-tested approach to literacy instruction, Reading Apprenticeship (RA), which integrates metacognitive inquiry into ongoing content area instruction to make explicit the tacit reasoning processes, strategies, and discourse rules that shape successful readers' and writers' work in the discipline. The central dynamic of the instructional model is routine metacognitive conversation; that is, talking about the reasoning and problem solving processes that accompany reading as students carry out learning tasks in the science curriculum. The study targeted schools serving high numbers of students from populations underrepresented in higher education. The multiple measures of teacher implementation give us a robust corroboration of teacher level outcomes. Teachers in the experimental group demonstrated increased support for history and biology literacy learning, increased use of metacognitive inquiry routines, increased reading comprehension instruction, and increased use of collaborative learning structures. In short, they were more able to integrate science and science literacy learning in classroom instruction. The results of the study thus present a positive picture with regards to the effectiveness of the Reading Apprenticeship framework for integrating academic literacy content with biology and history coursework and instructional practices.

This study investigates the effects of integrating literacy instruction with biology coursework on high school student learning. It would investigate the relationship between the growth in reading achievement and biology learning. It will target schools serving especially large populations of African American and Latino students. The project will include use a randomized experimental design with multiple measures of student learning, classroom observations, surveys, and interviews for validating quantitative results. The researchers have consulted research studies of reading practices. The project would create a professional development reading apprenticeship program to prepare teachers. Teachers will also be provided with funds to supplement their textbooks with science magazines, trade books, and fiction related to biology topics in the state curriculum frameworks. The study will occur in cities of California. Teachers will be assigned to immediate or delayed intervention randomly assigned to treatment and control groups.

This study sought to explore factors that affect the accessibility of reading comprehension assessments for students with disabilities. The study consisted of testing students using reading comprehension passages that were broken down into shorter "segments" or "chunks." The results of the segmenting study indicated that: (a) segmenting did not affect reading performance of students without disabilities; suggesting that it does not compromise the validity of reading assessment; (b) segmenting did not affect reading performance of students with disabilities; (c) the segmented version had a higher reliability for students with disabilities without affecting the reliability for students without disabilities; and (d) no trends were observed with student motivation, general emotions and moods with respect to the segmented assessment. The study also introduced the idea of incorporating some commonly used accommodations for students with disabilities, such as "test breaks," into the assessment. Limitations of the study included a disability sample with mostly students with specific learning disabilities and a high number of ELL students, as well as a reading assessment that only tested for reading comprehension and not other components of reading. More research using the methods in this study with different subjects can potentially shed additional light on accessibility issues in reading comprehension tests. (Contains 15 tables and 2 figures.)

This article articulates a framework for examining the quality of formative assessment practice and provides empirical evidence in support of one of its components. Based on a study of middle school science, the study examines the accuracy of teachers' judgments of students' understanding and the relationship of such accuracy to middle school students' learning. Analyses within and between teachers show a consistent, positive relationship between teacher accuracy and student learning. Study results lend support for the power of assessment in improving student learning and also suggest some potential challenges in assuring quality formative assessment practice. (Contains 6 tables, 3 figures, and 1 footnote.)

Based on the results of a generalizability study (G study) of measures of teacher knowledge for teaching mathematics developed at The National Center for Research, on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles, this report provides evidence that teachers are better at drawing reasonable inferences about student levels of understanding from assessment information than they are in deciding the next instructional steps. We discuss the implications of the results for effective formative assessment and end with considerations of how teachers can be supported to know what to teach next. (Contains 2 tables, 2 figures, and 1 footnote.)

Current educational policy highlights the important role that assessment can play in improving education. State standards and the assessments that are aligned with them establish targets for learning and promote school accountability for helping all students succeed; at the same time, feedback from assessment results is expected to provide districts, schools, and teachers with important information for guiding instructional planning and decision making. Yet even as No Child Left Behind (NCLB) and its requirements for adequate yearly progress put unprecedented emphasis on state tests, educators have discovered that annual state tests are too little and too late to guide teaching and learning. Recognizing the need for more frequent assessments to support student learning, many districts and schools have turned to benchmark testing--periodic assessments through which districts can monitor students' progress, and schools and teachers can refine curriculum and teaching--to help students succeed. We report in this document a collaborative effort of teachers, district administrators, professional developers, and assessment researchers to develop benchmark assessments for elementary school science. In the sections which follow we provide the rationale for our work and its research question, describe our collaborative assessment development process and its results, and present conclusions. (Contains 9 tables.)

In this report we explore the role of reviewer agreement in judgments about alignment between tests and standards. Specifically, we consider approaches to describing alignment that incorporate reviewer agreement information in different ways. The essential questions were whether and how taking into account reviewer agreement changes the picture of alignment between tests and standards. This study showed a wide range of reviewer agreement during the process of aligning standards and assessments, with substantial reviewer disagreement about important elements such as correspondence between objectives and items on the assessments. Taking this reviewer disagreement into account changed conclusions about alignment, not only showing weaker alignment than previously demonstrated, but also changing the profiles of alignment about, for example, relative coverage of specific standards. The results of this study point to the need for greater clarity in objectives and standards, more extensive reviewer training during the alignment process, and possibly also inspection of items to uncover characteristics that may lead to uncertainty among reviewers. (Contains 7 tables.)

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination (GSE) in High School Mathematics and the University of California (UC) "Statement on Competencies in Mathematics." UC faculty and high school mathematics teachers (n = 20) rated the mathematics items of the GSE relative to the expectations identified in the UC competency statement, identifying item features related to content and dimensionality. Raters assigned values for a primary topic, a secondary topic, item/topic centrality, and depth of knowledge. Agreement within these criteria was the basis of the assessment of alignment. Results showed considerable variability in judgments across raters and different pictures of alignment depending on the particular subset of raters providing the ratings. Results also varied by rater type and the method of determining rater agreement. Appended are: (1) Specific Topics Considered Essential for All Entering College Freshmen in the UC Statement on Competencies in Mathematics; (2) Detailed Description of Depth-of-Knowledge Levels; and (3) Technical Notes on the Generalizability Studies.

The progress of students and schools in California is examined by reviewing the state's recent test history and the state's progress in creating a sound, standards-based assessment system. The story about California is mixed. Overall performance on the Stanford Achievement Test, ninth edition (SAT-9) shows that the state average, overall grades and all subject matters, is below the national average. However, when allowance is made for the state policy that requires all students who have been in school 1 year to take the test regardless of their English proficiency, California students are around the national average. When achievement is examined through standards-based measures, of which the National Assessment of Educational Progress is an example, California's performance looks poor, especially when one considers the performance of children in poverty. What will be important to consider is whether performance rises as instruction becomes more relevant to the standards the assessments are measuring. The report offers recommendations for the best development of the California assessment system. Two appendixes contain the figures and tables. (Contains 67 figures, 6 tables, and 22 references.) (SLD)

To understand how assessment, particularly alternative assessment, and practice interactions may differ among schools with different implementation contexts and different school cultures, this project is studying over time a set of schools representing different state and local policy contexts and including different student populations. Research questions are: (1) whether the implementation strategies facilitate or inhibit teacher use of information from alternative assessments; (2) how alternative assessments interact with teachers' perceptions of achievement and student ability; (3) how alternative assessments interact with teachers' instructional decisions; (4) how alternative assessments interact with students' perceptions of performance and abilities; and (5) how each of the above differs in different schools. Longitudinal case-study methods are being used in two phases. In phase 1, four sites in California are being studied; phase 2 may expand to include other schools and states. The project will result in a series of products that should enhance the understanding of policymakers, researchers, and practitioners about ways to implement alternative assessments to improve learning and teaching. One table summarizes the study framework. There is a 35-item list of references. (SLD)

Findings from a study that explored the ways in which principals use information on school quality are presented in this paper. The effects of school level, years of administrative experience, students' socioeconomic status, and school performance are also examined. Methodology involved interviews and follow-up interviews with a total of 38 elementary, middle, and high school principals from 12 California school districts. Findings indicate that the most frequently used information source is school/classroom observation, and that test data are most frequently used to communicate with teachers and parents and to identify areas of instructional strengths and weaknesses. Recommendations are made for improving principal analysis of school data and communication with the public. Appendices contain the interview protocol, comment sheet, and survey data (8 graphs and 5 tables). (LMI)

This study analyzes the ways in which school boards assess and utilize evaluation data and their attitudes toward testing. Interviews with 27 school board members from 10 districts indicate that board members tend to rely on informal working knowledge over formal information, and on district administrators' judgments. Findings suggest that simple changes in data reporting, such as inclusion of executive summaries and clear graphic displays, will improve clarity and usefulness. However, members' lack of familiarity with basic research principles, reliance on experiential-based knowledge, and preoccupation with administrative concerns suggests that improving data quality and usefulness will have a marginal impact on educational quality. A conclusion is that the technical problems of providing reasonable, useable data are tractable, and that feasible solutions can be found; however, if they are to have an impact on educational quality, larger sociopolitical issues must be addressed. Appendices contain the interview guide, three graphs, and seven tables. (4 references) (LMI)

A study of the development of bilingual/bicultural instructional materials (mostly for Spanish-speaking populations) and the institutions involved in their use, preparation, and distribution is described. A review of representative materials and on-site visits to 75 institutions, programs, and publishers in New York, California, Florida, Puerto Rico, Texas, and the District of Columbia is presented, and a list of all 75 sites is included. Observations indicated that the best programs employed individuals with a high degree of dedication to the field. Personnel were highly critical of some instructional materials which were direct translations from other languages and cited the need to supplement state curricula with representative materials from the cultures represented in their classrooms. Two disturbing factors were observed: administrative decisions not to mass produce or disseminate developed materials; and lack of publicity or in some cases interagency rivalries which frequently prevent practitioners from hearing about such materials. A review of state and federal mandates and legislation regarding bilingual education mirrors the confusion sometimes seen in the classroom and in the development and use of materials, with some states having clear mandates and others not truly giving consideration to this field. Although there are some excellent instructional materials available, personnel interested in obtaining them must search them out. (CB)