Search

This study sought to investigate whether there were performance differences between the children who engaged with the Ignite by Hatch™ educational gaming system using the English- or Spanish-language versions of the games. Differential item functioning methods (DIF) were employed to investigate these differences. Specifically, DIF analyses can address these issues by providing evidence that the response process does not advantage any particular subgroup of children. DIF analyses can separate performance differences on specific games from individual differences in child ability. Focal and reference categories of children are identified to investigate DIF. We examined the focal group (Spanish) to gather evidence to demonstrate that the games are fair to the members of that particular subgroup. The reference group is typically the majority group (English), and the presumption is that the games are generally fair for the children in that group.

To test hypotheses in cross-cultural studies, it is necessary to investigate whether results might be affected systematically by language or cultural effects. In this paper the results of differential item functioning (DIF) analyses for the Perceived Stress Scale (PSS), the Preventive Resources Inventory (PRI), and the Brief COPE are presented. Data from N = 860 German and Singapore social work students were analyzed using the Rasch Partial Credit Model by comparing item difficulties. Large DIF was found in 3 of the 10 items from the PSS, in 15 of 82 items from the PRI and in 5 out of 28 items from the Brief COPE. Implications for the use of the measures are discussed.

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including rater-specific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet's AC1, and Gwet's AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 65 raters than Kappa. However, AC-2 was undetermined for 57 of the 65 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of Rater-Mediated Agreement to the data conditions that are problematic for Kappa.

The North Carolina Teacher Evaluation Process (NC TEP) serves over 100,000 teachers per year throughout the state (U.S. Department of Education, 2021). This process includes formative observations, summative assessments, and formal mentoring to ensure high quality educators. In an attempt to ensure a high-quality evaluation process, hubs at both the University of North Carolina at Charlotte and East Carolina University were developed in 2007 to serve early childhood educators that have a Birth to Kindergarten (B-K) license (de Kort-Young et al., 2016). Both universities house an Early Educator Support Office to provide mentorship and support. The purpose of this report is to provide a description of how the Early Educator Support Office has systematically planned and orchestrated evaluation training to provide valid and reliable teacher evaluations for B-K licensure educators. The report provides an explanation of the educator evaluation process in the state of North Carolina as well as background literature to comprehensively describe the need for a high-quality evaluation process. The report also provides a brief description of the NCTEP rubric and the phased Quality Improvement model. Finally, an overview of the next steps for the Early Educator Support Office research at the end of this report. Finally, an overview of the next steps for the Early Educator Support Office research which include gathering data for the next groups of evaluators and conducting a Many Facets Rasch Model with all evaluators that took part in the interrater reliability certification process with the aim to determine specific supports for evaluators and educators.

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including rater-specific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an inter-rater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet's AC1, and Gwet's AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 45 raters than Kappa. However, AC-2 was undetermined for 39 of the 45 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of Rater-Mediated Agreement to the data conditions that are problematic for Kappa. [This paper was presented to the virtual Annual Meeting of the National Council on Measurement in Education, June, 2021.]

This study gathered information from practitioners on their perspectives of experiences implementing the North Carolina Kindergarten Entry Assessment (NC KEA). Initially, practitioners were invited to complete an online survey asking them to rate or provide information about several aspects related to implementing the assessment process. In total, 385 practitioners completed the online survey. The survey was made up of open-ended questions and four-point Likert items (strongly disagree to strongly agree). At the end of the survey respondents could notify the research team of their interest in participating in a follow-up telephone interview. Out of 63 practitioners indicating interest in completing interviews, a final group of 15 interviews were conducted to provide a more in depth understanding of NC KEA experiences. The telephone interviews were conducted during late June to early July 2020. Teachers mentioned the importance of, and context for, using data from the NC KEA in Professional Learning Communities (PLC), with lead teachers, and on grade level teams. Several kindergarten teachers interviewed in the study indicated experiences with time constraints and difficulty managing several assessments simultaneously at the start of the school year. While some interview participants mentioned their appreciation for the NC KEA's whole-child approach to assessment, many teachers expressed concerns over the lack of alignment with other assessments or the redundancy of some kindergarten assessments. It is important to note that many of the responses from interview participants signaled misunderstandings or misuses of the NC KEA as a formative assessment process.

Differences in personal characteristics, professional qualifications, and contextual factors for congruent and incongruent Black, Hispanic, and White teachers were investigated using data from the National Teacher and Principal Survey (n[subscript teachers]=25,420; n[subscript schools]=5,500). Racial/ethnic congruence was defined as the match between the predominant race/ethnicity of students within a school and the teacher's own race/ethnicity. White incongruent elementary and secondary teachers were less qualified than their congruent counterparts. There were few differences in teacher qualifications between congruent and incongruent Black and Hispanic teachers. This study demonstrates that predominantly White schools employ more qualified teachers given the substantial differences between White congruent and incongruent teachers and thereby illustrates yet another aspect of systemic inequities in U.S. schools.

This report contains information on the background of and research on the North Carolina Kindergarten Entry Assessments (NC KEA). Evidence shows that implementing formative assessment is difficult for teachers and schools. Given that North Carolina will be adopting a new KEA measure for the fall of 2020, this report summarizes previous research on the NC KEA, as well as formative assessments throughout the United States, in order to provide a set of data-based recommendations to help support implementation during this transition. The recommendations provided in this report can be divided into three broad areas: (1) initial and ongoing professional development; (2) supporting implementation practices; and (3) administrative best-practices.

This report outlines the results of an evaluation of the Freedom School Partners' Children Defense Fund Freedom Schools© program in Charlotte, North Carolina, that took place during the summer of 2019. The evaluation was a collaboration between the Center for Adolescent Literacies (CAL) and the Center for Educational Measurement and Evaluation (CEME). Both centers are located on the University of North Carolina at Charlotte's campus within the College of Education. Each Center has extensive records of collaboration with community groups as well as state and local educational agencies. The CEME staff conducted the data collection process and conducted the data analysis. The focus of this evaluation is on the experiences of the Scholars and Servant Leader Interns (SLIs) by collection data via surveys developed by both CEME and CAL. Findings reveal that Freedom School Partners is a positive program that is making a difference for children in Charlotte, North Carolina, and with several small improvements to the program, it could be even more impactful. [For the 2017 report, see ED624228.]

This report contains reliability and validity evidence for the North Carolina Kindergarten Entry Assessment system (KEA) using a statewide sample of kindergarten children. The KEA was designed as a formative, developmental, authentic, and criterion referenced classroom resource for teachers. As a formative assessment, the KEA focuses on the learning process and is used to support learning while learning is taking place. It is a tool to support teachers and students, and helps provides valuable feedback to inform and adjust both teaching and learning. This study focused on the following research questions in an effort to begin to understand the rating scale category effectiveness of the KEA progressions as used by North Carolina kindergarten teachers: (1) What are the characteristics of the distributions of placements on each of the progressions? (2) Do the mean total scores of the children placed in each category increase monotonically along the rating scale for each of the progressions? (3) Do the thresholds between rating scale categories increase monotonically along the rating scale for each of the progressions? and (4) Do the category probability plots indicate distinct probability distributions for each rating scale point for each of the progressions? The results of the analyses related to dimensionality and reliability are all very strong and reflect very positively on the use of a total score for psychometric research purposes such as those outlined in this report. The results related to item or progression difficulty estimates were generally positive, though they suggest a need for a greater range of progression difficulties. The results of the examination of rating scale category effectiveness were mixed.

This report outlines the results of an evaluation of the Freedom Schools project as it was implemented in Charlotte, North Carolina, during the summer of 2017. The evaluation study was a collaborative project between the Center for Adolescent Literacies (CFAL) and the Center for Educational Measurement and Evaluation (CEME). Both centers are housed within the College of Education at the University of North Carolina (UNC) Charlotte and have extensive records of collaboration with community groups as well as state and local educational agencies. The survey items that were used in the data collection process were initially developed and piloted during the 2016 Freedom Schools program year. This process was accomplished by a collaborative effort of CFAL, CEME, and Freedom Schools. An initial set of draft survey items were developed through an iterative process by CFAL and CEME staff. The Freedom Schools program goals and objectives, and the goals of the program evaluation effort were carefully considered as each draft was reviewed and revised. Freedom schools staff were given an opportunity to review the draft survey items and were interviewed to gather their input regarding whether the survey would meet the needs of the program and effectively solicit the feedback of program participants.

The 2016-2017 academic year marks the second year of statewide implementation of the North Carolina K-3 Formative Assessment Process: Kindergarten Entry Assessment (FAPKEA). The assessment, developed and implemented by the Office of Early Learning (OEL) at the North Carolina Department of Public Instruction, is a comprehensive formative assessment process intended to assist teachers in both identifying and supporting the unique learning needs of their young students across all five domains of early childhood development. To support implementation fidelity and sustainability, OEL adopted a multi-tiered implementation team structure from the state level through the building level, an approach backed by implementation science research from the National Implementation Research Network (NIRN). These teams worked independently and collaboratively to identify both facilitators and barriers of implementation, and they developed action plans or resources to support efforts at every level. Since the assessment's inception in 2014, OEL has put significant energy into developing the teams for this implementation approach; however, little is known about practitioner's perceptions of the effectiveness of their teams in supporting implementation efforts. To better understand practitioner perceptions of this teaming structure and identify any resources that may still be needed to support both team functions and overall implementation, a partnership of researchers from the UNC Charlotte Center for Educational Measurement and Evaluation (CEME) and Child Trends, a non-profit organization focused on improving the lives and prospects of youth through social, economic, health, and education research, conducted case studies in four North Carolina school districts. These case studies included observations of implementation team meetings and in-depth interviews with team members at every level, from the state through the building. This report summarizes their findings.

The North Carolina K-3 Formative Assessment Process: Kindergarten Entry Assessment (NC K-3 FAP: KEA) is the initial step in the development of a comprehensive formative assessment process for young children from kindergarten entry through the end of third grade. The purpose of this qualitative study was to gather teacher and administrator feedback that could inform potential changes to the following aspects of KEA implementation: (1) professional development for the assessment process; (2) the content of the current KEA construct progressions; (3) teachers' perceptions of the overall utility of the overall KEA process; and (4) the usability and features of the NC K-3 FAP electronic platform. Researchers conducted case studies at six elementary schools selected for their demographic representativeness of the larger North Carolina kindergarten population. The case studies included classroom observations, as well as semi-structured interviews with teachers, administrators, and instructional coaches involved with implementation in each school. At the end of the 60-day KEA period, researchers conducted an electronic survey which was open to all North Carolina kindergarten teachers. Findings indicate that practitioner informed changes are occurring as part of the continual development of the NC K-3 FAP: KEA. Whether that knowledge affects overall teacher perceptions and buy-in for the assessment process is a question for future research

Based on the recommendations of the K-3 Assessment Think Tank, an assessment design team consisting of current and former educators and early childhood education scholars developed a kindergarten entry formative assessment (KEA) which was piloted in 193 classrooms during the first 60 instructional days of the 2014-2015 academic year. North Carolina (NC) intended the pilot to be a first step toward scaling-up to a full statewide K-3 formative assessment process. Over the course of the NC KEA pilot, the authors conducted case studies of eight pilot schools selected for their demographic representativeness of the total NC kindergarten student population. These case studies involved classroom observations and semi-structured interviews with pilot teachers, administrators, and instructional coaches. Following the case studies, an electronic survey that was open to all pilot participants was conducted. The purpose of this qualitative study was to gather teacher and administrator perspectives of the NC KEA and determine whether: (1) changes were necessary to the professional development provided for the assessment; (2) the content of the learning progressions were developmentally appropriate; (3) the documentation process worked within teachers' routine classroom instruction; (4) teachers found the assessment's data useful in driving their instructional decisions; (5) the assessment aligned with the curriculum and assessment practices currently in place within their district and school; and (6) there were characteristic patterns between schools and/or classrooms where the assessment was being implemented with fidelity and those where it was not.

The Preventive Coping Resources Inventory (PRI) was developed to assess specific coping resources most useful for stress prevention and also applicable to education settings. Undergraduates (N=501) at a large, southwestern university completed the inventory and other measures of adjustment and coping. Exploratory factor analysis revealed five factors that underlay the inventory were perceived control, maintaining perspective, social resourcefulness, humor, and organization. An additional cross-over factor, self-acceptance, was also retained. Evidence for the convergent and discriminant validity of the PRI was provided by theoretically consistent relationships with related constructs such as self-efficacy, general coping resources, and coping strategies. Evidence for its criterion-related validity was supported by hierarchical regression analyses in which scales from the PRI predicted perceived stress levels after controlling for the incidence of negative life events. Suggestions for future research and applications to educational settings are discussed. (Contains 47 references and 6 tables.) (Author/JDM)

This paper provides evidence for the reliability and validity of the Preventive Coping Resources Inventory (PCRI) instrument designed to measure coping resources useful for prevention based on previous research. It specifically looks at the construct validity of the PCRI; the convergent and discriminate validity of the PCRI with related constructs; and the criterion-related validity of the PCRI as a predictor of perceived stress levels and symptomatology. Data was collected from 252 undergraduate students from a large Southwestern university. Analysis conducted supported the construct validity of three of the six hypothesized preventive resources: perceived control, self-confidence, and social comfort. Further analysis is needed to determine whether the three scales (self-acceptance, organization, humor) that were not supported in the factor analysis should be included as dimensions of preventive coping resources. In general, the factor and scale scores of the PCRI correlated with higher coping resources than with other closely related constructs. Overall, findings suggest that the PCRI may provide meaningful factors useful for preventive coping such as perceived control, self-confidence, and social comfort. Further exploration is needed to determine if the scales of the PCTI are distinct from other coping instruments in measuring resources that are most useful for prevention. (Contains 53 references.) (JDM)

Family sources of stress and conflict are critical variables in the well-being of adolescents. This paper assesses the relationship of coping resources to negative emotions produced by parental conflict after controlling for social desirability; age; financial resources; and measures of parental attachment and family functioning. Undergraduate students (n=304) in a large southwestern university were given four instruments: Inventory of Parental and Peer Attachment (IPPA); Family Adaptability and Cohesion Scale II (FACES II); Coping Resources Inventory for Stress (CRIS); and Parental Conflict Emotions. Four separate models were created for: (1) male participants describing maternal conflict; (2) male participants describing parental conflict; (3) female participants describing maternal conflict; and (4) female participants describing parental conflict. The results provide support for the literature that a person's family background and coping resources are related to emotional functioning in the context of family relationships. The emergence of social desirability as a statistically significant predictor of variance at step one of each model was noted. The pattern of results with respect to female participants included more predictors emerging as statistically significant. Both family functioning and psychological coping resources were suggested as areas in which counselors could intervene. (Contains 6 tables and 32 references.) (Author/JDM)

Family sources of stress and conflict are important variables in the well-being of children, adolescents, and young adults. In this study, current models of stress are contrasted with earlier versions, definitions are reviewed, and ambiguities in conceptualization are brought out. Current research on affective information processing is thought to hold considerable promise for understanding the links between cognitions and emotions in families experiencing conflict. A comprehensive model is suggested that attempts to explain the importance of constructs such as family functioning and coping for appraisals of parental conflict and subsequent emotional response. The model tested includes several constructs thought to measure family conflict experiences. Specifically, this study explores the potential role of family functioning, coping resources, and attitudes about mood in influencing cognitive appraisals of family conflict and subsequent emotional responses among college students (N=609). A rationale for the inclusion of these constructs is provided, and additional measures are described. Results are reported and discussed. Further modeling strategies are described, and the resulting model is discussed. (Contains 2 figures, 3 tables, and 105 references.) (EMK)

A special case of the homogeneity of effect size test, as applied to pairwise comparisons of standardized mean differences, was evaluated. Procedures for comparing pairs of pretest to posttest effect sizes, as well as pairs of treatment versus control group effect sizes, were examined. Monte Carlo simulation was used to generate Type I error rates and power values for tests of the differences in independent effect sizes based on both the "g" and "d" methods. Type I error rate was evaluated by crossing 6 sample size conditions (5, 10, 20, 30, 50, and 100) by 5 population effect size conditions (0.00, 0.25, 0.50, 0.75, and 1.00). Power was evaluated by crossing the 6 sample size conditions by 4 conditions representing the magnitude of the difference between treatment and control conditions (0.25, 0.50, 0.75, and 1.00). The "d" based statistic yielded Type I error rates closer to the nominal level than did the "g" based statistic while yielding a slightly more conservative method for testing the difference between two effect size measures. Examples are provided that illustrate the use of these procedures as post hoc comparison techniques following factorial analysis of variance designs. (Contains 6 tables and 30 references.) (Author/SLD)