Search

Mental health organizations that serve youth are under pressure to adopt measurement-based care (MBC), defined as the continuous collection of client-report data used to support clinical decision making as part of standard care. However, few frameworks exist to help leadership ascertain how to select an MBC approach for a clinical setting. This paper seeks to define how an MBC approach can display clinical utility to provide such a framework. Broadly, we define clinical utility as evidence that an MBC approach assists stakeholders in fulfilling clinical goals related to care quality (i.e., improve client clinician alliance and clinical outcomes) at the client (i.e., youth and caregiver), clinician, supervisor, and administrator levels. More specifically, our definition of clinical utility is divided into two categories relevant to the usability and usefulness of an MBC approach for a specific setting: (a) implementability (i.e., evidence indicating ease of use in a clinical setting) and (b) usefulness in aiding clinical activities (i.e., evidence indicating the potential to improve communication and make clinical activities related to care quality easier or more effective). These categories provide valuable information about how easy an MBC approach is to use and the potential benefits that the MBC data will confer. To detail how we arrived at this definition, we review prior definitions of clinical utility, discuss how previous definitions inform our definition of clinical utility for MBC, and provide examples of how the concept of clinical utility can be applied to MBC. We finish with a discussion of future research directions.

Home-school partnerships between teachers and caregivers of students with or at risk of emotional/behavioral disorders (EBD) are critical to support positive student outcomes. Yet, effective home-school partnerships may be particularly challenging to foster for students with or at risk of EBD, and little is known about how teachers and caregivers view these partnerships. We used a directed content analysis qualitative approach to examine the presence or absence of components of a conceptual model, the CARES framework, within home-school partnerships from the perspectives of both teachers and caregivers of students with or at risk of EBD. Semi-structured interviews were conducted with six U.S. teachers and 11 U.S. caregivers who had participated in a study examining the promise of a Tier 2 intervention. Findings highlight themes unique to specific components of the CARES framework and themes common across components, with authentic relationships and effective communication emerging as particularly salient for both caregivers and teachers. Results suggest critical directions for future research and essential lessons for researchers developing interventions to support positive home-school partnerships for caregivers and teachers of students with or at risk of EBD.

Student responsiveness's role in promoting intervention outcomes for students who exhibit problem behavior is understudied. Due to the relational nature of many interventions delivered by teachers that target social, emotional, or behavioral outcomes of students in classrooms, it is essential to assess how responsive students are to teachers' "attempts" to engage them in the intervention, particularly for students with problem behaviors that may impede teachers' attempts to engage these students in intervention effectively. In the current study, we combine samples from four randomized controlled trials to examine the relationship between student outcomes and teacher attempts to deliver BEST in CLASS, a Tier 2 intervention, via student responsiveness. Delivery of BEST in CLASS and student responsiveness were assessed through direct observations and teachers' reported measures. Results suggest that teacher adherence and competence in delivering BEST in CLASS practices was associated with reductions in problem behavior from pretest to post-test via student responsiveness. Limitations of the current study and implications for future research are discussed.

Evidence-based programs (EBPs) delivered in elementary schools show great promise in reducing risk for emotional and behavioral disorders (EBDs). However, efforts to sustain EBPs in school face barriers. Improving EBP sustainment thus represents a priority, but little research exists to inform the development of sustainment strategies. To address this gap, the Sustaining Evidenced-Based Innovations through Multi-level Implementation Constructs (SEISMIC) project will (a) determine if malleable individual, intervention, and organizational factors predict EBP treatment fidelity and modifications during implementation, sustainment, or both; (b) assess the impact of EBP fidelity and modifications on child outcomes during implementation and sustainment; and (c) explore the mechanisms through which individual, intervention, and organizational factors influence sustainment outcomes. This protocol article describes SEISMIC, which builds upon a federally funded randomized clinical trial evaluating BEST in CLASS, a teacher-delivered program for K to Grade 3 children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools. A multi-level, interrupted time series design will be used to examine the relationship between baseline factors, treatment fidelity, modifications, and child outcomes, followed by a mixed-method approach to elucidate the mechanisms that influence sustainment outcomes. Findings will be used to create a strategy to improve EBP sustainment in schools.

An observational coding system was developed to track clinical change in children with autism spectrum disorder (ASD) during psychotherapy. The Pediatric Autism Spectrum Therapy Observation System (PASTOS) consists of 23 items divided into 5 subscales and is used to rate child behaviors in individual psychotherapy sessions. Manual-based cognitive behavioral therapy session transcripts of 22 children diagnosed with ASD (IQ > 70) and a concurrent anxiety disorder (M = 9.41 years, SD = 1.56 years) enrolled in a randomized, controlled trial were coded. Results suggested that the PASTOS exhibited promising interrater reliability, internal consistency, convergent validity at post-treatment, and treatment sensitivity. The PASTOS may be a useful tool for studying process and outcome in psychotherapy research on children with ASD.

Though treatment integrity measurement is important for research intended to promote social and behavioral outcomes of children at risk for emotional and behavioral disorders (EBDs) in early childhood settings, measurement gaps exist in the field. This paper reports on the development and preliminary psychometric assessment of the treatment integrity measure for early childhood settings (TIMECS), an observational measure designed to address existing measurement gaps related to treatment integrity with tier 2 interventions in the early childhood field. To assess the preliminary score reliability (interrater) and validity (construct, discriminant) of the TIMECS, live observations (N = 650) in early childhood classrooms from 54 teachers (92.6% female, 7.4% male; 61.1% White) and 91 children (M age = 4.53 years, SD = 0.44; 45.1% female, 54.9% male; 45.1% Black) at risk for EBDs were scored by 12 coders using the TIMECS and an observational measure designed to assess teacher-child interactions. Teachers also self-reported on the quality of the teacher-child relationship. Interrater reliability (intraclass correlation coefficients, ICC [2,2]) for the quantity (i.e., adherence) item scores had a mean of 0.81 (SD = 0.07; range from 0.68 to 0.95), and the quality (i.e., competence) item scores had a mean of 0.69 (SD = 0.08; range from 0.52 to 0.80). Scores on the TIMECS Quantity and Quality items and scales showed evidence of construct validity, with the magnitude of the correlations suggesting that the quantity and quality items assess distinct components of treatment integrity. A TIMECS quantity scale also showed promise for intervention evaluation research by discriminating between teachers who had and had not been trained in a specific evidence-based intervention targeting social and behavioral skills in early childhood. The findings support the potential of the TIMECS to assess treatment integrity of teacher-delivered practices designed to address child social and behavioral outcomes of children at risk for EBDs in early childhood settings. [For the corresponding grantee submission, see ED618135.]

Improving social, emotional, and behavioral outcomes of students with and at-risk for emotional and behavioral disorders (EBD) remains a challenge for educators, and it has long been noted that teachers do not consistently use effective instructional practices with students with EBD. Identifying evidence-based practices that address the problems experienced by students with EBD is a priority for the field, but there exist implementation challenges in authentic classroom settings. The purpose of this article is to address one implementation barrier by distilling the common practice elements found in evidence-based programs (EBPs) and practices delivered by teachers designed to target the social, emotional, and behavioral problems in young students. We conducted a systematic review of EBPs and early elementary classroom practices that have been evaluated in randomized group designs, quasi-experimental designs, and single-case experimental designs. A total of 103 articles employing 68 group (n = 53 randomized group designs, n = 15 quasi-experimental designs) and 35 single-case designs were identified, and an iterative process was used to identify common practice elements. Twenty-five practice elements were identified and submitted to review by experts. After expert review, 24 practice elements remained. Implications for practice and training as well as future research are discussed.

Students' early school experiences have a significant and long-term effect on key academic and social/behavioral outcomes. Evidence-based programs that both increase the quantity and quality of positive teacher-student interactions and decrease the frequency of negative teacher-student interactions may be critical for changing negative patterns and improving students' early learning environments. The purpose of this article is to describe the iterative development of one such program, BEST in CLASS--Elementary, which has been adapted from an evidence-based program designed to reduce risk for emotional/behavioral disorders by improving teachers' interactions and relationships with preschool-age children with chronic problem behavior. Following a brief description of the parent program, 4 overlapping phases of intervention adaptation and development are described. Future directions for the intervention and applications of this iterative process tointervention developmentwork and research are discussed. [This paper was published in "Elementary School Journal" (EJ1217810).]

Improving social, emotional, and behavioral outcomes of students with and at-risk for emotional and behavioral disorders (EBD) remains a challenge for educators, and it has long been noted that teachers do not consistently use effective instructional practices with students with EBD. Identifying evidence-based practices that address the problems experienced by students with EBD is a priority for the field, but there exist implementation challenges in authentic classroom settings. The purpose of this article is to address one implementation barrier by distilling the common practice elements found in evidence-based programs (EBPs) and practices delivered by teachers designed to target the social, emotional, and behavioral problems in young students. We conducted a systematic review of EBPs and early elementary classroom practices that have been evaluated in randomized group designs, quasi-experimental designs, and single-case experimental designs. A total of 103 articles employing 68 group (n = 53 randomized group designs, n = 15 quasi-experimental designs) and 35 single-case designs were identified, and an iterative process was used to identify common practice elements. Twenty-five practice elements were identified and submitted to review by experts. After expert review, 24 practice elements remained. Implications for practice and training as well as future research are discussed. [The article was published in "Journal of Emotional and Behavioral Disorders" (EJ1215242).]

While the measurement of treatment integrity is important to determine how much, and how well, interventions are delivered in schools, the science of treatment integrity is not well developed in education research. The purpose of this paper is to describe a program of research that has developed treatment integrity measures over the past 10 years to assess teacher delivery of an indicated program targeting reductions in problem behavior in early childhood and elementary school classrooms. Specifically, this paper will highlight the importance of active use of conceptual models to guide treatment integrity measure development, multidimensional assessment of treatment integrity and training procedures for observers, using several studies to illustrate the evolution and refinement of our measurement approach. Recommendations for researchers developing and evaluating interventions in schools are provided, as are recommendations to help the field move toward a more rigorous science of treatment integrity.

The purpose of this special series is to highlight how the measurement of integrity, both treatment integrity and integrity for implementation strategies, can support intervention development, evaluation, and implementation research. The seven papers and one commentary in this special series illustrate how diverse measurement approaches are used for integrity across various types of intervention research. This introductory paper provides a general overview of the themes that cut across the papers in this special series. We define key terms and discuss how the measurement of treatment integrity (i.e., extent to which the core practices of a protocol are delivered) and integrity to implementation strategies (i.e., extent to which the core practices of an implementation strategy are delivered) changes as an intervention progresses along the treatment development pipeline spanning from basic research to implementation and dissemination research. Then we discuss the ways that the measurement of integrity can change across the different steps in the pipeline to meet the goals associated with different stops along the pipeline. We finish the paper with a description of the papers in the special series and concluding thoughts.

Numerous evidence-based programs (EBPs) exist for delivery in schools to promote youth mental health outcomes. However, school systems often lack the internal infrastructure to support the effective implementation and sustainment of EBPs when external supports are withdrawn, resulting in notable attenuation in the benefits in youth clinical outcomes that are associated with EBPs. This paper illustrates how to leverage concepts from improvement science and implementation science to develop learning school systems dedicated to enhancing the infrastructure capacity of a school to advance the implementation and sustainment of EBPs. In particular, we discuss how treatment integrity (extent to which an EBP is delivered as designed) and youth mental health outcome data are: (1) collected; (2) analyzed and interpreted; and (3) fed back into the school system to increase organizational supports and promote school practitioners' behavior change to produce improvements in youth mental health outcomes. We also discuss psychological safety among the people within a school system as a key characteristic of a learning school system. We then present a theory-informed approach to learning school systems to demonstrate how data generated by a learning school system can lead to precise and effective plans that continuously improve implementation and result in the eventual sustainment of EBPs. We conclude with a brief research agenda with concrete steps toward realizing the potential of learning school systems to support the implementation and sustainment of EBPs for mental health problems.

To examine treatment fidelity in a randomized controlled trial of indicated school-based cognitive-behavioral therapy (CBT) delivered in groups to youth with anxiety. We investigated whether adherence and competence (a) differed across brief and standard-length CBT, and (b) if adherence and competence predicted change in anxiety symptoms and impairment. Method: Sessions were observationally coded with the "Competence and Adherence Scale for CBT for Anxiety Disorders in Youth." Coders (N = 7) rated 104 sessions from 52 groups delivered by 32 facilitators (M age = 43.2 years, SD = 8.1) to 295 youth (M age = 14.0 years, SD = 0.8). Outcomes were youth- and parent-reported anxiety symptoms and impairment at post-intervention and 1-year follow-up. Linear mixed effect models were used to analyze whether fidelity predicted clinical outcomes. Results: Levels of adherence and competence were adequate in both programs, but higher in brief compared to standard-length CBT p < 0.001 and p = 0.010, respectively). Neither adherence nor competence predicted clinical outcomes at any timepoints. Conclusion: Higher levels of adherence and competence in brief CBT suggest that it may be easier for novice CBT providers to achieve fidelity in simplified and less flexible interventions. Contrary to expectation, adherence and competence did not predict clinical outcomes.

Teacher-reported measures of treatment integrity (the extent to which prescribed practices are delivered as intended by teachers) have the potential to support efforts to evaluate and implement evidence-based interventions in early childhood settings. However, self-report treatment integrity measures have shown poor correspondence with observer-report treatment integrity measures, raising questions about score validity. This paper reports on the development and initial evaluation of the score reliability and validity of the Treatment Integrity Measure for Early Childhood Settings Teacher Report (TIMECS-TR), which is designed to address limitations of previous self-report treatment integrity measures that may have contributed to low correspondence with observer-rated measures. The TIMECS-TR includes 24 items designed to represent practices found in evidence-based interventions delivered in early childhood settings that target child social, emotional, and behavioral skills, rather than adherence to practices found in a specific evidence-based intervention. Fifty-four teachers (92.6% female, 7.4% male; 61.1% White) completed the TIMECS-TR weekly for a total of 618 times (M = 6.79 per child; SD = 2.16; range 2 to 11) about the practices they delivered with 91 children (45.1% female, 54.9% male; M = 4.53 years old; SD = 45.1% Black) who were at risk for emotional and behavioral disorders. Analyses indicated that the TIMECS-TR items evidenced mild to moderate test-retest score reliability over one week. However, analyses did not support the convergent score validity of the TIMECS-TR items or scale with observational ratings of the same practices. Teachers reported higher levels of practice delivery on the TIMECS-TR items relative to observer report. Overall, our findings raise concerns about the accuracy of teacher-report adherence measures. Lessons from this research can be used to identify possible reasons for the low correspondence between teacher- and observer-report treatment integrity measures so that future research can strive to dependably capture teacher delivery of the practices found in evidence-based interventions.

The purpose of this special series is to highlight how the measurement of integrity, both treatment integrity and integrity for implementation strategies, can support intervention development, evaluation, and implementation research. The seven papers and one commentary in this special series illustrate how diverse measurement approaches are used for integrity across various types of intervention research. This introductory paper provides a general overview of the themes that cut across the papers in this special series. We define key terms and discuss how the measurement of treatment integrity (i.e., extent to which the core practices of a protocol are delivered) and integrity to implementation strategies (i.e., extent to which the core practices of an implementation strategy are delivered) changes as an intervention progresses along the treatment development pipeline spanning from basic research to implementation and dissemination research. Then we discuss the ways that the measurement of integrity can change across the different steps in the pipeline to meet the goals associated with different stops along the pipeline. We finish the paper with a description of the papers in the special series and concluding thoughts. [This article was published in "School Mental Health" (EJ1328192).]

Teachers sometimes struggle to deliver evidence based programs designed to prevent and ameliorate chronic problem behaviors of young children with integrity. Identifying factors associated with variations in the quantity and quality of delivery is thus an important goal for the field. This study investigated factors associated with teacher treatment integrity of BEST in CLASS, a tier-2 prevention program designed for young children at risk for developing emotional/behavioral disorders. Ninety-two early childhood teachers and 231 young children at-risk for emotional/ behavioral disorders participated in the study. Latent growth curve analyses indicated that both adherence and competence of delivery increased across six observed time points. Results suggest that teacher education and initial levels of classroom quality may be important factors to consider when teachers deliver tier-2 (i.e., targeted to children who are not responsive to universal or tier-1 programming) prevention programs in early childhood settings. Teachers with higher levels of education delivered the program with more adherence and competence initially. Teachers with higher initial scores on the Emotional Support subscale of the Classroom Assessment Scoring System (CLASS) delivered the program with more competence initially and exhibited higher growth in both adherence and competence of delivery across time. Teachers with higher initial scores on the Classroom Organization subscale of the CLASS exhibited lower growth in adherence across time. Contrary to hypotheses, teacher self-efficacy did not predict adherence, and teachers who reported higher initial levels of Student Engagement self-efficacy exhibited lower growth in competence of delivery. Results are discussed in relation to teacher delivery of evidence-based programs in early childhood classrooms.

Educators are increasingly being encouraged to implement evidence-based interventions and practices to address the social, emotional, and behavioral needs of young children who exhibit problem behavior in early childhood settings. Given the nature of social-emotional learning during the early childhood years and the lack of a common set of core evidence-based practices within the early childhood literature, selection of instructional practices that foster positive social, emotional, and behavioral outcomes for children in early childhood settings can be difficult. The purpose of this paper is to report findings from a study designed to identify common practice elements found in comprehensive intervention models (i.e., manualized interventions that include a number of components) or discrete practices (i.e., a specific behavior or action) designed to target social, emotional, and behavioral learning of young children who exhibit problem behavior. We conducted a systematic review of early childhood classroom interventions that had been evaluated in randomized group designs, quasi-experimental designs, and single-case experimental designs. A total of 49 published articles were identified,and an iterative process was used to identify common practice elements. The practice elements were subsequently reviewed by experts in social-emotional and behavioral interventions for young children. Twenty-four practice elements were identified and classified into content (the goal or general principle that guides a practice element) and delivery (the way in which a teacher provides instruction to the child) categories. We discuss implications that the identification of these practice elements found in the early childhood literature has for efforts to implement models and practices.

The purpose of this paper is to determine how youth with anxiety recruited for a school-based randomized controlled trial (RCT) compare demographically and clinically to clinically referred youth with anxiety who participated in an RCT conducted in community mental health clinics. Youth ("N" = 99) with anxiety aged 12-15 years were: (a) 37 youth ("M" age = 13.70, SD = 0.89; 97.3% Norwegian; 19.0% male) from a school-based RCT and (b) 62 youth ("M" age = 13.8, SD = 1.0; 88.7% Caucasian; 29.5% male) from an RCT conducted in community mental health clinics. The youth were assessed for anxiety diagnoses, functional impairment, internalizing symptoms, and externalizing symptoms. Compared to the clinically referred youth who received care in community mental health clinics, the youth in the school sample met criteria for fewer anxiety diagnoses, lower severity of diagnoses, and less functional impairment caused by mental health problems. However, the school sample had significantly higher levels of youth-rated anxiety symptoms, "t"(95) = -2.33, "p" = 0.02, parent-rated depression, "t"(94) = 4.45, "p" < 0.001, and externalizing symptoms, "t"(96) = 2.86, "p" = 0.005. Finally, only 12.1% of the youth who met diagnostic criteria reported receiving services at a community mental health clinic in the last year. Although many of the youth in the school sample met diagnostic criteria for one or more anxiety disorders, few had received services in community mental health clinics. This suggests that recruiting in schools may help identify youth with anxiety that may not otherwise seek mental health services.

The measurement of treatment integrity is critical to evaluate the efficacy and effectiveness of evidence-based programs (EBPs) designed to improve the developmental outcomes of young children at risk of emotional/behavioral disorders. Unfortunately, the science of treatment integrity measurement lags behind the development and evaluation of EBP for young, high-risk children. This article describes the development and preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale (BiCACS), designed to measure the adherence and competence of delivery of the BEST in CLASS prevention program. Independent observers coded videotaped (n = 116) and live (n = 289) observations of teachers delivering the BEST in CLASS program. The BiCACS showed good interrater reliability and analyses provided some support for the validity of the measure. Implications for future research and integrity measurement work are discussed. [This paper was published in "Journal of Emotional and Behavioral Disorders" (EJ1044702).]

Students' early school experiences have a significant and long-term effect on key academic and social/behavioral outcomes. Evidence-based programs that both increase the quantity and quality of positive teacher-student interactions and decrease the frequency of negative teacher-student interactions may be critical for changing negative patterns and improving students' early learning environments. The purpose of this article is to describe the iterative development of one such program, BEST in CLASS--Elementary, which has been adapted from an evidence-based program designed to reduce risk for emotional/behavioral disorders by improving teachers' interactions and relationships with preschool-age children with chronic problem behavior. Following a brief description of the parent program, 4 overlapping phases of intervention adaptation and development are described. Future directions for the intervention and applications of this iterative process to intervention development work and research are discussed.

Background: Home-based parenting support within youth care services is one of the key interventions provided to families encountering difficulties with child rearing and child development. However, knowledge on factors contributing to positive outcomes of home-based parenting support is limited. Objective: The current study investigated the predictive value of (1) early parent-professional alliance and (2) change in alliance during care for outcomes of home-based parenting support. Method: Multi-informant self-report alliance and outcome data from 146 parents (Mage = 40.00, SD = 7.10; range 19-57 years) and their professionals collected early and late in care were analyzed using latent growth curve modeling. Results: Findings demonstrated that higher levels of early parent-reported alliance predicted higher levels of parent-reported satisfaction with care, and improved parent functioning. Higher levels of early professional-reported alliance predicted higher levels of parent- and professional-reported satisfaction, and improved parent functioning. Increases in professional-reported alliance during care predicted higher levels of professional-reported satisfaction and parent functioning but were not related to parent-reported outcomes. Change in parent-reported alliance was not related to outcomes. Conclusions: Together, our findings suggest that a strong parent-professional alliance represents a key process factor in realizing positive outcomes of home-based parenting support. Consequently, efforts in research and practice are needed to investigate precursors of strong alliances and to optimize professionals' ability to develop and maintain strong parent-professional alliances.

Young children with and at risk for emotional/behavioral disorders (EBD) present challenges for early childhood teachers. Evidence-based programs designed to address these young children's behavior problems exist, but there are a number of barriers to implementing these programs in early childhood settings. Advancing the science of treatment integrity measurement can assist researchers and consumers interested in implementing evidence-based programs in early childhood classrooms. To provide guidance for researchers interested in assessing the integrity of implementation efforts, we describe a conceptual model of implementation of evidence-based programs designed to prevent EBD when applied in early childhood settings. Next, we describe steps that can be used to develop treatment integrity measures. Last, we discuss factors to consider when developing treatment integrity measures with specific emphasis on psychometrically strong measures that have maximum utility for implementation research in early childhood classrooms. [This article was published in "Journal of Early Intervention," (EJ1019931).]

Although anxiety is frequently reported among children and adolescents with autism spectrum disorder (ASD), it has not been established that the manifest symptoms of anxiety in the context of ASD are the same as those seen in youth without ASD. This study sought to examine the metric and latent factor equivalence of anxiety as measured by the Multidimensional Anxiety Scale for Children, parent-report (MASC-P) and child-report (MASC-C), in youth with anxiety disorders and ASD with intact verbal ability (n=109, M(age)=11.67 years, 99 male) and a gender-matched comparison group of typically developing (TD) children and adolescents with anxiety disorders but without ASD (n=342, M(age)=11.25 years, 246 male). Multigroup factorial invariance (MFI) using structural equation modeling indicated equivalent latent factors in youth with and without ASD on the MASC-C (metric invariance). However, the item means and covariances along with the relations among the factor scores were different for the youth with ASD (i.e., lack of evidence for scalar or structural invariance). The MASC-P data did not fit the measure's established structure for either the ASD or TD group, and post-hoc exploratory factor analysis revealed a different factor structure in the ASD group. Findings suggest that the MASC-C may not measure identical constructs in anxious youth with and without ASD. Further research on the structure of the MASC in clinical samples is warranted.