Search

An overview of a direct observational system for continuous assessment of staff performance in residential treatment programs — the Staff-Resident Interaction Chronograph (SRIC) — is presented. Included are descriptions of data collection and summarization procedures, interobserver reliability data obtained over 4 1/2 years of ongoing use and during multi-institutional data collection, results of an evaluation of a self-contained package for training professional observers, and a brief description of normative-feasibility data collected across a wide range of adult residential treatment programs.

The ability of school-based measures of child behavior to predict differentially the IOWA Conners teacher rating scale Inattention/Overactivity (IO) and Aggression (A) factors was evaluated in a sample of 71 school-aged boys. Multiple measures and multiple methods of assessment were utilized to provide a comprehensive assessment of social and academic behavior. These included direct observations of children in the classroom and on the playground, examination of the organization of children's desks, measures of academic performance, peer nominations of popularity and rejection, and sociometric ratings using the Pupil Evaluation Inventory. Despite moderately high correlations between the teacher rating factors (r = .60), considerable evidence was provided for differential validity on measures of academic performance, peer rating measures, and measures of disruptive or inappropriate classroom and playground behavior. These differences between IO and A factors indicated that the factors were logically independent in important ways, supporting prior work validating separate hyperactivity and aggression dimensions.

The comparative effectiveness of two time-limited modes of training observers to code the behavior of clients in residential treatment programs on the Time-Sample Behavioral Checklist (TSBC) was evaluated. The susceptibility of training procedures to consensual observer drift and the predictability of TSBC mastery from trainee characteristics were also examined. Two equated groups of undergraduate students (N=15 each) participated in full-time training on the TSBC and another instrument for 27 days, followed by criterion testingin vivo and on videotapes. One group was trained by experienced personnel using procedures known to be effective but potentially subject to consensual observer drift. The other group was trained using a previously untested set of written and videotape procedures that do not rely on experienced personnel. Comparative effectiveness and observer drift were evaluated by multivariate and univariate ANOVAs on mastery scores reflecting both pattern agreement and level differences between each trainee and criterion codings. The new, more efficient training procedures were found to be as effective as the original training procedures in the degree of mastery achieved by trainees. Original training procedures were found to be resistant to consensual observer drift, with such a phenomenon appearing in only 1 instance of 156 opportunities. The high degree of mastery achieved by trainees during the time-limited training period was comparable to that previously achieved with the original training procedures. No meaningful predictions of coding mastery were found, with only one trainee characteristic being significantly related to one of six mastery criteria. The results document procedures that are both efficient and resistant to invalidity for training observers in the use of multidimensional observational systems, as well as providing guidelines for the development of standardized procedures.