Measuring Medical Housestaff Teamwork Performance Using Multiple Direct Observation Instruments: Comparing Apples and Apples.

Purpose: To examine the composition and concordance of existing instruments used to assess medical teams' performance.
Method: A trained observer joined 20 internal medicine housestaff teams for morning work rounds at Tufts Medical Center, a 415-bed Boston teaching hospital, from October through December 2015. The observer rated each team's performance using nine teamwork observation instruments that examined domains including team structure, leadership, situation monitoring, mutual support, and communication. Observations recorded on paper forms were stored electronically. Scores were normalized from 1 (low) to 5 (high) to account for different rating scales. Overall mean scores were calculated and graphed; weighted scores adjusted for the number of items in each teamwork domain. Teamwork scores were analyzed using t tests, pairwise correlations, and the Kruskal-Wallis statistic, and team performance was compared across instruments by domain.
Results: The nine tools incorporated five major domains, with 5 to 35 items per instrument, for a total of 161 items per observation session. In weighted and unweighted analyses, the overall teamwork performance score for a given team on a given day varied by instrument. While all of the tools identified the same low outlier, high performers on some instruments were low performers on others. Inconsistent scores for a given team across instruments persisted in domain-level analyses.
Conclusions: There was substantial variation in the rating of individual teams assessed concurrently by a single observer using multiple instruments. Because existing teamwork observation tools do not yield concordant assessments, researchers should create better tools for measuring teamwork performance.