Improving risk adjustment in the PRAiS (Partial Risk Adjustment in Surgery) model for mortality after paediatric cardiac surgery and improving public understanding of its use in monitoring outcomes

Background: In 2011, we developed a risk model for 30-day mortality after children’s heart surgery. The PRAiS (Partial Risk Adjustment in Surgery) model uses data on the procedure performed, diagnosis, age, weight and comorbidity. Our treatment of comorbidity was simplistic because of data quality. Software that implements PRAiS is used by the National Congenital Heart Disease Audit (NCHDA) in its audit work. The use of PRAiS triggered the temporary suspension of surgery at one unit in 2013. The public anger that surrounded this illustrated the need for public resources around outcomes monitoring. Objectives: (1) To improve the PRAiS risk model by incorporating more information about comorbidities. (2) To develop online resources for the public to help them to understand published mortality data. Design: Objective 1 The outcome measure was death within 30 days of the start of each surgical episode of care. The analysts worked with an expert panel of clinical and data management representatives. Model development followed an iterative process of clinical discussion of risk factors, development of regression models and assessment of model performance under cross-validation. Performance was measured using the area under the receiving operator characteristic (AUROC) curve and calibration in the cross-validation test sets. The final model was further assessed in a 2014–15 validation data set. Objective 2 We developed draft website material that we iteratively tested through four sets of two workshops (one workshop for parents of children who had undergone heart surgery and one workshop for other interested users). Each workshop recruited new participants. The academic psychologists ran two sets of three experiments to explore further understanding of the web content. Data: We used pseudonymised NCHDA data from April 2009 to April 2014. We later unexpectedly received a further year of data (2014–15), which became a prospective validation set. Results: Objective 1 The cleaned 2009–14 data comprised 21,838 30-day surgical episodes, with 539 deaths. The 2014–15 data contained 4207 episodes, with 97 deaths. The final regression model included four new comorbidity groupings. Under cross-validation, the model had a median AUROC curve of 0.83 (total range 0.82 to 0.83), a median calibration slope of 0.92 (total range 0.64 to 1.25) and a median intercept of –0.23 (range –1.08 to 0.85). In the validation set, the AUROC curve was 0.86 [95% confidence interval (CI) 0.83 to 0.89], and its calibration slope and intercept were 1.01 (95% CI 0.83 to 1.18) and 0.11 (95% CI –0.45 to 0.67), respectively. We recalibrated the final model on 2009–15 data and updated the PRAiS software. Objective 2 We coproduced a website (http://childrensheartsurgery.info/) that provides interactive exploration of the data, two animations and background information. It was launched in June 2016 and was very well received. Limitations: We needed to use discharge status as a proxy for 30-day life status for the 14% of overseas patients without a NHS number. We did not have sufficient time or resources to extensively test the usability and take-up of the website following its launch. Conclusions: The project successfully achieved its stated aims. A key theme throughout has been the importance of collaboration and coproduction. In particular for aim 2, we generated a great deal of generalisable learning about how to communicate complex clinical and mathematical information. Further work: Extending our codevelopment approach to cover many other aspects of quality measurement across congenital heart disease and other specialised NHS services. Funding: The National Institute for Health Research Health Services and Delivery Research programme.