Evaluation of the EsteR Toolkit for COVID-19 Decision Support: Sensitivity Analysis and Usability Study (Preprint)

BACKGROUND In the COVID-19 pandemic, local health authorities are responsible for managing and reporting the current cases in Germany. Since March 2020, the employees had to contain the spread of COVID-19 by monitoring and contacting infected persons as well as tracing their contacts. In the EsteR project, we implemented existing and newly developed statistical models as decision support tools to assist the work in the local health authorities. OBJECTIVE The main goal of this work is the validation of the EsteR toolkit in two complementing ways: first, investigating the stability of the answers provided by our statistical tools regarding model parameters in the backend, and second, evaluating the usability and applicability of our web application in the frontend by test users. METHODS For the model stability assessment, a sensitivity analysis was carried out for all five developed statistical models. The default parameters of our models as well as the test ranges of the model parameters were based on a prior literature review on COVID-19 properties. The obtained answers resulting from different parameters were compared with dissimilarity metrics and visualized with contour plots. In addition, the parameter ranges of general model stability were identified. For the usability evaluation of the web application, cognitive walkthroughs and focus group interviews were conducted with six containment scouts located at two different local health authorities. First, they were asked to fulfill small tasks with the tools and then, to express their general impression of the web application. RESULTS The simulation results showed that some statistical models are more sensitive to changes in their parameters than others. For each of the single person use cases we determined an area where the respective model can be rated as stable. On the other hand, the results of the group use cases highly depend on the user inputs and, thus, no area of parameters with general model stability could be identified. A detailed simulation report of the sensitivity analysis is given in Multimedia Appendix 1. In the user evaluation, the cognitive walkthroughs and the focus group interviews disclosed that the user interface needed to be simplified and more information was necessary as guidance. In general, the testers rated the web application as helpful, especially for new employees. The detailed cognitive walkthrough results can be found in Multimedia Appendix 2. CONCLUSIONS This evaluation study allowed us a refinement of the EsteR toolkit. With the sensitivity analysis, we identified suitable model parameters and analyzed how stable the statistical models are regarding changes in their parameters. Furthermore, the frontend of the web application was improved with the results of the conducted cognitive walkthroughs and focus group interviews regarding its user friendliness. CLINICALTRIAL