Search

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system’s actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

Background Online consultations (OCs) allow patients to contact their care provider online, and have been promoted as a way to address increasing workload and decreasing workforce capacity in primary care. Globally, OCs have been rolled out rapidly due to policy initiatives and the COVID-19 pandemic, though there is a lack of evidence regarding how their design and implementation influence care outcomes.Objective Informed by existing theories, synthesise quantitative and qualitative research on: 1) outcomes of OCs in primary care; 2) how these are influenced by OC system design and implementation.Methods We searched Ovid Medline, Embase, Web of Science, Scopus, NTIS, HMIC, and ZETOC from 2010 to November 2021. We included quantitative and qualitative studies of real-world OC use in primary care, written in English, and published 2010 onwards. Quantitative data were transformed into qualitative themes. For objective 1 we used thematic synthesis informed by the Institute of Medicine’s domains of healthcare quality. For objective 2 we used Framework Analysis informed by the NASSS framework and Realistic Evaluation. Critical appraisal was conducted using the Mixed Methods Appraisal Tool and strength of evidence judged using GRADE-CERQual.Results We synthesised 62 studies (quantitative n=32, qualitative n=12, mixed methods n=18) in nine countries covering 30 unique OC systems, 13 of which used Artificial Intelligence (AI). Twenty-six were published in 2020 onwards, and 11 were post-COVID-19. There was no quantitative evidence for negative impacts of OCs on patient safety, and qualitative studies suggested perceptions of OC safety varied. Some participants believed OCs improved safety, particularly when patients could describe their queries using unstructured free-text. Staff workload decreased when sufficient resources were allocated to implement OCs, and patients used them for simple problems or could describe their queries using free-text. Staff workload increased when OCs were not integrated with other software or organisational workflows, and patients used them for complex queries. OC systems that required patients to describe their queries using multiple choice questionnaires (MCQs) increased workload for both them and staff. Health costs were reduced when patients used OCs for simple queries, and increased when used for complex ones. Patients using OCs were more likely to be female, younger, native speakers, with higher socioeconomic status than those not using OCs. However, OCs increased primary care access for patients with mental health conditions, verbal communication difficulties, and barriers to attending in-person appointments. Access also increased by providing a timely response to patients’ queries. Patient satisfaction increased when using OCs due to better primary care access, though could decrease when using MCQ formats.Conclusions This is the first theoretically-informed synthesis of research on OCs in primary care, and includes studies conducted during COVID-19. It contributes new knowledge that in addition to producing positive outcomes such as increased access and patient satisfaction, they can also have negative outcomes such as increased workload and costs. These negative outcomes can be mitigated by appropriate OC system design (e.g. free-text format), incorporating advanced technologies (e.g. AI), and integration into technical and organisational workflows (e.g. timely responses).Study protocol PROSPERO (CRD42020191802).Competing Interest StatementDr Benjamin C Brown is clinical lead for a commercially available OC system (www.patchs.ai).Clinical Protocols https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=191802 Funding StatementThis research was funded by Innovate UK (105178) and a Wellcome Trust Clinical Research Career Development Fellowship for BCB (209593/Z/17/Z). NP was partially funded by the National Institute for Health Research Greater Manchester Patient Safety Translational Research Centre (NIHR Greater Manchester PSTRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. NIHR had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all of the data and the final responsibility to submit for publication.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll papers reviewed in our systematic review are publicly available.AIArtificial IntelligenceEHRElectronic Health RecordGRADE-CERQualGrading of Recommendations Assessment, Development, and Evaluation - Confidence in the Evidence from Reviews of Qualitative researchIOMInstitute of MedicineMMATMixed Methods Appraisal ToolNASSSNon-adoption, abandonment, scale-up, spread, sustainability (framework)MCQMultiple choice questionnaireNHSNational Health ServiceOCsOnline Triage and Consultation systemsPRISMAPreferred Reporting Items for Systematic Reviews and Meta-AnalysesPROSPEROThe International Prospective Register of Systematic ReviewsUKUnited KingdomUSUnited States