Search

The use of machine learning tools in health care is rapidly expanding. However, the processes that support these tools in deployment, that is, machine learning operations, are still emerging. The purpose of this work was not only to provide a comprehensive synthesis of existing literature in the field but also to identify gaps and offer insights for adoption in clinical practice. A scoping review was conducted using the MEDLINE, PubMed, Google Scholar, Embase, and Scopus databases. We used MeSH and non-MeSH search terms to identify pertinent articles, with the authors performing 2 screening phases and assigning relevance scores: 148 English language articles most salient to the review were eligible for inclusion; 98 offered the most unique information and these were supplemented by 50 additional sources, yielding 148 references. From the 148 references, we distilled 7 key topic areas, based on a synthesis of the available literature and how that aligned with practitioner needs. The 7 topic areas were machine learning model monitoring; automated retraining systems; ethics, equity, and bias; clinical workflow integration; infrastructure, human resources, and technology stack; regulatory considerations; and financial considerations. This review provides an overview of best practices and knowledge gaps of this domain in health care and identifies the strengths and weaknesses of the literature, which may be useful to health care machine learning practitioners and consumers.

Background We examined neighborhood characteristics concerning breast cancer screening annual adherence during the COVID-19 pandemic. Methods We analyzed 6673 female patients aged 40 or older at increased inherited cancer risk in 2 large health care systems (NYU Langone Health [NYULH] and the University of Utah Health [UHealth]). Multinomial models were used to identify predictors of mammogram screening groups (non-adherent, pre-pandemic adherent, pandemic period adherent) in comparison to adherent females. Potential determinants included sociodemographic characteristics and neighborhood factors. Results Comparing each cancer group in reference to the adherent group, a reduced likelihood of being non-adherent was associated with older age (OR: 0.97, 95% CI: 0.95, 0.99), a greater number of relatives with cancer (OR: 0.80, 95% CI: 0.75, 0.86), and being seen at NYULH study site (OR: 0.42, 95% CI: 0.29, 0.60). More relatives with cancer were correlated with a lesser likelihood of being pandemic period adherent (OR: 0.89, 95% CI: 0.81, 0.97). A lower likelihood of being pre-pandemic adherent was seen in areas with less education (OR: 0.77, 95% CI: 0.62, 0.96) and NYULH study site (OR: 0.35, 95% CI: 0.22, 0.55). Finally, greater neighborhood deprivation (OR: 1.47, 95% CI: 1.08, 2.01) was associated with being non-adherent. Conclusion Breast screening during the COVID-19 pandemic was associated with being older, having more relatives with cancer, residing in areas with less educational attainment, and being seen at NYULH; non-adherence was linked with greater neighborhood deprivation. These findings may mitigate risk of clinically important screening delays at times of disruptions in a population at greater risk for breast cancer.

ABSTRACT Background Individuals with high risk for lung cancer may benefit from lung cancer screening, but there are associated risks as well as benefits. Shared decision‐making (SDM) tools with personalized information may provide key support for patients. Understanding patient perspectives on educational tools to facilitate SDM for lung cancer screening may support tool development. Aim This study aimed to explore patient perspectives related to a SDM tool for lung cancer screening using a qualitative approach. Methods We elicited patient perspectives by showing a provider‐facing SDM tool. Focus group interviews that ranged in duration from 1.5 to 2 h were conducted with 23 individuals with high risk for lung cancer. Data were interpreted inductively using thematic analysis to identify patients' thoughts on and desires for a patient‐facing SDM tool. Results The findings highlight that patients would like to have educational information related to lung cancer screening. We identified several key themes to be considered in the future development of patient‐facing tools: barriers to acceptance, preference against screening and seeking empowerment. One further theme illustrated effects of patient–provider relationship as a limitation to meeting lung cancer screening information needs. Participants also noted several suggestions for the design of technology decision aids. Conclusion These findings suggest that patients desire additional information on lung cancer screening in advance of clinical visits. However, there are several issues that must be considered in the design and development of technology to meet the information needs of patients for lung cancer screening decisions. Patient or Public Contribution Patients, service users, caregivers or members of the public were not involved in the study design, conduct, analysis or interpretation of the data. However, clinical experts in health communication provided detailed feedback on the study protocol, including the focus group approach. The study findings contribute to a better understanding of patient expectations for lung cancer screening decisions and may inform future development of tools for SDM.

Introduction SCALE-UP II aims to investigate the effectiveness of population health management interventions using text messaging (TM), chatbots and patient navigation (PN) in increasing the uptake of at-home COVID-19 testing among patients in historically marginalised communities, specifically, those receiving care at community health centres (CHCs).Methods and analysis The trial is a multisite, randomised pragmatic clinical trial. Eligible patients are >18 years old with a primary care visit in the last 3 years at one of the participating CHCs. Demographic data will be obtained from CHC electronic health records. Patients will be randomised to one of two factorial designs based on smartphone ownership. Patients who self-report replying to a text message that they have a smartphone will be randomised in a 2×2×2 factorial fashion to receive (1) chatbot or TM; (2) PN (yes or no); and (3) repeated offers to interact with the interventions every 10 or 30 days. Participants who do not self-report as having a smartphone will be randomised in a 2×2 factorial fashion to receive (1) TM with or without PN; and (2) repeated offers every 10 or 30 days. The interventions will be sent in English or Spanish, with an option to request at-home COVID-19 test kits. The primary outcome is the proportion of participants using at-home COVID-19 tests during a 90-day follow-up. The study will evaluate the main effects and interactions among interventions, implementation outcomes and predictors and moderators of study outcomes. Statistical analyses will include logistic regression, stratified subgroup analyses and adjustment for stratification factors.Ethics and dissemination The protocol was approved by the University of Utah Institutional Review Board. On completion, study data will be made available in compliance with National Institutes of Health data sharing policies. Results will be disseminated through study partners and peer-reviewed publications.Trial registration number ClinicalTrials.gov: NCT05533918 and NCT05533359.

Purpose: To rigorously develop a prototype clinical decision support (CDS) system to help clinicians determine the appropriate timing for follow-up visual field testing for patients with glaucoma and to identify themes regarding the context of use for glaucoma CDS systems, design requirements, and design solutions to meet these requirements. Design: Semistructured qualitative interviews and iterative design cycles. Participants: Clinicians who care for patients with glaucoma, purposefully sampled to ensure a representation of a range of clinical specialties (glaucoma specialist, general ophthalmologist, optometrist) and years in clinical practice. Methods: Using the established User-Centered Design Process framework, we conducted semistructured interviews with 5 clinicians that addressed the context of use and design requirements for a glaucoma CDS system. We analyzed the interviews using inductive thematic analysis and grounded theory to generate themes regarding the context of use and design requirements. We created design solutions to address these requirements and used iterative design cycles with the clinicians to refine the CDS prototype. Main Outcome Measures: Themes regarding decision support for determining the timing of visual field testing for patients with glaucoma, CDS design requirements, and CDS design features. Results: We identified 9 themes that addressed the context of use for the CDS system, 9 design requirements for the prototype CDS system, and 9 design features intended to address these design requirements. Key design requirements included the preservation of clinician autonomy, incorporation of currently used heuristics, compilation of data, and increasing and communicating the level of certainty regarding the decision. After completing 3 iterative design cycles using this preliminary CDS system design solution, the design was satisfactory to the clinicians and was accepted as our prototype glaucoma CDS system. Conclusions: We used a systematic design process based on the established User-Centered Design Process to rigorously develop a prototype glaucoma CDS system, which will be used as a starting point for a future, large-scale iterative refinement and implementation process. Clinicians who care for patients with glaucoma need CDS systems that preserve clinician autonomy, compile and present data, incorporate currently used heuristics, and increase and communicate the level of certainty regarding the decision. Financial Disclosure(s): Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Objectives: Family history is an important tool for assessing disease risk, and tailoring recommendations for screening and genetic services referral. This study explored barriers to family history collection with Spanish-speaking patients. Methods: This qualitative study was conducted in two US healthcare systems. We conducted semi-structured interviews with medical assistants, physicians, and interpreters with experience collecting family history for Spanish-speaking patients. Results: The most common patient-level barrier was the perception that some Spanish-speaking patients had limited knowledge of family history. Interpersonal communication barriers related to dialectical differences and decisions about using formal interpreters vs. Spanish-speaking staff. Organizational barriers included time pressures related to using interpreters, and ad hoc workflow adaptations for Spanish-speaking patients that might leave gaps in family history collection. Conclusions: This study identified multi-level barriers to family history collection with Spanish-speaking patients in primary care. Findings suggest that a key priority to enhance communication would be to standardize processes for working with interpreters. Innovation: To improve communication with and care provided to Spanish-speaking patients, there is a need to increase healthcare provider awareness about implicit bias, to address ad hoc workflow adjustments within practice settings, to evaluate the need for professional interpreter services, and to improve digital tools to facilitate family history collection.

BackgroundWarnings about drug-drug interactions (DDIs) between warfarin and nonsteroidal anti-inflammatory drugs (NSAIDs) within electronic health records indicate potential harm but fail to account for contextual factors and preferences. We developed a tool called DDInteract to enhance and support shared decision-making (SDM) between patients and physicians when both warfarin and NSAIDs are used concurrently. DDInteract was designed to be integrated into electronic health records using interoperability standards. ObjectiveThe purpose of this study was to conduct a formative evaluation of a DDInteract that incorporates patient and product contextual factors to estimate the risk of bleeding. MethodsA randomized formative evaluation was conducted to compare DDInteract to usual care (UC) using physician-patient dyads. Using case vignettes, physicians and patients on warfarin participated in simulated virtual clinical encounters where they discussed the use of taking ibuprofen and warfarin concurrently and determined an appropriate therapeutic plan based on the patient’s individualized risk. Dyads were randomized to either DDInteract or UC. Participants completed a postsession interview and survey of the SDM process. This included the 9-item Shared Decision-Making Questionnaire (SDM-Q-9), tool usability and workload National Aeronautics and Space Administration (NASA) Task Load Index, Unified Theory of Acceptance and Use of Technology (UTAUT), Perceived Behavioral Control (PBC) scale, System Usability Scale (SUS), and Decision Conflict Scale (DCS). They also were interviewed after the session to obtain perceptions on DDInteract and UC resources for DDIs. ResultsTwelve dyad encounters were performed using virtual software. Most (n=11, 91.7%) patients were over 50 years of age, and 9 (75%) had been taking warfarin for more than 2 years (75%). Regarding scores on the SDM-Q-9, participants rated DDInteract higher than UC for questions pertaining to helping patients clarify the decision (P=.03), involving patients in the decision (P=.01), displaying treatment options (P

BackgroundFamily health history has been recognized as an essential factor for cancer risk assessment and is an integral part of many cancer screening guidelines, including genetic testing for personalized clinical management strategies. However, manually identifying eligible candidates for genetic testing is labor intensive. ObjectiveThe aim of this study was to develop a natural language processing (NLP) pipeline and assess its contribution to identifying patients who meet genetic testing criteria for hereditary cancers based on family health history data in the electronic health record (EHR). We compared an algorithm that uses structured data alone with structured data augmented using NLP. MethodsAlgorithms were developed based on the National Comprehensive Cancer Network (NCCN) guidelines for genetic testing for hereditary breast, ovarian, pancreatic, and colorectal cancers. The NLP-augmented algorithm uses both structured family health history data and the associated unstructured free-text comments. The algorithms were compared with a reference standard of 100 patients with a family health history in the EHR. ResultsRegarding identifying the reference standard patients meeting the NCCN criteria, the NLP-augmented algorithm compared with the structured data algorithm yielded a significantly higher recall of 0.95 (95% CI 0.9-0.99) versus 0.29 (95% CI 0.19-0.40) and a precision of 0.99 (95% CI 0.96-1.00) versus 0.81 (95% CI 0.65-0.95). On the whole data set, the NLP-augmented algorithm extracted 33.6% more entities, resulting in 53.8% more patients meeting the NCCN criteria. ConclusionsCompared with the structured data algorithm, the NLP-augmented algorithm based on both structured and unstructured family health history data in the EHR increased the number of patients identified as meeting the NCCN criteria for genetic testing for hereditary breast or ovarian and colorectal cancers.

Abstract Background Advances in genetics and sequencing technologies are enabling the identification of more individuals with inherited cancer susceptibility who could benefit from tailored screening and prevention recommendations. While cancer family history information is used in primary care settings to identify unaffected patients who could benefit from a cancer genetics evaluation, this information is underutilized. System-level population health management strategies are needed to assist health care systems in identifying patients who may benefit from genetic services. In addition, because of the limited number of trained genetics specialists and increasing patient volume, the development of innovative and sustainable approaches to delivering cancer genetic services is essential. Methods We are conducting a randomized controlled trial, entitled Broadening the Reach, Impact, and Delivery of Genetic Services (BRIDGE), to address these needs. The trial is comparing uptake of genetic counseling, uptake of genetic testing, and patient adherence to management recommendations for automated, patient-directed versus enhanced standard of care cancer genetics services delivery models. An algorithm-based system that utilizes structured cancer family history data available in the electronic health record (EHR) is used to identify unaffected patients who receive primary care at the study sites and meet current guidelines for cancer genetic testing. We are enrolling eligible patients at two healthcare systems (University of Utah Health and New York University Langone Health) through outreach to a randomly selected sample of 2780 eligible patients in the two sites, with 1:1 randomization to the genetic services delivery arms within sites. Study outcomes are assessed through genetics clinic records, EHR, and two follow-up questionnaires at 4 weeks and 12 months after last genetic counseling contactpre-test genetic counseling. Discussion BRIDGE is being conducted in two healthcare systems with different clinical structures and patient populations. Innovative aspects of the trial include a randomized comparison of a chatbot-based genetic services delivery model to standard of care, as well as identification of at-risk individuals through a sustainable EHR-based system. The findings from the BRIDGE trial will advance the state of the science in identification of unaffected patients with inherited cancer susceptibility and delivery of genetic services to those patients. Trial registration BRIDGE is registered as NCT03985852 . The trial was registered on June 6, 2019 at clinicaltrials.gov .

Abstract Background Studies that examine the adoption of clinical decision support (CDS) by healthcare providers have generally lacked a theoretical underpinning. The Unified Theory of Acceptance and Use of Technology (UTAUT) model may provide such a theory-based explanation; however, it is unknown if the model can be applied to the CDS literature. Objective Our overall goal was to develop a taxonomy based on UTAUT constructs that could reliably characterize CDS interventions. Methods We used a two-step process: (1) identified randomized controlled trials meeting comparative effectiveness criteria, e.g., evaluating the impact of CDS interventions with and without specific features or implementation strategies; (2) iteratively developed and validated a taxonomy for characterizing differential CDS features or implementation strategies using three raters. Results Twenty-five studies with 48 comparison arms were identified. We applied three constructs from the UTAUT model and added motivational control to characterize CDS interventions. Inter-rater reliability was as follows for model constructs: performance expectancy (κ = 0.79), effort expectancy (κ = 0.85), social influence (κ = 0.71), and motivational control (κ = 0.87). Conclusion We found that constructs from the UTAUT model and motivational control can reliably characterize features and associated implementation strategies. Our next step is to examine the quantitative relationships between constructs and CDS adoption.

Abstract Background Tobacco use remains the leading cause of death and disability in the USA and is disproportionately concentrated among low socioeconomic status (SES) populations. Community Health Centers (CHCs) are a key venue for reaching low SES populations with evidence-based tobacco cessation treatment such as Quitlines. Electronic health record (EHR)-based interventions at the point-of-care, text messaging (TM), and phone counseling have the potential to increase Quitline reach and are feasible to implement within CHCs. However, there is a lack of data to inform how, when, and in what combination these strategies should be implemented. The aims of this cluster-randomized trial are to evaluate multi-level implementation strategies to increase the Reach (i.e., proportion of tobacco-using patients who enroll in the Quitline) and Impact (i.e., Reach × Efficacy [efficacy is defined as the proportion of tobacco-using patients who enroll in Quitline treatment that successfully quit]) and to evaluate characteristics of healthcare system, providers, and patients that may influence tobacco-use outcomes. Methods This study is a multilevel, three-phase, Sequential Multiple Assignment Randomized Trial (SMART), conducted in CHCs (N = 33 clinics; N = 6000 patients). In the first phase, clinics will be randomized to two different EHR conditions. The second and third phases are patient-level randomizations based on prior treatment response. Patients who enroll in the Quitline receive no further interventions. In phase two, patients who are non-responders (i.e., patients who do not enroll in Quitline) will be randomized to receive either TM or continued-EHR. In phase three, patients in the TM condition who are non-responders will be randomized to receive either continued-TM or TM + phone coaching. Discussion This project will evaluate scalable, multi-level interventions to directly address strategic national priorities for reducing tobacco use and related disparities by increasing the Reach and Impact of evidence-based tobacco cessation interventions in low SES populations. Trial registration This trial was registered at ClinicalTrials.gov (NCT03900767) on April 4th, 2019.

Community engagement is critical to accelerate and improve implementation of evidence-based interventions to reduce health inequities. Community-engaged dissemination and implementation research (CEDI) emphasizes engaging stakeholders (e.g., community members, practitioners, community organizations, etc.) with diverse perspectives, experience, and expertise to provide tacit community knowledge regarding the local context, priorities, needs, and assets. Importantly, CEDI can help improve health inequities through incorporating unique perspectives from communities experiencing health inequities that have historically been left out of the research process. The community-engagement process that exists in practice can be highly variable, and characteristics of the process are often underreported, making it difficult to discern how engagement of community partners was used to improve implementation. This paper describes the community-engagement process for a multilevel, pragmatic randomized trial to increase the reach and impact of evidence-based tobacco cessation treatment among Community Health Center patients; describes how engagement activities and the resulting partnership informed the development of implementation strategies and improved the research process; and presents lessons learned to inform future CEDI research.

BackgroundCancer genetic testing to assess an individual’s cancer risk and to enable genomics-informed cancer treatment has grown exponentially in the past decade. Because of this continued growth and a shortage of health care workers, there is a need for automated strategies that provide high-quality genetics services to patients to reduce the clinical demand for genetics providers. Conversational agents have shown promise in managing mental health, pain, and other chronic conditions and are increasingly being used in cancer genetic services. However, research on how patients interact with these agents to satisfy their information needs is limited. ObjectiveOur primary aim is to assess user interactions with a conversational agent for pretest genetics education. MethodsWe conducted a feasibility study of user interactions with a conversational agent who delivers pretest genetics education to primary care patients without cancer who are eligible for cancer genetic evaluation. The conversational agent provided scripted content similar to that delivered in a pretest genetic counseling visit for cancer genetic testing. Outside of a core set of information delivered to all patients, users were able to navigate within the chat to request additional content in their areas of interest. An artificial intelligence–based preprogrammed library was also established to allow users to ask open-ended questions to the conversational agent. Transcripts of the interactions were recorded. Here, we describe the information selected, time spent to complete the chat, and use of the open-ended question feature. Descriptive statistics were used for quantitative measures, and thematic analyses were used for qualitative responses. ResultsWe invited 103 patients to participate, of which 88.3% (91/103) were offered access to the conversational agent, 39% (36/91) started the chat, and 32% (30/91) completed the chat. Most users who completed the chat indicated that they wanted to continue with genetic testing (21/30, 70%), few were unsure (9/30, 30%), and no patient declined to move forward with testing. Those who decided to test spent an average of 10 (SD 2.57) minutes on the chat, selected an average of 1.87 (SD 1.2) additional pieces of information, and generally did not ask open-ended questions. Those who were unsure spent 4 more minutes on average (mean 14.1, SD 7.41; P=.03) on the chat, selected an average of 3.67 (SD 2.9) additional pieces of information, and asked at least one open-ended question. ConclusionsThe pretest chat provided enough information for most patients to decide on cancer genetic testing, as indicated by the small number of open-ended questions. A subset of participants were still unsure about receiving genetic testing and may require additional education or interpersonal support before making a testing decision. Conversational agents have the potential to become a scalable alternative for pretest genetics education, reducing the clinical demand on genetics providers.

BackgroundExposure to life-threatening drug-drug interactions (DDIs) occurs despite the widespread use of clinical decision support. The DDI between warfarin and nonsteroidal anti-inflammatory drugs is common and potentially life-threatening. Patients can play a substantial role in preventing harm from DDIs; however, the current model for DDI decision-making is clinician centric. ObjectiveThis study aims to design and study the usability of DDInteract, a tool to support shared decision-making (SDM) between a patient and provider for the DDI between warfarin and nonsteroidal anti-inflammatory drugs. MethodsWe used an SDM framework and user-centered design methods to guide the design and usability of DDInteract—an SDM electronic health record app to prevent harm from clinically significant DDIs. The design involved iterative prototypes, qualitative feedback from stakeholders, and a heuristic evaluation. The usability evaluation included patients and clinicians. Patients participated in a simulated SDM discussion using clinical vignettes. Clinicians were asked to complete eight tasks using DDInteract and to assess the tool using a survey adapted from the System Usability Scale. ResultsThe designed DDInteract prototype includes the following features: a patient-specific risk profile, dynamic risk icon array, patient education section, and treatment decision tree. A total of 4 patients and 11 clinicians participated in the usability study. After an SDM session where patients and clinicians review the tool concurrently, patients generally favored pain treatments with less risk of gastrointestinal bleeding. Clinicians successfully completed the tasks with a mean of 144 (SD 74) seconds and rated the usability of DDInteract as 4.32 (SD 0.52) of 5. ConclusionsThis study expands the use of SDM to DDIs. The next steps are to determine if DDInteract can improve shared decision-making quality and to implement it across health systems using interoperable technology.

Objective To study whether clinicians who treat glaucoma are interested in using clinical decision support (CDS) tools for glaucoma, what glaucoma clinical decisions they feel would benefit from CDS, and what characteristics of CDS design they feel would be important in glaucoma clinical practice.Methods and analysis Working with the American Glaucoma Society, the Utah Ophthalmology Society and the Utah Optometric Association, we identified a group of clinicians who care for patients with glaucoma. We asked these clinicians about interest in CDS, what glaucoma clinical decisions would benefit from CDS, and what characteristics of CDS tool design would be important in glaucoma clinical practice.Results Of the 105 clinicians (31 optometrists, 10 general ophthalmologists and 64 glaucoma specialists), 93 (88.6%) were either ‘definitely’ or ‘probably’ interested in using CDS for glaucoma. There were no statistically significant differences in interest between clinical specialties (p=0.12), years in practice (p=0.85) or numbers of patients seen daily (p=0.99). Identifying progression of glaucoma was the clinical decision the largest number of clinicians felt would benefit from CDS (104/105, 99.1%). An easy to use interface was the CDS characteristic the largest number of clinicians felt would be ‘very important’ (93/105, 88.6%).Conclusion Of this group of clinicians who treat glaucoma, 88.6% were interested in using CDS for glaucoma and 99.1% felt that identification of glaucomatous progression could benefit from CDS. This level of interest supports future work to develop CDS for glaucoma.