Search

BACKGROUND: The mechanism for recording International Classification of Diseases (ICD) and diagnosis related groups (DRG) codes in a patients chart is through a certified medical coder who manually reviews the medical record at the completion of an admission. High-acuity ICD codes justify DRG modifiers, indicating the need for escalated hospital resources. In this manuscript, we demonstrate that value of rules-based computer algorithms that audit for omission of administrative codes and quantifying the downstream effects with regard to financial impacts and demographic findings did not indicate significant disparities. METHODS: All study data were acquired via the UCLA Department of Anesthesiology and Perioperative Medicines Perioperative Data Warehouse. The DataMart is a structured reporting schema that contains all the relevant clinical data entered into the EPIC (EPIC Systems, Verona, WI) electronic health record. Computer algorithms were created for eighteen disease states that met criteria for DRG modifiers. Each algorithm was run against all hospital admissions with completed billing from 2019. The algorithms scanned for the existence of disease, appropriate ICD coding, and DRG modifier appropriateness. Secondarily, the potential financial impact of ICD omissions was estimated by payor class and an analysis of ICD miscoding was done by ethnicity, sex, age, and financial class. RESULTS: Data from 34,104 hospital admissions were analyzed from January 1, 2019, to December 31, 2019. 11,520 (32.9%) hospital admissions were algorithm positive for a disease state with no corresponding ICD code. 1,990 (5.8%) admissions were potentially eligible for DRG modification/upgrade with an estimated lost revenue of $22,680,584.50. ICD code omission rates compared against reference groups (private payors, Caucasians, middle-aged patients) demonstrated significant p-values

BACKGROUND: Outcome measures are crucial to support a treat-to-target approach to rheumatoid arthritis (RA) care, yet their integration into clinical practice remains inconsistent. We developed an Electronic Heath Record-integrated, patient-facing side-car application to display RA outcomes (disease activity, functional status, pain scores), medications, and lab results during clinical visits (RA PRO Dashboard). The study aimed to evaluate patient perceptions and attitudes towards the implementation of a novel patient-facing dashboard during clinical visits using a mixed-methods approach. METHODS: RA patients whose clinicians used the dashboard at least once during their clinical visit were invited to complete a survey regarding its usefulness in care. We also conducted semi-structured interviews with a subset of patients to assess their perceptions of the dashboard. The interviews were transcribed verbatim and analyzed thematically using deductive and inductive techniques. Emerging themes and subthemes were organized into four domains of the Ecological Model of Health. RESULTS: Out of 173 survey respondents, 79% were interested in seeing the dashboard again at a future visit, 71% felt it improved their understanding of their disease, and 65% believed it helped with decision-making about their RA care. Many patients reported that the dashboard helped them discuss their RA symptoms (76%) and medications (72%) with their clinician. Interviews with 29 RA patients revealed 10 key themes: the dashboard was perceived as a valuable visual tool that improved patients understanding of RA outcome measures, enhanced their involvement in care, and increased their trust in clinicians and the clinic. Common reported limitations included concerns about reliability of RA outcome questionnaires for some RA patients and inconsistent collection and explanation of these measures by clinicians. CONCLUSIONS: In both the quantitative and qualitative components of the study, patients reported that the dashboard improved their understanding of their RA, enhanced patient-clinician communication, supported shared decision-making, and increased patient engagement in care. These findings support the use of dashboards or similar data visualization tools in RA care and can be used in future interventions to address challenges in data collection and patient education.

BACKGROUND: Improving shared decision-making using a treat-to-target approach, including the use of clinical outcome measures, is important to providing high quality care for rheumatoid arthritis (RA). We developed an Electronic Health Record (EHR) integrated, patient-facing sidecar dashboard application that displays RA outcomes, medications, and lab results for use during clinical visits (RA PRO dashboard). The purpose of this study was to assess clinician perceptions and experiences using the dashboard in a university rheumatology clinic. METHODS: We conducted focus group (FG) discussions with clinicians who had access to the dashboard as part of a randomized, stepped-wedge pragmatic trial. FGs explored clinician perceptions towards the usability, acceptability, and usefulness of the dashboard. FG data were analyzed thematically using deductive and inductive techniques; generated themes were categorized into the domains of the Technology Acceptance Model (TAM). RESULTS: 3 FG discussions were conducted with a total of 13 clinicians. Overall, clinicians were enthusiastic about the dashboard and expressed the usefulness of visualizing RA outcome trajectories in a graphical format for motivating patients, enhancing patient understanding of their RA outcomes, and improving communication about medications. Major themes that emerged from the FG analysis as barriers to using the dashboard included inconsistent collection of RA outcomes leading to sparse data in the dashboard and concerns about explaining RA outcomes, especially to patients with fibromyalgia. Other challenges included time constraints and technical difficulties refreshing the dashboard to display real-time data. Methods for integrating the dashboard into the visit varied: some clinicians used the dashboard at the beginning of the visit as they documented RA outcomes; others used it at the end to justify changes to therapy; and a few shared it only with stable patients. CONCLUSIONS: The study provides valuable insights into clinicians perceptions and experiences with the RA PRO dashboard. The dashboard showed promise in enhancing patient-clinician communication, shared decision-making, and overall acceptance among clinicians. Addressing challenges related to data collection, education, and tailoring dashboard use to specific patient populations will be crucial for maximizing its potential impact on RA care. Further research and ongoing improvements in dashboard design and implementation are warranted to ensure its successful integration into routine clinical practice.

Abstract Background Anhedonia and suicidal ideation are symptoms of major depressive disorder (MDD) that are not regularly captured in structured scales but may be captured in unstructured clinical notes. Natural language processing (NLP) techniques may be used to extract longitudinal data on suicidal behaviors and anhedonia within unstructured clinical notes. This study assessed the accuracy of using NLP techniques on electronic health records (EHRs) to identify these symptoms among patients with MDD. Methods EHR-derived, de-identified data were used from the NeuroBlu Database (version 23R1), a longitudinal behavioral health real-world database. Mental health clinicians annotated instances of anhedonia and suicidal symptoms in clinical notes creating a ground truth. Interrater reliability (IRR) was calculated using Krippendorff’s alpha. A novel transformer architecture-based NLP model was trained on clinical notes to recognize linguistic patterns and contextual cues. Each sentence was categorized into one of four labels: (1) anhedonia; (2) suicidal ideation without intent or plan; (3) suicidal ideation with intent or plan; (4) absence of suicidal ideation or anhedonia. The model was assessed using positive predictive values (PPV), negative predictive values, sensitivity, specificity, F1-score, and AUROC. Results The model was trained, tested, and validated on 2,198, 1,247, and 1,016 distinct clinical notes, respectively. IRR was 0.80. For anhedonia, suicidal ideation with intent or plan, and suicidal ideation without intent or plan the model achieved a PPV of 0.98, 0.93, and 0.87, an F1-score of 0.98, 0.91, and 0.89 during training and a PPV of 0.99, 0.95, and 0.87 and F1-score of 0.99, 0.95, and 0.89 during validation. Conclusions NLP techniques can leverage contextual information in EHRs to identify anhedonia and suicidal symptoms in patients with MDD. Integrating structured and unstructured data offers a comprehensive view of MDD’s trajectory, helping healthcare providers deliver timely, effective interventions. Addressing current limitations will further enhance NLP models, enabling more accurate extraction of critical clinical features and supporting personalized, proactive mental health care.

Abstract Background Machine learning (ML) is increasingly used to predict clinical deterioration in intensive care unit (ICU) patients through scoring systems. Although promising, such algorithms often overfit their training cohort and perform worse at new hospitals. Thus, external validation is a critical – but frequently overlooked – step to establish the reliability of predicted risk scores to translate them into clinical practice. We systematically reviewed how regularly external validation of ML-based risk scores is performed and how their performance changed in external data. Methods We searched MEDLINE, Web of Science, and arXiv for studies using ML to predict deterioration of ICU patients from routine data. We included primary research published in English before December 2023. We summarised how many studies were externally validated, assessing differences over time, by outcome, and by data source. For validated studies, we evaluated the change in area under the receiver operating characteristic (AUROC) attributable to external validation using linear mixed-effects models. Results We included 572 studies, of which 84 (14.7%) were externally validated, increasing to 23.9% by 2023. Validated studies made disproportionate use of open-source data, with two well-known US datasets (MIMIC and eICU) accounting for 83.3% of studies. On average, AUROC was reduced by -0.037 (95% CI -0.052 to -0.027) in external data, with more than 0.05 reduction in 49.5% of studies. Discussion External validation, although increasing, remains uncommon. Performance was generally lower in external data, questioning the reliability of some recently proposed ML-based scores. Interpretation of the results was challenged by an overreliance on the same few datasets, implicit differences in case mix, and exclusive use of AUROC.

Abstract Background [18F] Fluorodeoxyglucose (FDG) PET-CT is a clinical imaging modality widely used in diagnosing and staging lung cancer. The clinical findings of PET-CT studies are contained within free text reports, which can currently only be categorised by experts manually reading them. Pre-trained transformer-based language models (PLMs) have shown success in extracting complex linguistic features from text. Accordingly, we developed a multi-task ‘TNMu’ classifier to classify the presence/absence of tumour, node, metastasis (‘TNM’) findings (as defined by The Eight Edition of TNM Staging for Lung Cancer). This is combined with an uncertainty classification task (‘u’) to account for studies with ambiguous TNM status. Methods 2498 reports were annotated by a nuclear medicine physician and split into train, validation, and test datasets. For additional evaluation an external dataset (n = 461 reports) was created, and annotated by two nuclear medicine physicians with agreement reached on all examples. We trained and evaluated eleven publicly available PLMs to determine which is most effective for PET-CT reports, and compared multi-task, single task and traditional machine learning approaches. Results We find that a multi-task approach with GatorTron as PLM achieves the best performance, with an overall accuracy (all four tasks correct) of 84% and a Hamming loss of 0.05 on the internal test dataset, and 79% and 0.07 on the external test dataset. Performance on the individual TNM tasks approached expert performance with macro average F1 scores of 0.91, 0.95 and 0.90 respectively on external data. For uncertainty an F1 of 0.77 is achieved. Conclusions Our ‘TNMu’ classifier successfully extracts TNM staging information from internal and external PET-CT reports. We concluded that multi-task approaches result in the best performance, and better computational efficiency over single task PLM approaches. We believe these models can improve PET-CT services by assisting in auditing, creating research cohorts, and developing decision support systems. Our approach to handling uncertainty represents a novel first step but has room for further refinement.

Abstract Background Atrial fibrillation (AF) is a major risk factor for ischemic stroke, and early AF diagnosis may reduce associated morbidity and mortality. A 10-variable predictive model (UNAFIED) was previously developed to estimate patients’ 2-year AF risk. This study evaluated a clinical workflow incorporating UNAFIED for screening, education, and follow-up evaluation of patients visiting a cardiology clinic who may be at an elevated risk of developing AF within 2 years. Methods Patients were included if they were aged ≥ 40 years with a scheduled in-person visit at the Eskenazi Health Cardiology Clinic between October 25, 2021, and August 10, 2022. Clinical decision support identified patients with an elevated AF risk. Initial screening with 1-lead electrocardiogram devices was offered, and routine clinical practice for diagnosis and management was followed. Physicians were surveyed on their use of the workflow, attitudes toward implementation, and perceived impact on patient care. Results A total of 2827 patients had a clinic visit during the study period, of whom 1395 were eligible to be screened because they were classified as “elevated risk” by the UNAFIED predictive model. AF or atrial flutter diagnosis was newly documented for 29 patients during the study period. Of the newly diagnosed patients, 13 began anticoagulant therapy to mitigate stroke risk. Physicians (n = 13) who used the workflow most clinic days were more likely to indicate that it was easy to use, was not time-consuming, and improved patient care compared with physicians who only used the workflow occasionally. Conclusions To our knowledge, this study is the first of its kind to demonstrate clinical application of an electronic health record-based AF predictive model. The newly documented diagnoses, however, did not solely result from implementation of UNAFIED. This non-invasive, inexpensive approach could be adopted by other sites wishing to proactively screen patients at elevated risk for AF. Other sites should verify the model’s performance in their own settings and ensure compliance with evolving regulatory requirements where applicable.

Abstract Background Digital solutions can help monitor medication safety in children who are often excluded in clinical trials. The lack of reliable safety data often leads to either under- or over-dose of medications during clinical management which make them either not responding well to treatment or susceptible to adverse drug reactions (ADRs). Aim This study investigated ADR signalling techniques to detect serious ADRs in Malaysian children aged from birth to 12 years old using an electronic ADRs’ database. Methods Four techniques (Proportional Reporting Ratio (PRR), Reporting Odds Ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Multi-item Gamma Poisson Shrinker (MGPS)) were tested on ADR reports submitted to the National Pharmaceutical Regulatory Agency between 2016 and 2020. Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of the techniques were compared. Results A total of 31 medicine-Important Medical Event pairs were found and examined among the 3152 paediatric ADR reports. Three techniques (PRR, ROR, MGPS) signalled oculogyric crisis and dystonia for metoclopramide. BCPNN and MGPS signalled angioedema for paracetamol, amoxicillin and ibuprofen. Similar performances were found for PRR, ROR and BCPNN (sensitivity of 12%, specificity of 100%, PPV of 100% and NPV of 21%). MGPS revealed the highest sensitivity (20%) and NPV (23%), as well as similar specificity and PPV (100%). Conclusions This study suggests that medication safety signalling techniques could be applied on electronic health records to monitor medication safety issues in children. Clinicians and medication safety specialist could prioritise the signals for further clinical consideration and prompt response.

Abstract Background A prediction model that estimates the risk of elevated glycated hemoglobin (HbA1c) was developed from electronic health record (EHR) data to identify adult patients at risk for prediabetes who may otherwise go undetected. We aimed to assess the internal performance of a new penalized regression model using the same EHR data and compare it to the previously developed stepdown approximation for predicting HbA1c ≥ 5.7%, the cut-off for prediabetes. Additionally, we sought to externally validate and recalibrate the approximation model using 2017–2020 pre-pandemic National Health and Nutrition Examination Survey (NHANES) data. Methods We developed logistic regression models using EHR data through two approaches: the Least Absolute Shrinkage and Selection Operator (LASSO) and stepdown approximation. Internal validation was performed using the bootstrap method, with internal performance evaluated by the Brier score, C-statistic, calibration intercept and slope, and the integrated calibration index. We externally validated the approximation model by applying original model coefficients to NHANES, and we examined the approximation model’s performance after recalibration in NHANES. Results The EHR cohort included 22,635 patients, with 26% identified as having prediabetes. Both the LASSO and approximation models demonstrated similar discrimination in the EHR cohort, with optimism-corrected C-statistics of 0.760 and 0.763, respectively. The LASSO model included 23 predictor variables, while the approximation model contained 8. Among the 2,348 NHANES participants who met the inclusion criteria, 30.1% had prediabetes. External validation of the LASSO model was not possible due to the unavailability of some predictor variables. The approximation model discriminated well in the NHANES dataset, achieving a C-statistic of 0.787. Conclusion The approximation method demonstrated comparable performance to LASSO in the EHR development cohort, making it a viable option for healthcare organizations with limited resources to collect a comprehensive set of candidate predictor variables. NHANES data may be suitable for externally validating a clinical prediction model developed with EHR data to assess generalizability to a nationally representative sample, depending on the model’s intended use and the alignment of predictor variable definitions with those used in the model’s original development.

Abstract Background At the beginning of the COVID-19 pandemic in 2020, little was known about the spread of COVID-19 in Dutch nursing homes while older people were particularly at risk of severe symptoms. Therefore, attempts were made to develop a nationwide COVID-19 repository based on routinely recorded data in the electronic health records (EHRs) of nursing home residents. This study aims to describe the facilitators and barriers encountered during the development of the repository and the lessons learned regarding the reuse of EHR data for surveillance and research purposes. Methods Using inductive content analysis, we reviewed 325 documents written and saved during the development of the COVID-19 repository. This included meeting minutes, e-mails, notes made after phone calls with stakeholders, and documents developed to inform stakeholders. We also assessed the fitness for purpose of the data by evaluating the completeness, plausibility, conformity, and timeliness of the data. Results Key facilitators found in this study were: 1) inter-organizational collaboration to create support; 2) early and close involvement of EHR software vendors; and 3) coordination and communication between partners. Key barriers that hampered the fitness of EHR data for surveillance were: 1) changes over time in national SARS-CoV-2 testing policy; 2) differences between EHR systems; 3) increased workload in nursing homes and lack of perceived urgency; 4) uncertainty regarding the legal requirements for extracting EHR data; 5) the short notice at which complete and understandable information about the repository had to be developed; and 6) lack of clarity about the differences between various COVID-19 monitors. Conclusions Despite the urgent need for information on the spread of SARS‐CoV‐2 among nursing home residents, setting up a repository based on EHR data proved challenging. The facilitators and barriers found in this study affected the extent to which the data could be used. We formulated nine lessons learned for developing future repositories based on EHR data for surveillance and research purposes. These lessons were in three main areas: legal framework, contextual circumstances, and quality of the data. Currently, these lessons are being applied in setting up a new registry in the nursing home sector.

Abstract Background The proliferation of electronic health records (EHR) in health systems of many high-income countries has ushered in profound changes to how clinical information is used, stored, and disseminated. For patients, being able to easily access and share their health information electronically through interoperable EHRs can often impact safety and their experience when seeking care across healthcare providers. While extensive research exists examining how EHRs affected workflow and technical challenges such as limited interoperability, much of it was done from the viewpoint of healthcare staff rather than from patients themselves. This leaves a critical knowledge gap in our evidence base to inform better implementation of health information technologies which needs addressing. Aims and objectives This study aimed to explore how patients with chronic conditions or polypharmacy and their caregivers perceive the current state of EHR interoperability, identify instances where it was associated with negative health outcomes, and elucidate patient-driven recommendations to address concerns raised. Methods A total of 18 patients and caregivers participated in five online focus groups between May-July 2022. Thematic analysis was performed to generate codes and derive higher-order themes. Results Participants highlighted that EHR interoperability in the NHS does not meet patient needs and expectations. While patients’ understanding of the concept of EHR interoperability was mixed, most were able to describe how the inability to seamlessly share health information within EHR has negatively impacted care. Limited interoperability contributed to inaccurate medical records, perpetuated existing incorrect information, impaired clinical decision-making, and often required patients to resort to using workarounds. Patients also voiced ideas for potential solutions for consideration. These included a move towards a one-centralised system approach, strengthening data security measures to augment other efforts to increase interoperability, prioritising health information technology training for NHS staff, and involving more allied health professionals and patients themselves in the EHR data curation process. Conclusion Our study contributes to the existing body of literature by providing the perspectives of patients and carers most likely to encounter interoperability challenges and therefore those most ideally positioned to propose potential solutions. As highlighted by patients, researchers and policymakers should consider social, educational, and organisational solutions, in addition to technical solutions. Public interest Summary Lack of interoperability (i.e., the ability to share a patient’s health information electronically between healthcare providers) can affect the quality of care received. However, much of the existing research was done from the viewpoint of healthcare staff rather than from patients themselves. This study explored the views of patients regarding what they feel interoperability in the NHS is currently like, how they think it affects their care, and what they think can be done to improve it. Patients reported interoperability to often be poor. It caused inconvenience both to themselves and their healthcare provider, and negatively impacted their experience getting care overall. Patient suggestions for improvement included centralising and reducing the number of existing systems, having more training for healthcare staff, and supporting patients and other healthcare staff in managing their health data.

Abstract Background The digitisation of healthcare records has generated vast amounts of unstructured data, presenting opportunities for improvements in disease diagnosis when clinical coding falls short, such as in the recording of patient symptoms. This study presents an approach using natural language processing to extract clinical concepts from free-text which are used to automatically form diagnostic criteria for lung cancer from unstructured secondary-care data. Methods Patients aged 40 and above who underwent a chest x-ray (CXR) between 2016 and 2022 were included. ICD-10 and unstructured data were pulled from their electronic health records (EHRs) over the preceding 12 months to the CXR. The unstructured data were processed using named entity recognition to extract symptoms, which were mapped to SNOMED-CT codes. Subsumption of features up the SNOMED-CT hierarchy was used to mitigate against sparse features and a frequency-based criteria, combined with univariate logarithmic probabilities, was applied to select candidate features to take forward to the model development phase. A genetic algorithm was employed to identify the most discriminating features to form the diagnostic criteria. Results 75002 patients were included, with 1012 lung cancer diagnoses made within 12 months of the CXR. The best-performing model achieved an AUROC of 0.72. Results showed that an existing ‘disorder of the lung’, such as pneumonia, and a ‘cough’ increased the probability of a lung cancer diagnosis. ‘Anomalies of great vessel’, ‘disorder of the retroperitoneal compartment’ and ‘context-dependent findings’, such as pain, statistically reduced the risk of lung cancer, making other diagnoses more likely. The performance of the developed model was compared to the existing cancer risk scores, demonstrating superior performance. Conclusions The proposed methods demonstrated success in leveraging unstructured secondary-care data to derive diagnostic criteria for lung cancer, outperforming existing risk tools. These advancements show potential for enhancing patient care and results. However, it is essential to tackle specific limitations by integrating primary care data to ensure a more thorough and unbiased development of diagnostic criteria. Moreover, the study highlights the importance of contextualising SNOMED-CT concepts into meaningful terminology that resonates with clinicians, facilitating a clearer and more tangible understanding of the criteria applied.

Abstract Purpose of Review Embedding machine learning workflows into real-world hospital environments is essential to ensure model alignment with clinical workflows and real-world data. Many non-healthcare industries undergoing digital transformation have already developed data labelling and data quality management services as a vertically integrated business process. Recent Findings In this paper, we describe our experiences developing and implementing a first-of-its-kind clinical NLP (natural language processing) service in the National Health Service, United Kingdom using parallel harmonised platforms. We report on our work developing clinical NLP resources and implementation framework to distil expert clinical knowledge into our NLP models. To date, we have amassed over 26,086 annotations spanning 556 SNOMED CT concepts working with secondary care specialties. Summary Our integrated language modelling service has delivered numerous clinical and operational use-cases using named entity recognition (NER). Such services improve efficiency of healthcare delivery and drive downstream data-driven technologies. We believe it will only be a matter of time before NLP services become an integral part of healthcare providers.

Abstract Background Evaluating healthcare information systems, such as the Electronic Health Records (EHR), is both challenging and essential, especially in resource-limited countries. This study aims to psychometrically develop and validate an instrument (questionnaire) to assess the factors influencing the successful adoption of the EHR system by healthcare professionals in Moroccan university hospitals. Methods The questionnaire validation process occurred in two main stages. Initially, data collected from a pilot sample of 164 participants underwent analysis using exploratory factor analysis (EFA) to evaluate the validity and reliability of the retained factor structure. Subsequently, the validity of the overall measurement model was confirmed using confirmatory factor analysis (CFA) in a sample of 368 healthcare professionals. Results The structure of the modified HOT-fit model, comprising seven constructs (System Quality, Information Quality, Information technology Service Quality, User Satisfaction, Organization, Environment, and Clinical Performance), was confirmed through confirmatory factor analysis. Absolute, incremental, and parsimonious fit indices all indicated an appropriate level of acceptability, affirming the robustness of the measurement model. Additionally, the instrument demonstrated adequate reliability and convergent validity, with composite reliability values ranging from 0.75 to 0.89 and average variance extracted (AVE) values ranging from 0.51 to 0.63. Furthermore, the square roots of AVE values exceeded the correlations between different pairs of constructs, and the heterotrait-monotrait ratio of correlations (HTMT) was below 0.85, confirming suitable discriminant validity. Conclusions The resulting instrument, due to its rigorous development and validation process, can serve as a reliable and valid tool for assessing the success of information technologies in similar contexts.

Abstract Background Social and behavioral determinants of health (SBDH) are associated with a variety of health and utilization outcomes, yet these factors are not routinely documented in the structured fields of electronic health records (EHR). The objective of this study was to evaluate different machine learning approaches for detection of SBDH from the unstructured clinical notes in the EHR. Methods Latent Semantic Indexing (LSI) was applied to 2,083,180 clinical notes corresponding to 46,146 patients in the MIMIC-III dataset. Using LSI, patients were ranked based on conceptual relevance to a set of keywords (lexicons) pertaining to 15 different SBDH categories. For Generative Pretrained Transformer (GPT) models, API requests were made with a Python script to connect to the OpenAI services in Azure, using gpt-3.5-turbo-1106 and gpt-4-1106-preview models. Prediction of SBDH categories were performed using a logistic regression model that included age, gender, race and SBDH ICD-9 codes. Results LSI retrieved patients according to 15 SBDH domains, with an overall average PPV $$\ge$$ ≥ 83%. Using manually curated gold standard (GS) sets for nine SBDH categories, the macro-F1 score of LSI (0.74) was better than ICD-9 (0.71) and GPT-3.5 (0.54), but lower than GPT-4 (0.80). Due to document size limitations, only a subset of the GS cases could be processed by GPT-3.5 (55.8%) and GPT-4 (94.2%), compared to LSI (100%). Using common GS subsets for nine different SBDH categories, the macro-F1 of ICD-9 combined with either LSI (mean 0.88, 95% CI 0.82-0.93), GPT-3.5 (0.86, 0.82-0.91) or GPT-4 (0.88, 0.83-0.94) was not significantly different. After including age, gender, race and ICD-9 in a logistic regression model, the AUC for prediction of six out of the nine SBDH categories was higher for LSI compared to GPT-4.0. Conclusions These results demonstrate that the LSI approach performs comparable to more recent large language models, such as GPT-3.5 and GPT-4.0, when using the same set of documents. Importantly, LSI is robust, deterministic, and does not have document-size limitations or cost implications, which make it more amenable to real-world applications in health systems.

Abstract Background Electronic Health Record systems (EHRs) offer significant benefits and have transformed healthcare in developed countries. However, their implementation and adoption in low- and middle-income countries (LMICs) remains low due to challenges and competing interests. Health professionals’ perception of EHRs can influence their adoption and continued use. The objectives of this study are to explore the perception of health professionals regarding implemented EHR systems in three hospitals in Ghana and identify factors influencing their perception and satisfaction. Methods In this study, we employed a concurrent mixed method design to collect data from study participants from May to June 2023. The quantitative part employed a descriptive-survey and the qualitative (in-depth interview) techniques was applied. After obtaining written informed consent from each respondent, a structured survey questionnaire was filled out by the health professionals from three hospitals. An a priori power calculation was used to determine the sample size for the quantitative component. Two hundred and sixty-three (263) health professionals completed the questionnaire from the three facilities. A purposive sampling technique was used to select fifteen [1] participants for the interviews. A semi-structured interview guide was used for the in-depth interviews. The interviews were audio recorded, transcribed, and coded into themes using QSR Nvivo 12 software before thematic content analysis. Results Our findings revealed that 213 (80.99%) health professionals perceived the EHRs as beneficial to patients and were generally satisfied. An overwhelming majority, 197 (74.90%) of the health professionals, were satisfied with its use and expressed interest in continuing to use the system. The majority of health professionals viewed the EHRs to have improved their work and workflow processes and provided the desired results. However, few other health professionals were dissatisfied with the system because they viewed the EHRs as frustrating due to unstable internet connectivity and power supply. Other concerns were related to the privacy and confidentiality of patient information. They believe access to patient information should be on a need-to-know basis, and patient information should not be accessible to all other clinicians except those involved directly in their care processes. Conclusion The study revealed that health professionals have a positive perception of the implemented EHRs, are highly satisfied with them, and are interested in continuing to use them. However, health professionals’ concerns about the unstable power supply, poor internet connectivity, security, and confidentiality of patient’s information need attention, to mitigate their frustrations and boost their confidence in the system.

Abstract Background Recognizing the limitations of pre-market clinical data, regulatory authorities have embraced total product lifecycle management with post-market surveillance (PMS) data to assess medical device safety and performance. One method of proactive PMS involves the analysis of real-world data (RWD) through retrospective review of electronic health records (EHR). Because EHRs are patient-centered and focused on providing tools that clinicians use to determine care rather than collecting information on individual medical products, the process of transforming RWD into real-world evidence (RWE) can be laborious, particularly for medical devices with broad clinical use and extended clinical follow-up. This study describes a method to extract RWD from EHR to generate RWE on the safety and performance of embolization coils. Methods Through a partnership between a non-profit data institute and a medical device manufacturer, information on implantable embolization coils’ use was extracted, linked, and analyzed from clinical data housed in an electronic data warehouse from the state of Indiana’s largest health system. To evaluate the performance and safety of the embolization coils, technical success and safety were defined as per the Society of Interventional Radiology guidelines. A multi-prong strategy including electronic and manual review of unstructured (clinical chart notes) and structured data (International Classification of Disease codes), was developed to identify patients with relevant devices and extract data related to the endpoints. Results A total of 323 patients were identified as treated using Cook Medical Tornado, Nester, or MReye embolization coils between 1 January 2014 and 31 December 2018. Available clinical follow-up for these patients was 1127 ± 719 days. Indications for use, adverse events, and procedural success rates were identified via automated extraction of structured data along with review of available unstructured data. The overall technical success rate was 96.7%, and the safety events rate was 5.3% with 18 major adverse events in 17 patients. The calculated technical success and safety rates met pre-established performance goals (≥ 85% for technical success and ≤ 12% for safety), highlighting the relevance of this surveillance method. Conclusions Generating RWE from RWD requires careful planning and execution. The process described herein provided valuable longitudinal data for PMS of real-world device safety and performance. This cost-effective approach can be translated to other medical devices and similar RWD database systems.

IntroductionAdverse drug events (ADEs) are associated with poor outcomes and increased costs but may be prevented with prediction tools. With the National Institute of Health All of Us (AoU) database, we employed machine learning (ML) to predict selective serotonin reuptake inhibitor (SSRI)-associated bleeding.MethodsThe AoU program, beginning in 05/2018, continues to recruit ≥ 18 years old individuals across the United States. Participants completed surveys and consented to contribute electronic health record (EHR) for research. Using the EHR, we determined participants who were exposed to SSRIs (citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, sertraline, vortioxetine). Features (n = 88) were selected with clinicians' input and comprised sociodemographic, lifestyle, comorbidities, and medication use information. We identified bleeding events with validated EHR algorithms and applied logistic regression, decision tree, random forest, and extreme gradient boost to predict bleeding during SSRI exposure. We assessed model performance with area under the receiver operating characteristic curve statistic (AUC) and defined clinically significant features as resulting in > 0.01 decline in AUC after removal from the model, in three of four ML models.ResultsThere were 10,362 participants exposed to SSRIs, with 9.6% experiencing a bleeding event during SSRI exposure. For each SSRI, performance across all four ML models was relatively consistent. AUCs from the best models ranged 0.632-0.698. Clinically significant features included health literacy for escitalopram, and bleeding history and socioeconomic status for all SSRIs.ConclusionsWe demonstrated feasibility of predicting ADEs using ML. Incorporating genomic features and drug interactions with deep learning models may improve ADE prediction.

Abstract Background Electronic Health Records (EHR) are widely used to develop clinical prediction models (CPMs). However, one of the challenges is that there is often a degree of informative missing data. For example, laboratory measures are typically taken when a clinician is concerned that there is a need. When data are the so-called Not Missing at Random (NMAR), analytic strategies based on other missingness mechanisms are inappropriate. In this work, we seek to compare the impact of different strategies for handling missing data on CPMs performance. Methods We considered a predictive model for rapid inpatient deterioration as an exemplar implementation. This model incorporated twelve laboratory measures with varying levels of missingness. Five labs had missingness rate levels around 50%, and the other seven had missingness levels around 90%. We included them based on the belief that their missingness status can be highly informational for the prediction. In our study, we explicitly compared the various missing data strategies: mean imputation, normal-value imputation, conditional imputation, categorical encoding, and missingness embeddings. Some of these were also combined with the last observation carried forward (LOCF). We implemented logistic LASSO regression, multilayer perceptron (MLP), and long short-term memory (LSTM) models as the downstream classifiers. We compared the AUROC of testing data and used bootstrapping to construct 95% confidence intervals. Results We had 105,198 inpatient encounters, with 4.7% having experienced the deterioration outcome of interest. LSTM models generally outperformed other cross-sectional models, where embedding approaches and categorical encoding yielded the best results. For the cross-sectional models, normal-value imputation with LOCF generated the best results. Conclusion Strategies that accounted for the possibility of NMAR missing data yielded better model performance than those did not. The embedding method had an advantage as it did not require prior clinical knowledge. Using LOCF could enhance the performance of cross-sectional models but have countereffects in LSTM models.

Abstract Many state-of-the-art results in natural language processing (NLP) rely on large pre-trained language models (PLMs). These models consist of large amounts of parameters that are tuned using vast amounts of training data. These factors cause the models to memorize parts of their training data, making them vulnerable to various privacy attacks. This is cause for concern, especially when these models are applied in the clinical domain, where data are very sensitive. Training data pseudonymization is a privacy-preserving technique that aims to mitigate these problems. This technique automatically identifies and replaces sensitive entities with realistic but non-sensitive surrogates. Pseudonymization has yielded promising results in previous studies. However, no previous study has applied pseudonymization to both the pre-training data of PLMs and the fine-tuning data used to solve clinical NLP tasks. This study evaluates the effects on the predictive performance of end-to-end pseudonymization of Swedish clinical BERT models fine-tuned for five clinical NLP tasks. A large number of statistical tests are performed, revealing minimal harm to performance when using pseudonymized fine-tuning data. The results also find no deterioration from end-to-end pseudonymization of pre-training and fine-tuning data. These results demonstrate that pseudonymizing training data to reduce privacy risks can be done without harming data utility for training PLMs.

Abstract The analysis of extensive electronic health records (EHR) datasets often calls for automated solutions, with machine learning (ML) techniques, including deep learning (DL), taking a lead role. One common task involves categorizing EHR data into predefined groups. However, the vulnerability of EHRs to noise and errors stemming from data collection processes, as well as potential human labeling errors, poses a significant risk. This risk is particularly prominent during the training of DL models, where the possibility of overfitting to noisy labels can have serious repercussions in healthcare. Despite the well-documented existence of label noise in EHR data, few studies have tackled this challenge within the EHR domain. Our work addresses this gap by adapting computer vision (CV) algorithms to mitigate the impact of label noise in DL models trained on EHR data. Notably, it remains uncertain whether CV methods, when applied to the EHR domain, will prove effective, given the substantial divergence between the two domains. We present empirical evidence demonstrating that these methods, whether used individually or in combination, can substantially enhance model performance when applied to EHR data, especially in the presence of noisy/incorrect labels. We validate our methods and underscore their practical utility in real-world EHR data, specifically in the context of COVID-19 diagnosis. Our study highlights the effectiveness of CV methods in the EHR domain, making a valuable contribution to the advancement of healthcare analytics and research.

Abstract Background Irregular time series (ITS) are common in healthcare as patient data is recorded in an electronic health record (EHR) system as per clinical guidelines/requirements but not for research and depends on a patient’s health status. Due to irregularity, it is challenging to develop machine learning techniques to uncover vast intelligence hidden in EHR big data, without losing performance on downstream patient outcome prediction tasks. Methods In this paper, we propose Perceiver, a cross-attention-based transformer variant that is computationally efficient and can handle long sequences of time series in healthcare. We further develop continuous patient state attention models, using Perceiver and transformer to deal with ITS in EHR. The continuous patient state models utilise neural ordinary differential equations to learn patient health dynamics, i.e., patient health trajectory from observed irregular time steps, which enables them to sample patient state at any time. Results The proposed models’ performance on in-hospital mortality prediction task on PhysioNet-2012 challenge and MIMIC-III datasets is examined. Perceiver model either outperforms or performs at par with baselines, and reduces computations by about nine times when compared to the transformer model, with no significant loss of performance. Experiments to examine irregularity in healthcare reveal that continuous patient state models outperform baselines. Moreover, the predictive uncertainty of the model is used to refer extremely uncertain cases to clinicians, which enhances the model’s performance. Code is publicly available and verified at https://codeocean.com/capsule/4587224 . Conclusions Perceiver presents a computationally efficient potential alternative for processing long sequences of time series in healthcare, and the continuous patient state attention models outperform the traditional and advanced techniques to handle irregularity in the time series. Moreover, the predictive uncertainty of the model helps in the development of transparent and trustworthy systems, which can be utilised as per the availability of clinicians.

Abstract Background Systemic lupus erythematosus (SLE) is a rare autoimmune disorder characterized by an unpredictable course of flares and remission with diverse manifestations. Lupus nephritis, one of the major disease manifestations of SLE for organ damage and mortality, is a key component of lupus classification criteria. Accurately identifying lupus nephritis in electronic health records (EHRs) would therefore benefit large cohort observational studies and clinical trials where characterization of the patient population is critical for recruitment, study design, and analysis. Lupus nephritis can be recognized through procedure codes and structured data, such as laboratory tests. However, other critical information documenting lupus nephritis, such as histologic reports from kidney biopsies and prior medical history narratives, require sophisticated text processing to mine information from pathology reports and clinical notes. In this study, we developed algorithms to identify lupus nephritis with and without natural language processing (NLP) using EHR data from the Northwestern Medicine Enterprise Data Warehouse (NMEDW). Methods We developed five algorithms: a rule-based algorithm using only structured data (baseline algorithm) and four algorithms using different NLP models. The first NLP model applied simple regular expression for keywords search combined with structured data. The other three NLP models were based on regularized logistic regression and used different sets of features including positive mention of concept unique identifiers (CUIs), number of appearances of CUIs, and a mixture of three components (i.e. a curated list of CUIs, regular expression concepts, structured data) respectively. The baseline algorithm and the best performing NLP algorithm were externally validated on a dataset from Vanderbilt University Medical Center (VUMC). Results Our best performing NLP model incorporated features from both structured data, regular expression concepts, and mapped concept unique identifiers (CUIs) and showed improved F measure in both the NMEDW (0.41 vs 0.79) and VUMC (0.52 vs 0.93) datasets compared to the baseline lupus nephritis algorithm. Conclusion Our NLP MetaMap mixed model improved the F-measure greatly compared to the structured data only algorithm in both internal and external validation datasets. The NLP algorithms can serve as powerful tools to accurately identify lupus nephritis phenotype in EHR for clinical research and better targeted therapies.

Abstract Background Physicians’ work is often stressful. The digitalization of healthcare aims to streamline work, but not all physicians have experienced its realization. We examined associations of perceived changes in work due to digitalization and the amount of digital work with job strain among physicians. The moderating role of the length of work experience was investigated for these associations. Methods We used representative survey data on Finnish physicians’ (N = 4271) experiences of digitalization from 2021. The independent variables included perceptions on statements about work transformations aligned with digitalization goals, and the extent that information systems and teleconsultations were utilized. Stress related to information systems (SRIS), time pressure, and psychological stress were the dependent variables. We analyzed the associations using multivariable linear and logistic regressions. Results Respondents had a mean SRIS score of 3.5 and a mean time pressure score of 3.7 on a scale of 1–5. Psychological stress was experienced by 60%. Perceptions associated with higher SRIS comprised disagreements with statements asserting that digitalization accelerates clinical encounters (b = .23 [95% CI: .16–.30]), facilitates access to patient information (b = .15 [.07–.23]), and supports decision-making (b = .11 [.05–.18]). Disagreement with accelerated clinical encounters (b = .12 [.04–.20]), and agreements with patients’ more active role in care (b = .11 [.04–.19]) and interprofessional collaboration (b = .10 [.02–.18]) were opinions associated with greater time pressure. Disagreeing with supported decision-making (OR = 1.26 [1.06–1.48]) and agreeing with patients’ active role (OR = 1.19 [1.02–1.40]) were associated with greater psychological stress. However, perceiving improvements in the pace of clinical encounters and access to patient information appeared to alleviate job strain. Additionally, extensive digital work was consistently linked to higher strain. Those respondents who held teleconsultations frequently and had less than 6 years of work experience reported the greatest levels of time pressure. Conclusions Physicians seem to be strained by frequent teleconsultations and work that does not meet the goals of digitalization. Improving physicians’ satisfaction with digitalization through training specific to the stage of career and system development can be crucial for their well-being. Schedules for digital tasks should be planned and allocated to prevent strain related to achieving the digitalization goals.

Abstract Background Depression is one of the most significant health conditions in personal, social, and economic impact. The aim of this review is to summarize existing literature in which machine learning methods have been used in combination with Electronic Health Records for prediction of depression. Methods Systematic literature searches were conducted within arXiv, PubMed, PsycINFO, Science Direct, SCOPUS and Web of Science electronic databases. Searches were restricted to information published after 2010 (from 1st January 2011 onwards) and were updated prior to the final synthesis of data (27th January 2022). Results Following the PRISMA process, the initial 744 studies were reduced to 19 eligible for detailed evaluation. Data extraction identified machine learning methods used, types of predictors used, the definition of depression, classification performance achieved, sample size, and benchmarks used. Area Under the Curve (AUC) values more than 0.9 were claimed, though the average was around 0.8. Regression methods proved as effective as more developed machine learning techniques. Limitations The categorization, definition, and identification of the numbers of predictors used within models was sometimes difficult to establish, Studies were largely Western Educated Industrialised, Rich, Democratic (WEIRD) in demography. Conclusion This review supports the potential use of machine learning techniques with Electronic Health Records for the prediction of depression. All the selected studies used clinically based, though sometimes broad, definitions of depression as their classification criteria. The reported performance of the studies was comparable to or even better than that found in primary care. There are concerns with generalizability and interpretability.

Abstract Background Primary care electronic health records (EHR) are widely used to study long-term conditions in epidemiological and health services research. Therefore, it is important to understand how well the recorded prevalence of these conditions in EHRs, compares to other reliable sources overall, and varies by socio-demographic characteristics. We aimed to describe the prevalence and socio-demographic variation of cardiovascular, renal, and metabolic (CRM) and mental health (MH) conditions in a large, nationally representative, English primary care database and compare with prevalence estimates from other population-based studies. Methods This was a cross-sectional study using the Clinical Practice Research Datalink (CPRD) Aurum primary care database. We calculated prevalence of 18 conditions and used logistic regression to assess how this varied by age, sex, ethnicity, and socio-economic status. We searched the literature for population prevalence estimates from other sources for comparison with the prevalences in CPRD Aurum. Results Depression (16.0%, 95%CI 16.0–16.0%) and hypertension (15.3%, 95%CI 15.2–15.3%) were the most prevalent conditions among 12.4 million patients. Prevalence of most conditions increased with socio-economic deprivation and age. CRM conditions, schizophrenia and substance misuse were higher in men, whilst anxiety, depression, bipolar and eating disorders were more common in women. Cardiovascular risk factors (hypertension and diabetes) were more prevalent in black and Asian patients compared with white, but the trends in prevalence of cardiovascular diseases by ethnicity were more variable. The recorded prevalences of mental health conditions were typically twice as high in white patients compared with other ethnic groups. However, PTSD and schizophrenia were more prevalent in black patients. The prevalence of most conditions was similar or higher in the primary care database than diagnosed disease prevalence reported in national health surveys. However, screening studies typically reported higher prevalence estimates than primary care data, especially for PTSD, bipolar disorder and eating disorders. Conclusions The prevalence of many clinically diagnosed conditions in primary care records closely matched that of other sources. However, we found important variations by sex and ethnicity, which may reflect true variation in prevalence or systematic differences in clinical presentation and practice. Primary care data may underrepresent the prevalence of undiagnosed conditions, particularly in mental health.

Abstract Background Important clinical information of patients is present in unstructured free-text fields of Electronic Health Records (EHRs). While this information can be extracted using clinical Natural Language Processing (cNLP), the recognition of negation modifiers represents an important challenge. A wide range of cNLP applications have been developed to detect the negation of medical entities in clinical free-text, however, effective solutions for languages other than English are scarce. This study aimed at developing a solution for negation recognition in Spanish EHRs based on a combination of a customized rule-based NegEx layer and a convolutional neural network (CNN). Methods Based on our previous experience in real world evidence (RWE) studies using information embedded in EHRs, negation recognition was simplified into a binary problem (‘affirmative’ vs. ‘non-affirmative’ class). For the NegEx layer, negation rules were obtained from a publicly available Spanish corpus and enriched with custom ones, whereby the CNN binary classifier was trained on EHRs annotated for clinical named entities (cNEs) and negation markers by medical doctors. Results The proposed negation recognition pipeline obtained precision, recall, and F1-score of 0.93, 0.94, and 0.94 for the ‘affirmative’ class, and 0.86, 0.84, and 0.85 for the ‘non-affirmative’ class, respectively. To validate the generalization capabilities of our methodology, we applied the negation recognition pipeline on EHRs (6,710 cNEs) from a different data source distribution than the training corpus and obtained consistent performance metrics for the ‘affirmative’ and ‘non-affirmative’ class (0.95, 0.97, and 0.96; and 0.90, 0.83, and 0.86 for precision, recall, and F1-score, respectively). Lastly, we evaluated the pipeline against two publicly available Spanish negation corpora, the IULA and NUBes, obtaining state-of-the-art metrics (1.00, 0.99, and 0.99; and 1.00, 0.93, and 0.96 for precision, recall, and F1-score, respectively). Conclusion Negation recognition is a source of low precision in the retrieval of cNEs from EHRs’ free-text. Combining a customized rule-based NegEx layer with a CNN binary classifier outperformed many other current approaches. RWE studies highly benefit from the correct recognition of negation as it reduces false positive detections of cNE which otherwise would undoubtedly reduce the credibility of cNLP systems.

Abstract Background Clinical events suggestive of nutrition care found in electronic health records (EHRs) are rarely explored for their associations with hypertension outcomes. Methods Longitudinal analysis using structured EHR data from primary care visits at a health system in the US from December 2017—December 2020 of adult patients with hypertension (n = 4,237) tested for associations between last visit blood pressure (BP) control (≤ 140 Systolic BP and ≤ 90 Diastolic BP) and ≥ 1 nutrition care clinical event operationalized as (overweight or obesity (BMI > 25 or 30, respectively) diagnoses, preventive care visits, or provision of patient education materials (PEM)). Descriptive statistics and longitudinal targeted maximum likelihood estimation (LTMLE) models were conducted to explore average treatment effects (ATE) of timing and dose response from these clinical events on blood pressure control overall and by race. Results The median age was 62 years, 29% were male, 52% were Black, 25% were from rural areas and 50% had controlled BP at baseline. Annual documentation of overweight/obesity diagnoses ranged 3.0–7.8%, preventive care visits ranged 6.2–15.7%, and PEM with dietary and hypertension content were distributed to 8.5–28.8% patients. LTMLE models stratified by race showed differences in timing, dose, and type of nutrition care. Black patients who had nutrition care in Year 3 only compared to none had lower odds for BP control (ATE -0.23, 95% CI: -0.38,-0.08, p = 0.003), preventive visits in the last 2 years high higher odds for BP control (ATE 0.31, 95% CI: 0.07,0.54, p = 0.01), and early or late PEMs had lower odds for BP control (ATE -0.08, 95% CI: -0.15,-0.01, p = 0.03 and ATE -0.23, 95% CI: -0.41,-0.05, p = 0.01, respectively). Conclusions In this study, clinical events suggestive of nutrition care are significantly associated with BP control, but are infrequent and effects differ by type, timing, and patient race. Preventive visits appear to have the most effect; additional research should include examining clinical notes for evidence of nutrition care among different populations, which may uncover areas for improving nutrition care for patients with chronic disease.

Abstract Background Data mining of electronic health records (EHRs) has a huge potential for improving clinical decision support and to help healthcare deliver precision medicine. Unfortunately, the rule-based and machine learning-based approaches used for natural language processing (NLP) in healthcare today all struggle with various shortcomings related to performance, efficiency, or transparency. Methods In this paper, we address these issues by presenting a novel method for NLP that implements unsupervised learning of word embeddings, semi-supervised learning for simplified and accelerated clinical vocabulary and concept building, and deterministic rules for fine-grained control of information extraction. The clinical language is automatically learnt, and vocabulary, concepts, and rules supporting a variety of NLP downstream tasks can further be built with only minimal manual feature engineering and tagging required from clinical experts. Together, these steps create an open processing pipeline that gradually refines the data in a transparent way, which greatly improves the interpretable nature of our method. Data transformations are thus made transparent and predictions interpretable, which is imperative for healthcare. The combined method also has other advantages, like potentially being language independent, demanding few domain resources for maintenance, and able to cover misspellings, abbreviations, and acronyms. To test and evaluate the combined method, we have developed a clinical decision support system (CDSS) named Information System for Clinical Concept Searching (ICCS) that implements the method for clinical concept tagging, extraction, and classification. Results In empirical studies the method shows high performance (recall 92.6%, precision 88.8%, F-measure 90.7%), and has demonstrated its value to clinical practice. Here we employ a real-life EHR-derived dataset to evaluate the method’s performance on the task of classification (i.e., detecting patient allergies) against a range of common supervised learning algorithms. The combined method achieves state-of-the-art performance compared to the alternative methods we evaluate. We also perform a qualitative analysis of common word embedding methods on the task of word similarity to examine their potential for supporting automatic feature engineering for clinical NLP tasks. Conclusions Based on the promising results, we suggest more research should be aimed at exploiting the inherent synergies between unsupervised, supervised, and rule-based paradigms for clinical NLP.

Abstract Background The exponential growth of digital healthcare data is fueling the development of Knowledge Discovery in Databases (KDD). Extracting temporal relationships between medical events is essential to reveal hidden patterns that can help physicians find optimal treatments, diagnose illnesses, detect drug adverse reactions, and more. This paper presents an approach for the extraction of patient evolution patterns from electronic health records written in Catalan and/or Spanish. Methods We propose a robust formulation for extracting Temporal Association Rules (TARs) that goes beyond simple rule extraction by considering the sequence of multiple visits. Our highly configurable algorithm leverages this formulation to extract Temporal Association Rules from sequences of medical instances. We can generate rules in the desired format, content, and temporal factors while accounting for different levels of abstraction of medical instances. To demonstrate the effectiveness of our methodology, we applied it to extract patient evolution patterns from clinical histories of multimorbid patients suffering from heart disease and stroke who visited Primary Care Centers (CAP) in Catalonia. Our main objective is to uncover complex rules with multiple temporal steps, that comprise a set of medical instances. Results As we are working with real-world, error-prone data, we propose a process of validation of the results by expert practitioners in primary care. Despite our limited dataset, the high percentage of patterns deemed correct and relevant by the experts is promising. The insights gained from these patterns can inform preventive measures and help detect risk factors, ultimately leading to better treatments and outcomes for patients. Conclusion Our algorithm successfully extracted a set of meaningful and relevant temporal patterns, especially for the specific type of multimorbid patients considered. These patterns were evaluated by experts and demonstrated the ability to predict risk factors that are commonly associated with certain diseases. Moreover, the average time gap between the occurrence of medical events provided critical insight into the term of these risk factors. This information holds significant value in the context of primary healthcare and preventive medicine, highlighting the potential of our method to serve as a valuable medical tool.

Abstract Background In the era of electronic health records (EHR), the ability to share clinical data is a key facilitator of healthcare delivery. Since the introduction of EHRs, this aspect has been extensively studied from the perspective of healthcare providers. Less often explored are the day-to-day challenges surrounding the procurement, deployment, maintenance, and use of interoperable EHR systems, from the perspective of healthcare administrators, such as chief clinical information officers (CCIOs). Objective Our study aims to capture the perceptions of CCIOs on the current state of EHR interoperability in the NHS, its impact on patient safety, the perceived facilitators and barriers to improving EHR interoperability, and what the future of EHR development in the NHS may entail. Methods Semi-structured interviews were conducted between November 2020 – October 2021. Convenience sampling was employed to recruit NHS England CCIOs. Interviews were digitally recorded and transcribed verbatim. A thematic analysis was performed by two independent researchers to identify emerging themes. Results Fifteen CCIOs participated in the study. Participants reported that limited EHR interoperability contributed to the inability to easily access and transfer data into a unified source, thus resulting in data fragmentation. The resulting lack of clarity on patients' health status negatively impacts patient safety through suboptimal care coordination, duplication of efforts, and more defensive practice. Facilitators to improving interoperability included the recognition of the need by clinicians, patient expectations, and the inherent centralised nature of the NHS. Barriers included systems usability difficulties, and institutional, data management, and financial-related challenges. Looking ahead, participants acknowledged that realising that vision across the NHS would require a renewed focus on mandating data standards, user-centred design, greater patient involvement, and encouraging inter-organisational collaboration. Conclusion Tackling poor interoperability will require solutions both at the technical level and in the wider policy context. This will involve demanding interoperability functionalities from the outset in procurement contracts, fostering greater inter-organisation cooperation on implementation strategies, and encouraging systems vendors to prioritise interoperability in their products. Only by comprehensively addressing these challenges would the full potential promised by the use of fully interoperable EHRs be realised.

Abstract Background The risk of mortality in intensive care units (ICUs) is currently addressed by the implementation of scores using admission data. Their performances are satisfactory when complications occur early after admission; however, they may become irrelevant in the case of long hospital stays. In this study, we developed predictive models of short-term mortality in the ICU from longitudinal data. Methods Using data collected throughout patients’ stays of at least 48 h from the MIMIC-III database, several statistical learning approaches were compared, including deep neural networks and penalized regression. Missing data were handled using complete-case analysis or multiple imputation. Results Complete-case analyses from 19 predictors showed good discrimination (AUC > 0.77 for several approaches) to predict death between 12 and 24 h onward, yet excluded 75% of patients from the initial target cohort, as data was missing for some of the predictors. Multiple imputation allowed us to include 70 predictors and keep 95% of patients, with similar performances. Conclusion This proof-of-concept study supports that automated analysis of electronic health records can be of great interest throughout patients’ stays as a surveillance tool. Although this framework relies on a large set of predictors, it is robust to data imputation and may be effective early after admission, when data are still scarce.

Abstract Background Real-world evidence (RWE)—based on information obtained from sources such as electronic health records (EHRs), claims and billing databases, product and disease registries, and personal devices and health applications—is increasingly used to support healthcare decision making. There is variability in the collection of EHR data, which includes “structured data” in predefined fields (e.g., problem list, open claims, medication list, etc.) and “unstructured data” as free text or narrative. Healthcare providers are likely to provide more complete information as free text, but extracting meaning from these fields requires newer technologies and a rigorous methodology to generate higher-quality evidence. Herein, an approach to identify concepts associated with the presence and progression of migraine was developed and validated using the complete patient record in EHR data, including both the structured and unstructured portions. Methods “Traditional RWE” approaches (i.e., capture from structured EHR fields and extraction using structured queries) and “Advanced RWE” approaches (i.e., capture from unstructured EHR data and processing by artificial intelligence [AI] technology, including natural language processing and AI-based inference) were evaluated against a manual chart abstraction reference standard for data collected from a tertiary care setting. The primary endpoint was recall; differences were compared using chi square. Results Compared with manual chart abstraction, recall for migraine and headache were 66.6% and 29.6%, respectively, for Traditional RWE, and 96.8% and 92.9% for Advanced RWE; differences were statistically significant (absolute differences, 30.2% and 63.3%; P

Abstract Introduction Adverse drug events (ADEs) are associated with poor outcomes and increased costs but may be prevented with prediction tools. With the National Institute of Health All of Us (AoU) database, we employed machine learning (ML) to predict selective serotonin reuptake inhibitor (SSRI)-associated bleeding. Methods The AoU program, beginning in 05/2018, continues to recruit ≥ 18 years old individuals across the United States. Participants completed surveys and consented to contribute electronic health record (EHR) for research. Using the EHR, we determined participants who were exposed to SSRIs (citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, sertraline, vortioxetine). Features (n = 88) were selected with clinicians’ input and comprised sociodemographic, lifestyle, comorbidities, and medication use information. We identified bleeding events with validated EHR algorithms and applied logistic regression, decision tree, random forest, and extreme gradient boost to predict bleeding during SSRI exposure. We assessed model performance with area under the receiver operating characteristic curve statistic (AUC) and defined clinically significant features as resulting in > 0.01 decline in AUC after removal from the model, in three of four ML models. Results There were 10,362 participants exposed to SSRIs, with 9.6% experiencing a bleeding event during SSRI exposure. For each SSRI, performance across all four ML models was relatively consistent. AUCs from the best models ranged 0.632–0.698. Clinically significant features included health literacy for escitalopram, and bleeding history and socioeconomic status for all SSRIs. Conclusions We demonstrated feasibility of predicting ADEs using ML. Incorporating genomic features and drug interactions with deep learning models may improve ADE prediction.

Abstract Background Epidemiological research may require linkage of information from multiple organizations. This can bring two problems: (1) the information governance desirability of linkage without sharing direct identifiers, and (2) a requirement to link databases without a common person-unique identifier. Methods We develop a Bayesian matching technique to solve both. We provide an open-source software implementation capable of de-identified probabilistic matching despite discrepancies, via fuzzy representations and complete mismatches, plus de-identified deterministic matching if required. We validate the technique by testing linkage between multiple medical records systems in a UK National Health Service Trust, examining the effects of decision thresholds on linkage accuracy. We report demographic factors associated with correct linkage. Results The system supports dates of birth (DOBs), forenames, surnames, three-state gender, and UK postcodes. Fuzzy representations are supported for all except gender, and there is support for additional transformations, such as accent misrepresentation, variation for multi-part surnames, and name re-ordering. Calculated log odds predicted a proband’s presence in the sample database with an area under the receiver operating curve of 0.997–0.999 for non-self database comparisons. Log odds were converted to a decision via a consideration threshold θ and a leader advantage threshold δ. Defaults were chosen to penalize misidentification 20-fold versus linkage failure. By default, complete DOB mismatches were disallowed for computational efficiency. At these settings, for non-self database comparisons, the mean probability of a proband being correctly declared to be in the sample was 0.965 (range 0.931–0.994), and the misidentification rate was 0.00249 (range 0.00123–0.00429). Correct linkage was positively associated with male gender, Black or mixed ethnicity, and the presence of diagnostic codes for severe mental illnesses or other mental disorders, and negatively associated with birth year, unknown ethnicity, residential area deprivation, and presence of a pseudopostcode (e.g. indicating homelessness). Accuracy rates would be improved further if person-unique identifiers were also used, as supported by the software. Our two largest databases were linked in 44 min via an interpreted programming language. Conclusions Fully de-identified matching with high accuracy is feasible without a person-unique identifier and appropriate software is freely available.

Abstract Background Clinical practices have demonstrated that disease treatment can be very complex. Patients with chronic diseases often suffer from more than one disease. Complex diseases are often treated with a variety of drugs, including both primary and auxiliary treatments. This complexity and multidimensionality increase the difficulty of extracting knowledge from clinical data. Methods In this study, we proposed a subgroup identification algorithm for complex prescriptions (SIAP). We applied the SIAP algorithm to identify the importance level of each drug in complex prescriptions. The algorithm quickly classified and determined valid prescription combinations for patients. The algorithm was validated through classification matching of classical prescriptions in traditional Chinese medicine. We collected 376 formulas and their compositions from a formulary to construct a database of standard prescriptions. We also collected 1438 herbal prescriptions from clinical data for automated prescription identification. The prescriptions were divided into training and test sets. Finally, the parameters of the two sub-algorithms of SIAP and SIAP-All, as well as those of the combination algorithm SIAP + All, were optimized on the training set. A comparison analysis was performed against the baseline intersection set rate (ISR) algorithm. The algorithm for this study was implemented with Python 3.6. Results The SIAP-All and SIAP + All algorithms outperformed the benchmark ISR algorithm in terms of accuracy, recall, and F1 value. The F1 values were 0.7568 for SIAP-All and 0.7799 for SIAP + All, showing improvements of 8.73% and 11.04% over the existing ISR algorithm, respectively. Conclusion We developed an algorithm, SIAP, to automatically match sub-prescriptions of complex drugs with corresponding standard or classic prescriptions. The matching algorithm weights the drugs in the prescription according to their importance level. The results of this study can help to classify and analyse the drug compositions of complex prescriptions.

BACKGROUND: The study aimed to assess the performance of a multidisciplinary-team diabetes care program called DIABETIMSS on glycemic control of type 2 diabetes (T2D) patients, by using available observational patient data and machine-learning-based targeted learning methods. METHODS: We analyzed electronic health records and laboratory databases from the year 2012 to 2016 of T2D patients from six family medicine clinics (FMCs) delivering the DIABETIMSS program, and five FMCs providing routine care. All FMCs belong to the Mexican Institute of Social Security and are in Mexico City and the State of Mexico. The primary outcome was glycemic control. The study covariates included: patient sex, age, anthropometric data, history of glycemic control, diabetic complications and comorbidity. We measured the effects of DIABETIMSS program through 1) simple unadjusted mean differences; 2) adjusted via standard logistic regression and 3) adjusted via targeted machine learning. We treated the data as a serial cross-sectional study, conducted a standard principal components analysis to explore the distribution of covariates among clinics, and performed regression tree on data transformed to use the prediction model to identify patient sub-groups in whom the program was most successful. To explore the robustness of the machine learning approaches, we conducted a set of simulations and the sensitivity analysis with process-of-care indicators as possible confounders. RESULTS: The study included 78,894 T2D patients, from which 37,767patients received care through DIABETIMSS. The impact of DIABETIMSS ranged, among clinics, from 2 to 8% improvement in glycemic control, with an overall (pooled) estimate of 5% improvement. T2D patients with fewer complications have more significant benefit from DIABETIMSS than those with more complications. At the FMCs delivering the conventional model the predicted impacts were like what was observed empirically in the DIABETIMSS clinics. The sensitivity analysis did not change the overall estimate average across clinics. CONCLUSIONS: DIABETIMSS program had a small, but significant increase in glycemic control. The use of machine learning methods yields both population-level effects and pinpoints the sub-groups of patients the program benefits the most. These methods exploit the potential of routine observational patient data within complex healthcare systems to inform decision-makers.

Abstract Background Electronic Health Records (EHR) has been espoused to be an innovation from the paper-based system, with benefits such as fast access to patient information thereby facilitating healthcare provider communication, healthcare continuity and improved quality of care. However, it is the extent of the quality of the electronic health records that determines the access to these stated benefits. The quality of health care records indirectly contributes to patient safety because inaccurate patient data can lead to improper diagnosis and consequently wrong treatment of patients. Most hospitals in Ghana, have recently transitioned into the EHR system, hence, there is a need to assess its accuracy, impact on workflow, staff training on usage, support from the EHR team, and the overall satisfaction of the EHR system. As health leaders are at the frontline of its implementation, their views on the challenges and successes of the EHR system are imperative. Method This qualitative study sought to explore the views of the health leaders on the implemented electronic health records system in nine (9) hospitals within three (3) regions in Ghana. Following ethical approval, GHS-ERC:007/04/21, focus group discussions were conducted with a minimum of 10 hospital leaders in each facility. These included quasi, government and private hospitals. Data was collected between September and November 2021. Results The study found poor quality of records, lack of involvement of frontline clinicians in the development of the EHR system, inadequate training of staff and limited workstations as some of the challenges associated with the use of EHR in hospitals. Health leaders were generally not satisfied with the EHR system. Conclusion It is recommended that addressing inputs from end-users as well as circulating more computers will motivate EHR usage and acceptance. Provision of additional workstations for the various units and involvement of staff in the system development would be most prudent to enable health workers to accept the EHR system in improving the quality of care.

Abstract Background Since no effective therapies exist for Alzheimer’s disease (AD), prevention has become more critical through lifestyle status changes and interventions. Analyzing electronic health records (EHRs) of patients with AD can help us better understand lifestyle’s effect on AD. However, lifestyle information is typically stored in clinical narratives. Thus, the objective of the study was to compare different natural language processing (NLP) models on classifying the lifestyle statuses (e.g., physical activity and excessive diet) from clinical texts in English. Methods Based on the collected concept unique identifiers (CUIs) associated with the lifestyle status, we extracted all related EHRs for patients with AD from the Clinical Data Repository (CDR) of the University of Minnesota (UMN). We automatically generated labels for the training data by using a rule-based NLP algorithm. We conducted weak supervision for pre-trained Bidirectional Encoder Representations from Transformers (BERT) models and three traditional machine learning models as baseline models on the weakly labeled training corpus. These models include the BERT base model, PubMedBERT (abstracts + full text), PubMedBERT (only abstracts), Unified Medical Language System (UMLS) BERT, Bio BERT, Bio-clinical BERT, logistic regression, support vector machine, and random forest. The rule-based model used for weak supervision was tested on the GSC for comparison. We performed two case studies: physical activity and excessive diet, in order to validate the effectiveness of BERT models in classifying lifestyle status for all models were evaluated and compared on the developed Gold Standard Corpus (GSC) on the two case studies. Results The UMLS BERT model achieved the best performance for classifying status of physical activity, with its precision, recall, and F-1 scores of 0.93, 0.93, and 0.92, respectively. Regarding classifying excessive diet, the Bio-clinical BERT model showed the best performance with precision, recall, and F-1 scores of 0.93, 0.93, and 0.93, respectively. Conclusion The proposed approach leveraging weak supervision could significantly increase the sample size, which is required for training the deep learning models. By comparing with the traditional machine learning models, the study also demonstrates the high performance of BERT models for classifying lifestyle status for Alzheimer’s disease in clinical notes.

Abstract Background People live a long time in pre-diabetes/early diabetes without a formal diagnosis or management. Heterogeneity of progression coupled with deficiencies in electronic health records related to incomplete data, discrete events, and irregular event intervals make identification of pre-diabetes and critical points of diabetes progression challenging. Methods We utilized longitudinal electronic health records of 9298 patients with type 2 diabetes or prediabetes from 2005 to 2016 from a large regional healthcare delivery network in China. We optimized a generative Markov-Bayesian-based model to generate 5000 synthetic illness trajectories. The synthetic data were manually reviewed by endocrinologists. Results We build an optimized generative progression model for type 2 diabetes using anchor information to reduce the number of parameters learning in the third layer of the model from $$O\left(N\times W\right)$$ O N × W to $$O\left((N-C)\times W\right)$$ O ( N - C ) × W , where $$N$$ N is the number of clinical findings, $$W$$ W is the number of complications, $$C$$ C is the number of anchors. Based on this model, we infer the relationships between progression stages, the onset of complication categories, and the associated diagnoses during the whole progression of type 2 diabetes using electronic health records. Discussion Our findings indicate that 55.3% of single complications and 31.8% of complication patterns could be predicted early and managed appropriately to potentially delay (as it is a progressive disease) or prevented (by lifestyle modifications that keep patient from developing/triggering diabetes in the first place). Conclusions The full type 2 diabetes patient trajectories generated by the chronic disease progression model can counter a lack of real-world evidence of desired longitudinal timeframe while facilitating population health management.

Abstract Background Fragile X syndrome (FXS), the most common inherited cause of intellectual disability and autism, is significantly underdiagnosed in the general population. Diagnosing FXS is challenging due to the heterogeneity of the condition, subtle physical characteristics at the time of birth and similarity of phenotypes to other conditions. The medical complexity of FXS underscores an urgent need to develop more efficient and effective screening methods to identify individuals with FXS. In this study, we evaluate the effectiveness of using artificial intelligence (AI) and electronic health records (EHRs) to accelerate FXS diagnosis. Methods The EHRs of 2.1 million patients served by the University of Wisconsin Health System (UW Health) were the main data source for this retrospective study. UW Health includes patients from south central Wisconsin, with approximately 33 years (1988–2021) of digitized health data. We identified all participants who received a code for FXS in the form of International Classification of Diseases (ICD), Ninth or Tenth Revision (ICD9 = 759.83, ICD10 = Q99.2). Only individuals who received the FXS code on at least two occasions (“Rule of 2”) were classified as clinically diagnosed cases. To ensure the availability of sufficient data prior to clinical diagnosis to test the model, only individuals who were diagnosed after age 10 were included in the analysis. A supervised random forest classifier was used to create an AI-assisted pre-screening tool to identify cases with FXS, 5 years earlier than the time of clinical diagnosis based on their medical records. The area under receiver operating characteristic curve (AUROC) was reported. The AUROC shows the level of success in identification of cases and controls (AUROC = 1 represents perfect classification). Results 52 individuals were identified as target cases and matched with 5200 controls. AI-assisted pre-screening tool successfully identified cases with FXS, 5 years earlier than the time of clinical diagnosis with an AUROC of 0.717. A separate model trained and tested on UW Health cases achieved the AUROC of 0.798. Conclusions This result shows the potential utility of our tool in accelerating FXS diagnosis in real clinical settings. Earlier diagnosis can lead to more timely intervention and access to services with the goal of improving patients’ health outcomes.

Abstract Introduction State cancer prevention and control programs rely on public health surveillance data to set objectives to improve cancer prevention and control, plan interventions, and evaluate state-level progress towards achieving those objectives. The goal of this project was to evaluate the validity of using electronic health records (EHRs) based on common data model variables to generate indicators for surveillance of cancer prevention and control for these public health programs. Methods Following the methodological guidance from the PRISMA Extension for Scoping Reviews, we conducted a literature scoping review to assess how EHRs are used to inform cancer surveillance. We then developed 26 indicators along the continuum of the cascade of care, including cancer risk factors, immunizations to prevent cancer, cancer screenings, quality of initial care after abnormal screening results, and cancer burden. Indicators were calculated within a sample of patients from the New York City (NYC) INSIGHT Clinical Research Network using common data model EHR data and were weighted to the NYC population using post-stratification. We used prevalence ratios to compare these estimates to estimates from the raw EHR of NYU Langone Health to assess quality of information within INSIGHT, and we compared estimates to results from existing surveillance sources to assess validity. Results Of the 401 identified articles, 15% had a study purpose related to surveillance. Our indicator comparisons found that INSIGHT EHR-based measures for risk factor indicators were similar to estimates from external sources. In contrast, cancer screening and vaccination indicators were substantially underestimated as compared to estimates from external sources. Cancer screenings and vaccinations were often recorded in sections of the EHR that were not captured by the common data model. INSIGHT estimates for many quality-of-care indicators were higher than those calculated using a raw EHR. Conclusion Common data model EHR data can provide rich information for certain indicators related to the cascade of care but may have substantial biases for others that limit their use in informing surveillance efforts for cancer prevention and control programs.

Abstract Background Health providers create Electronic Health Records (EHRs) to describe the conditions and procedures used to treat their patients. Medical notes entered by medical staff in the form of free text are a particularly insightful component of EHRs. There is a great interest in applying machine learning tools on medical notes in numerous medical informatics applications. Learning vector representations, or embeddings, of terms in the notes, is an important pre-processing step in such applications. However, learning good embeddings is challenging because medical notes are rich in specialized terminology, and the number of available EHRs in practical applications is often very small. Methods In this paper, we propose a novel algorithm to learn embeddings of medical terms from a limited set of medical notes. The algorithm, called definition2vec, exploits external information in the form of medical term definitions. It is an extension of a skip-gram algorithm that incorporates textual definitions of medical terms provided by the Unified Medical Language System (UMLS) Metathesaurus. Results To evaluate the proposed approach, we used a publicly available Medical Information Mart for Intensive Care (MIMIC-III) EHR data set. We performed quantitative and qualitative experiments to measure the usefulness of the learned embeddings. The experimental results show that definition2vec keeps the semantically similar medical terms together in the embedding vector space even when they are rare or unobserved in the corpus. We also demonstrate that learned vector embeddings are helpful in downstream medical informatics applications. Conclusion This paper shows that medical term definitions can be helpful when learning embeddings of rare or previously unseen medical terms from a small corpus of specialized documents such as medical notes.