Search

Pulmonary embolism (PE) is a life-threatening condition associated with ~10% of deaths of hospitalized patients. Machine learning algorithms (MLAs) which predict the onset of pulmonary embolism (PE) could enable earlier treatment and improve patient outcomes. However, the extent to which they generalize to broader patient populations impacts their clinical utility.To conduct the first large-scale external validation of a machine learning-based PE prediction model which uses EHR data from the first three hours of a patient's hospital stay to predict the occurrence of PE within the next 10 days of the inpatient stay.This retrospective study included approximately two million adult hospital admissions across 44 medical institutions in the US from 2011 to 2017. Demographics, vital signs, and lab tests from adult inpatients at 12 institutions (n = 331,268; 3.3% PE positive) were used for training an XGBoost model. External validation of the model was conducted on patient populations from each of 32 medical institutions (total n = 1,660,715; 3.7% PE positive) without retraining. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC). Backward elimination regression was used to identify correlations between characteristics of the external validation sets and AUROC.The model performed well (AUROC = 0.87) on the 20% hold-out subset of the training set. Despite demographic differences between the 32 external validation populations (percent PE positive: min = 1.54%, max = 6.47%), without retraining, the model had excellent discrimination, with a mean AUROC of 0.88 (min = 0.79, max = 0.93). Fixing sensitivity at 0.80, the model had a mean specificity of 0.85 (min = 0.64, max = 0.93). Backward elimination regression identified a negative association (β = -0.015, p 0.001) between the percentage of PE positive encounters and AUROC.A PE prediction model performed remarkably well across 32 different external patient populations without retraining and despite significant differences in demographic characteristics, demonstrating its generalizability and potential as a clinical decision support tool to aid PE detection and improve patient outcomes in a clinical setting.

Importance: Despite sex and race disparities in the symptom presentation, diagnosis, and management of acute coronary syndrome (ACS), these differences have not been investigated in the development and validation of machine learning (ML) models using individualized patient information from electronic health records (EHRs) to diagnose ACS. Objective: To evaluate ML-based ACS diagnosis performance across different subpopulations in a multi-site emergency department (ED) setting and determine how bias mitigating techniques influence ML performance. Design, Setting, and Participants: This retrospective observational study included data from 2,334,316 ED patients ( >18 years) from January 2007 to June 2020. Exposure: Logistic regression (LR) and neural network (NN) models were assessed in ED encounters grouped by sex, race, presence or absence of chest pain, EHR data quality, and timeliness of several key ED procedures. Prejudice regularization, reweighting, and within-subpopulation training were evaluated for bias mitigation. Main Outcomes/Measures: Metrics including area under the receiver operating characteristic (AUROC) were used to assess performances. Results: We analyzed 4,268,165 ED visits in which patient demographics by race were 67.40% White, 19.20% Black, 2.40% Asian, and 11.00% Other or Unknown. Patient composition was 54.80% female and 45.20% male. Both models’ AUROCs were significantly higher in White vs. Black patients (LR: z-score = 3.23 and NN: 4.26 for NN; P < 0.0006), in males vs. females (z-score = 3.81 for LR and 4.16 for NN; P < 0.0001) and in no chest pain subpopulation vs. chest pain (z-score = 13.32 for LR and 17.70 for NN; P < 0.0001). Prejudice regularization and reweighting techniques did not reduce biases. Training in race-specific and sex-specific training populations also did not yeild statistically signficant improvements in ML algorithm performance. Chest pain-specific training led to significantly improved AUROC.Conclusion: EHR-derived ML models trained and tested within similar demographic subpopulations and symptom groups may perform better than ML models that are trained in random populations, and provide less biased clinical decision support for ACS diagnosis.

(1) Background: Ventilator-associated pneumonia (VAP) causes high mortality among patients with respiratory disease and imposes major burdens on healthcare infrastructure. Models that use electronic health record data to predict the onset of VAP may spur earlier treatment and improve patient outcomes. We developed and studied the performance of interpretable machine learning (ML) models that predict the onset of VAP from electronic health records (EHRs); (2) Methods: We trained Logistic Regression (LR), full feature Explainable Boosting Machine (fEBM), and eXtreme Gradient Boosting (XGBoost) ML models on data from the MIMIC- III (v1.3) database. Model performance was measured by area under the receiver operating characteristic curves (AUCs). We trained a minimal-feature EBM model (mEBM) with features derived from white blood cell (WBC) counts, duration of ventilation, and Glasgow Coma Scale (GCS). Finally, model robustness was evaluated on randomly sparsified EHR datasets; (3) Results: The fEBM model outperformed the XGBoost and LR models at 24 hours post-intubation. The mEBM model maintained an AUC of 0.893. The fEBM model performance remained robust on sparsified datasets; (4) Conclusions: Our novel interpretable ML algorithm reliably predicts the onset of VAP in intubated patients. Integration of this EBM-based model into clinical practice may enable clinicians to better anticipate and prevent VAP.

Background Electronic health records (EHRs) contain individualized patient data that can be used to develop diagnostic and risk prediction models with artificial intelligence (AI) algorithms. Explicit and implicit sources of bias embedded in EHRs may hinder generalizable model performance and perpetuate bias. Objective This study explores the question of how existing sex and racial disparities in the clinical assessment and management of acute coronary syndrome (ACS) are reflected in patients’ EHRs. We outlined recommendations on how these sources of bias should be examined and mitigated within the framework of development and validation of AI-based diagnosis and risk stratification algorithms. Methods This retrospective study examined several previously unrecognized EHR-embedded biases in a multisite emergency departments (ED) setting. We assessed sex and race differences in EHR data missingness and timeliness of several key ED procedures following ACS suspicion. We additionally conducted a data-driven clustering analysis to detect latent groups of ACS patients. Results 10,043 ACS-associated ED visits were included. We identified sex and race differences in the prevalence of ACS symptoms, data missingness (vitals and labs), and waiting time for troponin order and treatment initiation within 24 hours post-admission. Our cluster analysis discovered four groups of ACS patients corresponding to common symptoms. Additionally, we identified differences in clinical management and patient demographics across these clusters. Conclusions We discovered several sources of bias inherent in the clinical practice of ACS diagnosis and treatment that are also present in EHR data. Our study supports the inclusion of a wider range of symptoms into AI-based diagnosis and risk stratification tools. These models should be also validated in ED admissions without chest pain and in patients who experience a longer waiting time for ED procedures.

BackgroundData drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals.MethodsWe devise a series of simulations that measure the effects of data drift in patients with sepsis. We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN).ResultsOur results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8.ConclusionOur simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.

During embryogenesis, the initial vascular network forms by the process of vasculogenesis, or the specification of vascular progenitors de novo. In contrast, the majority of later-forming vessels arise by angiogenesis from the already established vasculature. Here, we show that new vascular progenitors in zebrafish embryos emerge from a distinct site along the yolk extension, or secondary vascular field (SVF), incorporate into the posterior cardinal vein, and contribute to subintestinal vasculature even after blood circulation has been initiated. We further demonstrate that SVF cells participate in vascular recovery after chemical ablation of vascular endothelial cells. Inducible inhibition of the function of vascular progenitor marker etv2/etsrp prevented SVF cell differentiation and resulted in the defective formation of subintestinal vasculature. Similar late-forming etv2+ progenitors were also observed in mouse embryos, suggesting that SVF cells are evolutionarily conserved. Our results characterize a distinct mechanism by which new vascular progenitors incorporate into established vasculature.

BACKGROUND Acute respiratory distress syndrome (ARDS) is a condition that is often considered to have broad and subjective diagnostic criteria and is associated with significant mortality and morbidity. Early and accurate prediction of ARDS and related conditions such as hypoxemia and sepsis could allow timely administration of therapies, leading to improved patient outcomes. OBJECTIVE The aim of this study is to perform an exploration of how multilabel classification in the clinical setting can take advantage of the underlying dependencies between ARDS and related conditions to improve early prediction of ARDS in patients. METHODS The electronic health record data set included 40,703 patient encounters from 7 hospitals from April 20, 2018, to March 17, 2021. A recurrent neural network (RNN) was trained using data from 5 hospitals, and external validation was conducted on data from 2 hospitals. In addition to ARDS, 12 target labels for related conditions such as sepsis, hypoxemia, and COVID-19 were used to train the model to classify a total of 13 outputs. As a comparator, XGBoost models were developed for each of the 13 target labels. Model performance was assessed using the area under the receiver operating characteristic curve. Heat maps to visualize attention scores were generated to provide interpretability to the neural networks. Finally, cluster analysis was performed to identify potential phenotypic subgroups of patients with ARDS. RESULTS The single RNN model trained to classify 13 outputs outperformed the individual XGBoost models for ARDS prediction, achieving an area under the receiver operating characteristic curve of 0.842 on the external test sets. Models trained on an increasing number of tasks resulted in improved performance. Earlier prediction of ARDS nearly doubled the rate of in-hospital survival. Cluster analysis revealed distinct ARDS subgroups, some of which had similar mortality rates but different clinical presentations. CONCLUSIONS The RNN model presented in this paper can be used as an early warning system to stratify patients who are at risk of developing one of the multiple risk outcomes, hence providing practitioners with the means to take early action.

During embryonic development, cells differentiate into a variety of distinct cell types and subtypes with diverse transcriptional profiles. To date, transcriptomic signatures of different cell lineages that arise during development have been only partially characterized. Here we used single-cell RNA-seq to perform transcriptomic analysis of over 20,000 cells disaggregated from the trunk region of zebrafish embryos at the 30 hpf stage. Transcriptional signatures of 27 different cell types and subtypes were identified and annotated during this analysis. This dataset will be a useful resource for many researchers in the fields of developmental and cellular biology and facilitate the understanding of molecular mechanisms that regulate cell lineage choices during development.

ETS transcription factors play an important role in the specification and differentiation of endothelial cells during vascular development. Despite previous studies, the role of the founding member of the ETS family, Ets1, in vascular development in vivo is only partially understood. Here, we generated a zebrafish ets1 mutant by TALEN genome editing and tested functional redundancy between Ets1 and a related ETS factor Etv2 / Etsrp / ER71. While zebrafish ets1(−/−) mutants have a normal functional vascular system, etv2(−/−);ets1(−/) embryos had more severe angiogenic defects and lower expression levels of kdr and kdrl, the two zebrafish homologs of the mammalian Vascular Endothelial Growth Factor Receptor 2 VEGFR2/Flk1, than etv2(−/−)embryos. Expression of constitutively active Mitogen-Activated Protein Kinase1 (MAP2K1) within endothelial cells partially rescued this angiogenic defect. Interestingly, ets1(−/−) embryos displayed extensive apoptosis within the trunk vasculature despite exhibiting normal vascular patterning. Loss of Ets1 combined with a partial knockdown of Etv2 function resulted in a decrease in endothelial cell numbers in the axial vasculature, which argues for a role of Ets1 in promoting vasculogenesis. We also demonstrate that although both Ets1 and Etv2 can induce ectopic vascular marker expression in zebrafish embryos, Ets1 activity is dependent on MAPK-mediated phosphorylation of its Thr30 and Ser33 residues, while Etv2 activity is not. Together, our results identify a novel function of Ets1 in regulating endothelial cell survival during vasculogenesis in vivo. Based on these findings, we propose a revised model of how Ets1 and Etv2 play unique and partially redundant roles to promote vascular development.

Vasculogenesis involves the differentiation of vascular endothelial progenitors de novo from undifferentiated mesoderm, their migration and coalescence to form the major embryonic vessels and the acquisition of arterial or venous identity. Vascular Endothelial Growth Factor (Vegf) signaling plays multiple roles during vascular development. However, its function during embryonic vasculogenesis has been controversial. Previous studies have implicated Vegf signaling in either regulating arteriovenous specification or overall vascular endothelial differentiation. To clarify the role of Vegf in embryonic vasculogenesis and identify its downstream targets, we used chemical inhibitors of Vegf receptor (Vegfr) signaling in zebrafish embryos as well as zebrafish genetic mutants. A high level of chemical inhibition of Vegfr signaling resulted in the reduction of overall vascular endothelial marker gene expression, including downregulation of both arterial and venous markers, ultimately leading to the apoptosis of vascular endothelial cells. In contrast, a low level of Vegfr inhibition specifically blocked arterial specification while the expression of venous markers appeared largely unaffected or increased. Inhibition of Vegfr signaling prior to the initiation of vasculogenesis reduced overall vascular endothelial differentiation, while inhibition of Vegfr signaling starting at mid-somitogenesis stages largely inhibited arterial specification. Conversely, Vegf overexpression resulted in the expansion of both arterial and pan-endothelial markers, while the expression of several venous-specific markers was downregulated. We further show that Vegf signaling affects overall endothelial differentiation by modulating the expression of the ETS transcription factor etv2/ etsrp. etv2 expression was downregulated in Vegfr- inhibited embryos, and expanded in Vegfaa-overexpressing embryos. Furthermore, vascular-specific overexpression of etv2 in Vegfr-inhibited embryos rescued defects in vascular endothelial differentiation. Similarly, vegfaa genetic mutants displayed a combination of the two phenotypes observed with chemical Vegfr inhibition: the expression of arterial and pan-endothelial markers including etv2 was downregulated while the expression of most venous markers was either expanded or unchanged. Based on these results we propose a revised model which explains the different phenotypes observed upon inhibition of Vegf signaling: low levels of Vegf signaling promote overall vascular endothelial differentiation and cell survival by upregulating etv2 expression, while high levels of Vegf signaling promote arterial and inhibit venous specification.

Cell fate decisions involved in vascular and hematopoietic embryonic development are still poorly understood. An ETS transcription factor Etv2 functions as an evolutionarily conserved master regulator of vasculogenesis. Here we report a single-cell transcriptomic analysis of hematovascular development in wild-type and etv2 mutant zebrafish embryos. Distinct transcriptional signatures of different types of hematopoietic and vascular progenitors are identified using an etv2ci32Gt gene trap line, in which the Gal4 transcriptional activator is integrated into the etv2 gene locus. We observe a cell population with a skeletal muscle signature in etv2-deficient embryos. We demonstrate that multiple etv2ci32Gt; UAS:GFP cells differentiate as skeletal muscle cells instead of contributing to vasculature in etv2-deficient embryos. Wnt and FGF signaling promote the differentiation of these putative multipotent etv2 progenitor cells into skeletal muscle cells. We conclude that etv2 actively represses muscle differentiation in vascular progenitors, thus restricting these cells to a vascular endothelial fate., The signals restricting specification of vascular progenitors are unclear. Here, the authors use scRNAseq to identify transitional steps during blood and vascular development in zebrafish and identify Etv2 as repressing skeletal muscle differentiation in vascular progenitors.

The vascular system is one of the earliest functional organ systems to develop in a vertebrate embryo. The major blood vessels form by the mechanism of vasculogenesis which involves differentiation of mesodermal cells into vascular endothelial progenitors. Here we overview the current knowledge on the cellular and molecular mechanisms that regulate vasculogenesis, including specification of vascular progenitors, their migration and colaescence into primordial vessels, arteriovenous differentiation, and lumen formation.