Search

Handling missing data in clinical prognostic studies is an essential yet challenging task. This study aimed to provide a comprehensive assessment of the effectiveness and reliability of different machine learning (ML) imputation methods across various analytical perspectives. Specifically, it focused on three distinct classes of performance metrics used to evaluate ML imputation methods: post-imputation bias of regression estimates, post-imputation predictive accuracy, and substantive model-free metrics. As an illustration, we applied data from a real-world breast cancer survival study. This comprehensive approach aimed to provide a thorough assessment of the effectiveness and reliability of ML imputation methods across various analytical perspectives. A simulated dataset with 30% Missing At Random (MAR) values was used. A number of single imputation (SI) methods - specifically KNN, missMDA, CART, missForest, missRanger, missCforest - and multiple imputation (MI) methods - specifically miceCART and miceRF - were evaluated. The performance metrics used were Gower's distance, estimation bias, empirical standard error, coverage rate, length of confidence interval, predictive accuracy, proportion of falsely classified (PFC), normalized root mean squared error (NRMSE), AUC, and C-index scores. The analysis revealed that in terms of Gower's distance, CART and missForest were the most accurate, while missMDA and CART excelled for binary covariates; missForest and miceCART were superior for continuous covariates. When assessing bias and accuracy in regression estimates, miceCART and miceRF exhibited the least bias. Overall, the various imputation methods demonstrated greater efficiency than complete-case analysis (CCA), with MICE methods providing optimal confidence interval coverage. In terms of predictive accuracy for Cox models, missMDA and missForest had superior AUC and C-index scores. Despite offering better predictive accuracy, the study found that SI methods introduced more bias into the regression coefficients compared to MI methods. This study underlines the importance of selecting appropriate imputation methods based on study goals and data types in time-to-event research. The varying effectiveness of methods across the different performance metrics studied highlights the value of using advanced machine learning algorithms within a multiple imputation framework to enhance research integrity and the robustness of findings. [ABSTRACT FROM AUTHOR]

Objectives: COVID-19 epidemics often lead to elevated levels of depression. To accurately identify and predict depression levels in home-quarantined individuals during a COVID-19 epidemic, this study constructed a depression prediction model based on multiple machine learning algorithms and validated its effectiveness. Methods: A cross-sectional method was used to examine the depression status of individuals quarantined at home during the epidemic via the network. Characteristics included variables on sociodemographics, COVID-19 and its prevention and control measures, impact on life, work, health and economy after the city was sealed off, and PHQ-9 scale scores. The home-quarantined subjects were randomly divided into training set and validation set according to the ratio of 7:3, and the performance of different machine learning models were compared by 10-fold cross-validation, and the model algorithm with the best performance was selected from 15 models to construct and validate the depression prediction model for home-quarantined subjects. The validity of different models was compared based on accuracy, precision, receiver operating characteristic (ROC) curve, and area under the ROC curve (AUC), and the best model suitable for the data framework of this study was identified. Results: The prevalence of depression among home-quarantined individuals during the epidemic was 31.66% (202/638), and the constructed Adaboost depression prediction model had an ACC of 0.7917, an accuracy of 0.7180, and an AUC of 0.7803, which was better than the other 15 models on the combination of various performance measures. In the validation sets, the AUC was greater than 0.83. Conclusions: The Adaboost machine learning algorithm developed in this study can be used to construct a depression prediction model for home-quarantined individuals that has better machine learning performance, as well as high effectiveness, robustness, and generalizability. [ABSTRACT FROM AUTHOR]

Background: Variation in laboratory healthcare data due to seasonal changes is a widely accepted phenomenon. Seasonal variation is generally not systematically accounted for in healthcare settings. This study applies a newly developed adjustment method for seasonal variation to analyze the effect seasonality has on machine learning model classification of diagnoses. Methods: Machine learning methods were trained and tested on ~ 22 million unique records from ~ 575,000 unique patients admitted to Danish hospitals. Four machine learning models (adaBoost, decision tree, neural net, and random forest) classifying 35 diseases of the circulatory system (ICD-10 diagnosis codes, chapter IX) were run before and after seasonal adjustment of 23 laboratory reference intervals (RIs). The effect of the adjustment was benchmarked via its contribution to machine learning models trained using hyperparameter optimization and assessed quantitatively using performance metrics (AUROC and AUPRC). Results: Seasonally adjusted RIs significantly improved cardiovascular disease classification in 24 of the 35 tested cases when using neural net models. Features with the highest average feature importance (via SHAP explainability) across all disease models were sex, C- reactive protein, and estimated glomerular filtration. Classification of diseases of the vessels, such as thrombotic diseases and other atherosclerotic diseases consistently improved after seasonal adjustment. Conclusions: As data volumes increase and data-driven methods are becoming more advanced, it is essential to improve data quality at the pre-processing level. This study presents a method that makes it feasible to introduce seasonally adjusted RIs into the clinical research space in any disease domain. Seasonally adjusted RIs generally improve diagnoses classification and thus, ought to be considered and adjusted for in clinical decision support methods. [ABSTRACT FROM AUTHOR]

Background: Metabolic pathway prediction is one possible approach to address the problem in system biology of reconstructing an organism's metabolic network from its genome sequence. Recently there have been developments in machine learning-based pathway prediction methods that conclude that machine learning-based approaches are similar in performance to the most used method, PathoLogic which is a rule-based method. One issue is that previous studies evaluated PathoLogic without taxonomic pruning which decreases its performance. Results: In this study, we update the evaluation results from previous studies to demonstrate that PathoLogic with taxonomic pruning outperforms previous machine learning-based approaches and that further improvements in performance need to be made for them to be competitive. Furthermore, we introduce mlXGPR, a XGBoost-based metabolic pathway prediction method based on the multi-label classification pathway prediction framework introduced from mlLGPR. We also improve on this multi-label framework by utilizing correlations between labels using classifier chains. We propose a ranking method that determines the order of the chain so that lower performing classifiers are placed later in the chain to utilize the correlations between labels more. We evaluate mlXGPR with and without classifier chains on single-organism and multi-organism benchmarks. Our results indicate that mlXGPR outperform other previous pathway prediction methods including PathoLogic with taxonomic pruning in terms of hamming loss, precision and F1 score on single organism benchmarks. Conclusions: The results from our study indicate that the performance of machine learning-based pathway prediction methods can be substantially improved and can even outperform PathoLogic with taxonomic pruning. [ABSTRACT FROM AUTHOR]

Purpose: With the in-depth application of machine learning(ML) in clinical practice, it has been used to predict the mortality risk in patients with traumatic brain injuries(TBI). However, there are disputes over its predictive accuracy. Therefore, we implemented this systematic review and meta-analysis, to explore the predictive value of ML for TBI. Methodology: We systematically retrieved literature published in PubMed, Embase.com, Cochrane, and Web of Science as of November 27, 2022. The prediction model risk of bias(ROB) assessment tool (PROBAST) was used to assess the ROB of models and the applicability of reviewed questions. The random-effects model was adopted for the meta-analysis of the C-index and accuracy of ML models, and a bivariate mixed-effects model for the meta-analysis of the sensitivity and specificity. Result: A total of 47 papers were eligible, including 156 model, with 122 newly developed ML models and 34 clinically recommended mature tools. There were 98 ML models predicting the in-hospital mortality in patients with TBI; the pooled C-index, sensitivity, and specificity were 0.86 (95% CI: 0.84, 0.87), 0.79 (95% CI: 0.75, 0.82), and 0.89 (95% CI: 0.86, 0.92), respectively. There were 24 ML models predicting the out-of-hospital mortality; the pooled C-index, sensitivity, and specificity were 0.83 (95% CI: 0.81, 0.85), 0.74 (95% CI: 0.67, 0.81), and 0.75 (95% CI: 0.66, 0.82), respectively. According to multivariate analysis, GCS score, age, CT classification, pupil size/light reflex, glucose, and systolic blood pressure (SBP) exerted the greatest impact on the model performance. Conclusion: According to the systematic review and meta-analysis, ML models are relatively accurate in predicting the mortality of TBI. A single model often outperforms traditional scoring tools, but the pooled accuracy of models is close to that of traditional scoring tools. The key factors related to model performance include the accepted clinical variables of TBI and the use of CT imaging. [ABSTRACT FROM AUTHOR]

Background: Accurately predicting the risk of atherosclerotic cardiovascular disease (ASCVD) is crucial for implementing individualized prevention strategies and improving patient outcomes. Our objective is to develop machine learning (ML)-based models for predicting ASCVD risk in a prospective Chinese population and compare their performance with conventional regression models. Methods: A hybrid dataset consisting of 551 features was used, including 98 demographic, behavioral, and psychological features, 444 Electrocardiograph (ECG) features, and 9 Echocardiography (Echo) features. Seven machine learning (ML)-based models were trained, validated, and tested after selecting the 30 most informative features. We compared the discrimination, calibration, net benefit, and net reclassification improvement (NRI) of the ML models with those of conventional ASCVD risk calculators, such as the Pooled Cohort Equations (PCE) and Prediction for ASCVD Risk in China (China-PAR). Results: The study included 9,609 participants (mean age 53.4 ± 10.4 years, 53.7% female), and during a median follow-up of 4.7 years, 431 (4.5%) participants developed ASCVD. In the testing set, the final ML-based ANN model outperformed PCE, China-PAR, recalibrated PCE, and recalibrated China-PAR in predicting ASCVD. This was demonstrated by the model's higher area under the curve (AUC) of 0.800, compared to 0.777, 0.780, 0.779, and 0.779 for the other models, respectively. Additionally, the model had a lower Hosmer–Lemeshow χ2 of 9.1, compared to 37.3, 67.6, 126.6, and 18.6 for the other models. The net benefit at a threshold of 5% was also higher for the ML-based ANN model at 0.017, compared to 0.016, 0.013, 0.017, and 0.016 for the other models, respectively. Furthermore, the NRI was 0.089 for the ML-based ANN model, while it was 0.355, 0.098, and 0.088 for PCE, China-PAR, and recalibrated PCE, respectively. Conclusions: Compared to conventional regression ASCVD risk calculators, such as PCE and China-PAR, the ANN prediction model may help optimize identification of individuals at heightened cardiovascular risk by flexibly incorporating a wider range of potential predictors. The findings may help guide clinical decision-making and ultimately contribute to ASCVD prevention and management. [ABSTRACT FROM AUTHOR]

Background: Machine learning (ML) methods to build prediction models starting from electrocardiogram (ECG) signals are an emerging research field. The aim of the present study is to investigate the performances of two ML approaches based on ECGs for the prediction of new-onset atrial fibrillation (AF), in terms of discrimination, calibration and sample size dependence. Methods: We trained two models to predict new-onset AF: a convolutional neural network (CNN), that takes as input the raw ECG signals, and an eXtreme Gradient Boosting model (XGB), that uses the signal's extracted features. A penalized logistic regression model (LR) was used as a benchmark. Discrimination was evaluated with the area under the ROC curve, while calibration with the integrated calibration index. We investigated the dependence of models' performances on the sample size and on class imbalance corrections introduced with random under-sampling. Results: CNN's discrimination was the most affected by the sample size, outperforming XGB and LR only around n = 10.000 observations. Calibration showed only a small dependence on the sample size for all the models considered. Balancing the training set with random undersampling did not improve discrimination in any of the models. Instead, the main effect of imbalance corrections was to worsen the models' calibration (for CNN, integrated calibration index from 0.014 [0.01, 0.018] to 0.17 [0.16, 0.19]). The sample size emerged as a fundamental point for developing the CNN model, especially in terms of discrimination (AUC = 0.75 [0.73, 0.77] when n = 10.000, AUC = 0.80 [0.79, 0.81] when n = 150.000). The effect of the sample size on the other two models was weaker. Imbalance corrections led to poorly calibrated models, for all the approaches considered, reducing the clinical utility of the models. Conclusions: Our results suggest that the choice of approach in the analysis of ECG should be based on the amount of data available, preferring more standard models for small datasets. Moreover, imbalance correction methods should be avoided when developing clinical prediction models, where calibration is crucial. [ABSTRACT FROM AUTHOR]

Background: Machine-learning models are susceptible to external influences which can result in performance deterioration. The aim of our study was to elucidate the impact of a sudden shift in covariates, like the one caused by the Covid-19 pandemic, on model performance. Methods: After ethical approval and registration in Clinical Trials (NCT04092933, initial release 17/09/2019), we developed different models for the prediction of perioperative mortality based on preoperative data: one for the pre-pandemic data period until March 2020, one including data before the pandemic and from the first wave until May 2020, and one that covers the complete period before and during the pandemic until October 2021. We applied XGBoost as well as a Deep Learning neural network (DL). Performance metrics of each model during the different pandemic phases were determined, and XGBoost models were analysed for changes in feature importance. Results: XGBoost and DL provided similar performance on the pre-pandemic data with respect to area under receiver operating characteristic (AUROC, 0.951 vs. 0.942) and area under precision-recall curve (AUPR, 0.144 vs. 0.187). Validation in patient cohorts of the different pandemic waves showed high fluctuations in performance from both AUROC and AUPR for DL, whereas the XGBoost models seemed more stable. Change in variable frequencies with onset of the pandemic were visible in age, ASA score, and the higher proportion of emergency operations, among others. Age consistently showed the highest information gain. Models based on pre-pandemic data performed worse during the first pandemic wave (AUROC 0.914 for XGBoost and DL) whereas models augmented with data from the first wave lacked performance after the first wave (AUROC 0.907 for XGBoost and 0.747 for DL). The deterioration was also visible in AUPR, which worsened by over 50% in both XGBoost and DL in the first phase after re-training. Conclusions: A sudden shift in data impacts model performance. Re-training the model with updated data may cause degradation in predictive accuracy if the changes are only transient. Too early re-training should therefore be avoided, and close model surveillance is necessary. [ABSTRACT FROM AUTHOR]

Objectives: Automatic speech and language assessment methods (SLAMs) can help clinicians assess speech and language impairments associated with dementia in older adults. The basis of any automatic SLAMs is a machine learning (ML) classifier that is trained on participants' speech and language. However, language tasks, recording media, and modalities impact the performance of ML classifiers. Thus, this research has focused on evaluating the effects of the above-mentioned factors on the performance of ML classifiers that can be used for dementia assessment. Methodology: Our methodology includes the following steps: (1) Collecting speech and language datasets from patients and healthy controls; (2) Using feature engineering methods which include feature extraction methods to extract linguistic and acoustic features and feature selection methods to select most informative features; (3) Training different ML classifiers; and (4) Evaluating the performance of ML classifiers to investigate the impacts of language tasks, recording media, and modalities on dementia assessment. Results: Our results show that (1) the ML classifiers trained with the picture description language task perform better than the classifiers trained with the story recall language task; (2) the data obtained from phone-based recordings improves the performance of ML classifiers compared to data obtained from web-based recordings; and (3) the ML classifiers trained with acoustic features perform better than the classifiers trained with linguistic features. Conclusion: This research demonstrates that we can improve the performance of automatic SLAMs as dementia assessment methods if we: (1) Use the picture description task to obtain participants' speech; (2) Collect participants' voices via phone-based recordings; and (3) Train ML classifiers using only acoustic features. Our proposed methodology will help future researchers to investigate the impacts of different factors on the performance of ML classifiers for assessing dementia. [ABSTRACT FROM AUTHOR]

Aims: In the present study, we aimed to evaluate the performance of machine learning (ML) models for identification of acute myocardial infarction (AMI) or death within 30 days among emergency department (ED) chest pain patients. Methods and results: Using data from 9519 consecutive ED chest pain patients, we created ML models based on logistic regression or artificial neural networks. Model inputs included sex, age, ECG and the first blood tests at patient presentation: High sensitivity TnT (hs-cTnT), glucose, creatinine, and hemoglobin. For a safe rule-out, the models were adapted to achieve a sensitivity > 99% and a negative predictive value (NPV) > 99.5% for 30-day AMI/death. For rule-in, we set the models to achieve a specificity > 90% and a positive predictive value (PPV) of > 70%. The models were also compared with the 0 h arm of the European Society of Cardiology algorithm (ESC 0 h); An initial hs-cTnT < 5 ng/L for rule-out and ≥ 52 ng/L for rule-in. A convolutional neural network was the best model and identified 55% of the patients for rule-out and 5.3% for rule-in, while maintaining the required sensitivity, specificity, NPV and PPV levels. ESC 0 h failed to reach these performance levels. Discussion: An ML model based on age, sex, ECG and blood tests at ED arrival can identify six out of ten chest pain patients for safe early rule-out or rule-in with no need for serial blood tests. Future studies should attempt to improve these ML models further, e.g. by including additional input data. [ABSTRACT FROM AUTHOR]

Background: Clinical diagnosis based on machine learning usually uses case samples as training samples, and uses machine learning to construct disease prediction models characterized by descriptive texts of clinical manifestations. However, the problem of sample imbalance often exists in the medical field, which leads to a decrease in classification performance of the machine learning. Methods: To solve the problem of sample imbalance in medical dataset, we propose a hybrid sampling algorithm combining synthetic minority over-sampling technique (SMOTE) and edited nearest neighbor (ENN). Firstly, the SMOTE is used to over-sampling missed abortion and diabetes datasets, so that the number of samples of the two classes is balanced. Then, ENN is used to under-sampling the over-sampled dataset to delete the "noisy sample" in the majority. Finally, Random forest is used to model and predict the sampled missed abortion and diabetes datasets to achieve an accurate clinical diagnosis. Results: Experimental results show that Random forest has the best classification performance on missed abortion and diabetes datasets after SMOTE-ENN sampled, and the MCC index is 95.6% and 90.0%, respectively. In addition, the results of pairwise comparison and multiple comparisons show that the SMOTE-ENN is significantly better than other sampling algorithms. Conclusion: Random forest has significantly improved all indexes on the missed abortion dataset after SMOTE-ENN sampled. [ABSTRACT FROM AUTHOR]

Background: Despite the immense importance of transmembrane proteins (TMP) for molecular biology and medicine, experimental 3D structures for TMPs remain about 4–5 times underrepresented compared to non-TMPs. Today's top methods such as AlphaFold2 accurately predict 3D structures for many TMPs, but annotating transmembrane regions remains a limiting step for proteome-wide predictions. Results: Here, we present TMbed, a novel method inputting embeddings from protein Language Models (pLMs, here ProtT5), to predict for each residue one of four classes: transmembrane helix (TMH), transmembrane strand (TMB), signal peptide, or other. TMbed completes predictions for entire proteomes within hours on a single consumer-grade desktop machine at performance levels similar or better than methods, which are using evolutionary information from multiple sequence alignments (MSAs) of protein families. On the per-protein level, TMbed correctly identified 94 ± 8% of the beta barrel TMPs (53 of 57) and 98 ± 1% of the alpha helical TMPs (557 of 571) in a non-redundant data set, at false positive rates well below 1% (erred on 30 of 5654 non-membrane proteins). On the per-segment level, TMbed correctly placed, on average, 9 of 10 transmembrane segments within five residues of the experimental observation. Our method can handle sequences of up to 4200 residues on standard graphics cards used in desktop PCs (e.g., NVIDIA GeForce RTX 3060). Conclusions: Based on embeddings from pLMs and two novel filters (Gaussian and Viterbi), TMbed predicts alpha helical and beta barrel TMPs at least as accurately as any other method but at lower false positive rates. Given the few false positives and its outstanding speed, TMbed might be ideal to sieve through millions of 3D structures soon to be predicted, e.g., by AlphaFold2. [ABSTRACT FROM AUTHOR]

Objectives: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. Materials and Methods: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. Results: The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. Discussion: Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. Conclusion: Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application. [ABSTRACT FROM AUTHOR]

Background: Balance training exercise games (exergames) are a promising tool for reducing fall risk in elderly. Exergames can be used for in-home guided exercise, which greatly increases availability and facilitates independence. Providing biofeedback on weight-shifting during in-home balance exercise improves exercise efficiency, but suitable equipment for measuring weight-shifting is lacking. Exergames often use kinematic data as input for game control. Being able to useg such data to estimate weight-shifting would be a great advantage. Machine learning (ML) models have been shown to perform well in weight-shifting estimation in other settings. Therefore, the aim of this study was to investigate the performance of ML models in estimation of weight-shifting during exergaming using kinematic data.Methods: Twelve healthy older adults (mean age 72 (± 4.2), 10 F) played a custom exergame that required repeated weight-shifts. Full-body 3D motion capture (3DMoCap) data and standard 2D digital video (2D-DV) was recorded. Weight shifting was directly measured by 3D ground reaction forces (GRF) from force plates, and estimated using a linear regression model, a long-short term memory (LSTM) model and a decision tree model (XGBoost). Performance was evaluated using coefficient of determination ([Formula: see text]) and root mean square error (RMSE).Results: Results from estimation of GRF components using 3DMoCap data show a mean (± 1SD) RMSE (% total body weight, BW) of the vertical GRF component ([Formula: see text]) of 4.3 (2.5), 11.1 (4.5), and 11.0 (4.7) for LSTM, XGBoost and LinReg, respectively. Using 2D-DV data, LSTM and XGBoost achieve mean RMSE (± 1SD) in [Formula: see text] estimation of 10.7 (9.0) %BW and 19.8 (6.4) %BW, respectively. [Formula: see text] was [Formula: see text] for the LSTM in the [Formula: see text] component using 3DMoCap data, and [Formula: see text] using 2D-DV data. For XGBoost, [Formula: see text] [Formula: see text] was [Formula: see text] using 3DMoCap data, and [Formula: see text] using 2D-DV data.Conclusion: This study demonstrates that an LSTM model can estimate 3-dimensional GRF components using 2D kinematic data extracted from standard 2D digital video cameras. The [Formula: see text] component is estimated more accurately than [Formula: see text] and [Formula: see text] components, especially when using 2D-DV data. Weight-shifting performance during exergaming can thus be extracted using kinematic data only, which can enable effective independent in-home balance exergaming. [ABSTRACT FROM AUTHOR]