Evaluating and Enhancing the Generalization Performance of Machine Learning Models for Physical Activity Intensity Prediction From Raw Acceleration Data

Purpose: To evaluate and enhance the generalization performance of machine learning physical activity intensity prediction models developed with raw acceleration data on populations monitored by different activity monitors. Method: Five datasets from four studies, each containing only hip- or wrist-based raw acceleration data (two hip- and three wrist-based) were extracted. The five datasets were then used to develop and validate artificial neural networks (ANN) in three setups to classify activity intensity categories (sedentary behavior, light, and moderate-to-vigorous). To examine generalizability, the ANN models were developed using within-dataset (leave-one-subject-out) cross-validation, and then cross-tested to other datasets with different accelerometers. To enhance the models’ generalizability, a combination of four of the five datasets was used for training and the fifth dataset for validation. Finally, all the five datasets were merged to develop a single model that is generalizable across the datasets (50% of the subjects from each dataset for training, the remaining for validation). Results: The datasets showed high performance in within-dataset cross-validation (accuracy 71.9–95.4%, Kappa K=0.63–0.94). The performance of the within-dataset validated models decreased when applied to datasets with different accelerometers (41.2–59.9%, K=0.21–0.48). The trained models on merged datasets consisting hip and wrist data predicted the left-out dataset with acceptable performance (65.9–83.7%, K=0.61–0.79). The model trained with all five datasets performed with acceptable performance across the datasets (80.4–90.7%, K=0.68–0.89). Conclusions: Integrating heterogeneous datasets in training sets seems a viable approach for enhancing the generalization performance of the models. Instead, within-dataset validation is not sufficient to understand the models’ performance on other populations with different accelerometers.