Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors.

• Activity recognition methods can achieve 96% accuracy on the SHL Challenge dataset. • End-to-end DL methods perform as good as the classic ML methods. • Combining the ML and DL methods in an ensemble improves the results. • Correcting the predictions using a Hidden Markov Model is highly beneficial. • An energy-efficient solution achieves 90% accuracy using only 5% of the data. The Sussex-Huawei Locomotion-Transportation Recognition Challenge presented a unique opportunity to the activity-recognition community to test their approaches on a large, real-life benchmark dataset with activities different from those typically recognized. The goal of the challenge was to recognize, as accurately as possible, eight locomotion activities (Still, Walk, Run, Bike, Car, Bus, Train, Subway) using smartphone sensor data. This paper describes the method we developed to win this challenge, and provides an analysis of the effectiveness of its components. We used complex feature extraction and selection methods to train classical machine learning models. In addition, we trained deep learning models using a novel end-to-end architecture for deep multimodal spectro-temporal fusion. All the models were fused into an ensemble with the final predictions smoothed by a hidden Markov model to account for temporal dependencies of the activities. The presented method achieved an F1 score of 94.9% on the challenge test data. We tested different sampling frequencies, window sizes, feature types, classification models and the importance of stand-alone sensors and their fusion for the task. Finally, we present an energy-efficient smartphone implementation of the method. [ABSTRACT FROM AUTHOR]