Human pose tracking using environmental features as functional surface constraints

Proposed in this work is a method for real-time 3D human pose tracking using depth video with the ability to learn and utilize functional environmental features from the scene. Learning environmental features is done by pose classifications of the tracked trajectory for recovering pose-to-surface interaction points, resulting in surfaces represented by floating planes, either bounded or unbounded. In the tracking method surfaces can be utilized for inferring likely poses for interaction with the surfaces, as well as be utilized as hard physical constraints for reducing freedom of interpretation in the tracker.The method employed for tracking is a top-down approach, with the only significant learning criterion being a library of prefabricated body poses in this implementation inhabited by 15 variants distributed into three categories.The method used for inferring likely poses of interaction with the environment is created explicitly for use in this work, and includes a nearness measure along with a 1NN classification using the tracked state and the defined pose library.The method used for pose classifications for learning surfaces is a 1NN approach using the tracked state and the defined pose library to gather interaction points, to which an orthogonal least squares plane fitting method is applied.For evaluation of the method a dataset consisting of two sequences was created, with synchronized depth- and motion capture data, containing challenging poses. Additionally two other data sources were included, also having synchronized depth- and motion capture data.The results suggest an achievable benefit in tracking robustness from knowing the environment, as well as an achievable method for learning specific parts of this environment during tracking. However, not every type of environmental features could be feasibly detected and used.