Tracking Stuffed Toy for Naturally Mapped Interactive Play via a Soft-Pose Estimator

Have you ever picked up a stuffed toy and pretended to play with it in your childhood? We are motivated by the novel use of stuffed toys in enhancing extended reality interaction. A key goal of extended reality is to induce the feeling of presence in its users. Naturally mapped control interface has been shown to enhance presence. The literature also indicates that a high degree of freedom tracking is important to extended reality. Based on these observations, we show that a free-form naturally mapped control interface is well-motivated via a theoretical contextualization. We explore the possibility of building such a controller in the form of stuffed toys. To realize stuffed toys as controllers, a novel soft-pose estimator empowered by cage-based deformation is proposed. It is shown to be effective in tracking the poses and deformations of real soft objects even by training with synthetic data only. Three gameplay prototypes are developed to demonstrate that interactive play can be enabled by the soft-pose estimator. They also form the basis for two user studies that validate the success of tracking stuffed toys with the soft-pose estimator for interactive play.