Multi-finger AR Typing Interface for Mobile Devices Using High-Speed Hand Motion Recognition

In this paper, we propose a new user interface designed to improve the usability of mobile devices. The interface overlays a virtual keyboard on real images captured by a camera attached to the back of a mobile device and recognizes user's hand motions by using optical flow information. The user can operate the virtual keyboard with his/her multiple fingers in the space behind the mobile device. The interface recognizes only hand motions and does not recognize poses of a hand or fingers, which enables high-speed stable recognition without complicated processing. The experimental system recognized hand motions with a processing time of about 8.33 [ms/frame] (about 120 [fps]). We implemented a keyboard application using our interface on the experimental system and evaluated the performance. The result of a simple user study showed that typing speed became faster as a trial is repeated.