Supporting Older Adults in Locating Mobile Interface Features with Voice Input

As mobile applications continue to offer more features, tackling the complexity of mobile interfaces can become challenging for older adults. Owing to a small screen and frequent updates that modify the visual layouts of menus and buttons, older adults can find it challenging to locate a function on a mobile interface quickly—even when familiar with the application. To address this issue, we present a system that supports older adults to quickly locate an on-screen feature on a mobile interface using speech queries. Our system allows users to ask for a function related to the current mobile screen using voice input. When that function is available, it provides visual guidance for users to engage with the pertinent user interface (UI) widget. The label and location of all UI components on the current screen are acquired via the Android’s Assist API. We discuss four scenarios of use.