Search

Millions of Deaf and hard-of-hearing (DHH) people primarily use a sign language for communication, but there is a lack of adequate sign language interpreting to fill these communication needs. Development of automatic sign language translation (ASLT) systems could help translate between a sign language and spoken language in situations where human interpreters are unavailable, and recent advances in large multi-lingual language models may soon enable ASLT to become a reality. Despite the potential for ASLT, Deaf community perspectives on and requirements for such technologies are poorly understood. In this work, we conduct a survey of Deaf community perspectives in the U.S. on ASLT in order to inform the development of ASLT systems that meet user needs and minimize harms. Our results shed light on scenarios where DHH users in the U.S. might want to use ASLT, their performance expectations for ASLT in these scenarios, design preferences for ASLT interfaces, and the benefits and harms they see in the development of ASLT. [ABSTRACT FROM AUTHOR]

The Deaf and Hard-of-hearing (DHH) community faces a lack of information in American Sign Language (ASL) and other signed languages. Most informational resources are text-based (e.g. books, encyclopedias, newspapers, magazines, etc.). Because DHH signers typically prefer ASL and are often less fluent in written English, text is often insufficient. At the same time, there is also a lack of large continuous sign language datasets from representative signers, which are essential to advancing sign langauge research and technology. In this work, we explore the possibility of crowdsourcing English-to-ASL translations to help address these barriers. To do this, we present a novel bilingual interface that enables the community to both contribute and consume translations. To shed light on the user experience with such an interface, we present a user study with 19 participants using the interface to both generate and consume content. To better understand the potential impact of the interface on translation quality, we also present a preliminary translation quality analysis. Our results suggest that DHH community members find real-world value in the interface, that the quality of translations is comparable to those created with state-of-the-art setups, and shed light on future research avenues. [ABSTRACT FROM AUTHOR]

The COVID-19 pandemic forced many people to convert their daily work lives to a "virtual" format where everyone connected remotely from their home. In this new, virtual environment, accessibility barriers changed, in some respects for the better (e.g., more flexibility) and in other aspects, for the worse (e.g., problems including American Sign Language interpreters over video calls). Microsoft Research held its first cohort of all virtual interns in 2020. We the authors, full time and intern members and affiliates of the Ability Team, a research team focused on accessibility, reflect on our virtual work experiences as a team consisting of members with a variety of abilities, positions, and seniority during the summer intern season. Through our autoethnographic method, we provide a nuanced view into the experiences of a mixed-ability, virtual team, and how the virtual setting affected the team's accessibility. We then reflect on these experiences, noting the successful strategies we used to promote access and the areas in which we could have further improved access. Finally, we present guidelines for future virtual mixed-ability teams looking to improve access. [ABSTRACT FROM AUTHOR]

Many Deaf and Hard-of-Hearing (DHH) individuals rely on sign language interpreting to communicate with hearing peers. If on-site interpreting is not available, DHH individuals may use remote interpreting over a smartphone video-call. However, this solution requires the DHH individual to give up either 1) the use of one signing hand by holding the smartphone or 2) their ability to multitask and move around by propping the smartphone up in a fixed location. We explore this problem within the context of the workplace, and present a prototype hands-free device using augmented reality glasses with a hat-mounted fisheye camera and mic/speaker. To explore the validity of our design, we conducted 1) a video interpretability experiment, and 2) a user study with 18 participants (9 DHH, 9 hearing) in a workplace environment. Our results suggest that a hands-free device can support accurate interpretation while enhancing personal interactions. [ABSTRACT FROM AUTHOR]

As machine learning algorithms continue to improve, collecting training data becomes increasingly valuable. At the same time, increased focus on data collection may introduce compounding privacy concerns. Accessibility projects in particular may put vulnerable populations at risk, as disability status is sensitive, and collecting data from small populations limits anonymity. To help address privacy concerns while maintaining algorithmic performance on machine learning tasks, we propose privacy-enhancing distortions of training datasets. We explore this idea through the lens of sign language video collection, which is crucial for advancing sign language recognition and translation. We present a web study exploring signers' concerns in contributing to video corpora and their attitudes about using filters, and a computer vision experiment exploring sign language recognition performance with filtered data. Our results suggest that privacy concerns may exist in contributing to sign language corpora, that filters (especially expressive avatars and blurred faces) may impact willingness to participate, and that training on more filtered data may boost recognition accuracy in some cases. [ABSTRACT FROM AUTHOR]

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of fields, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the field? and 3) What are the calls to action for people working in the field? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community. [ABSTRACT FROM AUTHOR]

Sign languages lack a standard written form, preventing millions of Deaf people from accessing text in their primary language. A major barrier to adoption is difficulty learning a system which represents complex 3D movements with stationary symbols. In this work, we leverage the animation capabilities of modern screens to create the first animated character system prototype for sign language, producing text that combines iconic symbols and movement. Using animation to represent sign movements can increase resemblance to the live language, making the character system easier to learn. We explore this idea through the lens of American Sign Language (ASL), presenting 1) a pilot study underscoring the potential value of an animated ASL character system, 2) a structured approach for designing animations for an existing ASL character system, and 3) a design probe workshop with ASL users eliciting guidelines for the animated character system design. [ABSTRACT FROM AUTHOR]

Sounds provide informative signals about the world around us. In situations where non-auditory cues are inaccessible, it can be useful for deaf and hard-of-hearing people to be notified about sounds. Through a survey, we explored which sounds are of interest to deaf and hard-of-hearing people, and which means of notification are appropriate. Motivated by these findings, we designed a mobile phone app that alerts deaf and hard-of-hearing people to sounds they care about. The app uses training examples of personally relevant sounds recorded by the user to learn a model of those sounds. It then screens the incoming audio stream from the phone's microphone for those sounds. When it detects a sound, it alerts the user by vibrating and providing a pop-up notification. To evaluate the interface design independent of sound detection errors, we ran a Wizard-of-Oz user study, and found that the app design successfully facilitated deaf and hard-of-hearing users recording training examples. We also explored the viability of a basic machine learning algorithm for sound detection. [ABSTRACT FROM AUTHOR]