Modeling Closed Captioning Subjective Quality Assessment by Deaf and Hard of Hearing Viewers

Closed Captioning (CC) is a service primarily designed for deaf and hard of hearing (D/HoH) viewers. The CC translates spoken speech into text for television or film screen display. The quality assessment methods for live captioning are limited to quantitative measures, while the viewers are still dissatisfied with the current quality. One method to improve the current quality assessment procedure is to include D/HoH viewers in the evaluation procedure for their subjective assessment input. However, it could be costly and impractical to perform evaluations for the entire broadcasted shows. Therefore, it would be helpful to model subjective assessments that could replicate and predict human decisions. In this article, we report on a model of probabilities of D/HoH viewer assessment decisions for CC quality factors based on actual user preferences. An online survey was designed and conducted to collect assessment data for 22 error variation samples from four quality factors: delay, speed, missing words, and paraphrasing of captions. The results are analyzed using the signal detection theory framework to create decision probability models for D/HoH viewers.