A Proof-of-Concept Study on Scoring Oral Presentation Videos in Higher Education. Research Report. ETS RR-19-22

This proof-of-concept study examined the feasibility of a new scoring procedure designed to reduce the time of scoring a video-based public speaking assessment task. Instead of scoring the video in its entirety, the performance was evaluated based on content-related (e.g., speech organization, word choice) and delivery-related (e.g., vocal expression, nonverbal behaviors) dimensions. Content-related dimensions were scored based on speech transcripts, while delivery dimensions were scored using a video thin-slicing technique, where scores were assigned based on brief vignettes of a video rather than the complete performance. Initial feasibility data were collected from 4 novice raters scoring 10 video performances. Results indicated that for delivery dimensions, four 10 second thin slices yielded interrater consistency reliability somewhat similar to that of full-video scoring, and additional slices resulted in only small improvements. For transcription-based scoring, while raters were consistent, their scores had low correlations with criterion scores, which was likely due to differences in the modality (video vs. text). Video thin-slicing appears to be a promising scoring technique for relevant constructs. Further testing of a combination of audio and transcript is recommended for scoring content-related constructs.