Do passive cross-modal validation processes occur when processing multimedia materials?

In text comprehension research, a passive validation mechanism has been observed that checks the consistency between incoming and previous text information (or prior knowledge). In two pre-registered online studies, we investigated whether a passive cross-modal validation mechanism occurs during the processing of multimedia materials (i.e., text combined with pictures). Participants (Experiment 1: N = 146; Experiment 2: N = 235) were recruited via Prolific. We used the epistemic Stroop paradigm (Richter et al., 2009), which makes use of the fact that the passive validation mechanism induces positive or negative response tendencies that can interfere with the processing of an unrelated task if it requires an opposite response. Participants received either matching (valid) or mismatching (invalid) text-picture stimuli. Following each stimulus, participants performed an unrelated probe-word task reacting to the probe words "wrong" or "right". The dependent variables were reaction time and error rates in the unrelated probe-word task. Experiment 1 used one-sentence-picture stimuli, while Experiment 2 used longer text-segments-picture stimuli. Linear mixed-effects models showed interactions of validity and probe word for reaction times (Experiments 1 & 2) and error rates (Experiment 1). Post-hoc comparisons indicated prolonged reaction times or higher error rates when the probe word task required a response opposite to the outcome of the validation process. This study is the first to demonstrate that a passive cross-modal validation mechanism checks the consistency between written text and accompanying pictures. We discuss the implications of these findings for understanding information processing in multimedia contexts. • During processing multimedia passive cross-modal validation mechanisms occur. • The mechanism checks whether incoming text and picture information is consistent. • The mechanism occurs with one-sentence-picture and longer-text-picture stimuli. [ABSTRACT FROM AUTHOR]