Factors Affecting Judgment Accuracy When Scoring Children's Responses to Non-Word Repetition Stimuli in Real Time

Background: Non-word repetition (NWR) tests are an important way speech and language therapists (SaLTs) assess language development. NWR tests are often scored whilst participants make their responses (i.e., in real time) in clinical and research reports (documented here via a secondary analysis of a published systematic review). Aims: The main aim was to determine the extent to which real-time coding of NWR stimuli at the whole-item level (as correct/incorrect) was predicted by models that had varying levels of detail provided from phonemic transcriptions using several linear mixed method (LMM) models. Methods & Procedures: Live scores and recordings of responses on the universal non-word repetition (UNWR) test were available for 146 children aged between 3 and 6 years where the sample included all children starting in five UK schools in one year or two consecutive years. Transcriptions were made of responses to two-syllable NWR stimuli for all children and these were checked for reliability within and between transcribers. Signal detection analysis showed that consonants were missed when judgments were made live. Statistical comparisons of the discrepancies between target stimuli and transcriptions of children's responses were then made and these were regressed against live score accuracy. Six LMM models (three normalized: 1a, 2a, 3a; and three non-normalized: 1b, 2b, 3b) were examined to identify which model(s) best captured the data variance. Errors on consonants for live scores were determined by comparison with the transcriptions in the following ways (the dependent variables for each pair of models): (1) consonants alone; (2) substitutions, deletions and insertions of consonants identified after automatic alignment of live and transcribed materials; and (3) as with (2) but where substitutions were coded further as place, manner and voicing errors. Outcomes & Results: The normalized model that coded consonants in non-words as 'incorrect' at the level of substitutions, deletions and insertions (2b) provided the best fit to the real-time coding responses in terms of marginal R[superscript 2], Akaike's information criterion (AIC) and Bayesian information criterion (BIC) statistics. Conclusions & Implications: Errors that occur on consonants when non-word stimuli are scored in real time are characterized solely by the substitution, deletion and insertion measure. It is important to know that such errors arise when real-time judgments are made because NWR tasks are used to assess and diagnose several cognitive-linguistic impairments. One broader implication of the results is that future work could automate the analysis procedures to provide the required information objectively and quickly without having to transcribe data.