Search

The negative affective priming (NAP) task is a behavioral measure of inhibition of emotional stimuli. Previous studies using the NAP task have found that individuals with depression show reduced inhibition of negative stimuli, suggesting that inhibition biases may play a role in the etiology and maintenance of depression. However, the psychometric properties of the NAP task have not been evaluated or reported. In the present study, we report data on the association between NAP task performance and depression symptoms in three independent samples, and we evaluate the internal consistency and test-retest reliability of the NAP effect indices. The NAP effect for both negative and positive target words had poor internal consistency in all three samples, as well as poor 2-week (Study 2) and 6-month (Study 3) test-retest reliability. The internal consistency and test-retest reliability of response times (RT) for the individual trial types were moderate to high, as were the intercorrelations between trial types. This pattern of results indicates that overall RT is reliable but variance in RTs for the different trial types in the NAP task is indistinguishable from variance in overall RT. Depression symptom severity was not associated with the NAP effect for negative or positive target words in any of the samples, which could be due to the poor reliability of the NAP effect. Based on these findings, we do not recommend that researchers use the NAP task as a measure of individual differences in the inhibition of emotional stimuli., (© 2023. The Psychonomic Society, Inc.)

Objective: To examine the sensitivity of split-sample reliability estimates to the random split of the data and propose alternative methods for improving the stability of the split-sample method., Data Sources and Study Setting: Data were simulated to reflect a variety of real-world quality measure distributions and scenarios. There is no date range to report as the data are simulated., Study Design: Simulation studies of split-sample reliability estimation were conducted under varying practical scenarios., Data Collection/extraction Methods: All data were simulated using functions in R., Principal Findings: Single split-sample reliability estimates can be very dependent on the random split of the data, especially in low sample size and low variability settings. Averaging split-sample estimates over many splits of the data can yield a more stable reliability estimate., Conclusions: Measure developers and evaluators using the split-sample reliability method should average a series of reliability estimates calculated from many resamples of the data without replacement to obtain a more stable reliability estimate., (Published 2024. This article is a U.S. Government work and is in the public domain in the USA.)

Introduction Many psychometric studies have scrutinized the dependability of different instruments for evaluating and treating autism using applied behavior analysis (ABA). However, there has been no exploration into the psychometric attributes of the Catalyst Datafinch Applied Behavior Analysis Data Collection Application, namely, internal consistency reliability measures. Materials and methods  Four datasets were extracted (n=100, 98, 103, and 62) from published studies at The Oxford Center, Brighton, MI, ranging from March 19, 2023, through January 8, 2024, using Catalyst Datafinch as the data collection tool. All data were gathered by Board Certified Behavior Analysts (BCBAs) and behavioral technicians and designed to replicate how practitioners collect traditional paper and pencil data. SPSS Statistics (v. 29.0) computed internal consistency reliability measures, including Cronbach's alpha, inter-item, split-half, and interclass correlation coefficients. Results  Dataset #1: Cronbach's alpha was 0.916 with seven items, indicating excellent reliability. Cronbach's split-half reliability for Part 1 was 0.777, indicating good reliability, and for Part 2 was 0.972, indicating excellent reliability. Guttman split-half coefficient was 0.817, indicating good reliability. Inter-item correlation coefficients ranged from 0.474 to 0.970. The average measures interclass correlation (ICC) was 0.916, indicating excellent reliability. Single measures (ICC) reliability was 0.609, indicating acceptable reliability. Dataset #2: Cronbach's alpha was 0.954 with three items, indicating excellent reliability. Cronbach's split-half reliability for Part 1 was 0.912, indicating excellent reliability, and for Part 2 was 0.975, indicating excellent reliability. Guttman split-half coefficient was 0.917, indicating excellent reliability. Inter-item correlation coefficients ranged from 0.827 to 0.977. Average measures (ICC) was 0.954, indicating excellent reliability. Single measures (ICC) reliability was 0.875, indicating good reliability. Dataset #3: Cronbach's alpha was 0.974 with three items, indicating excellent reliability. Cronbach's split-half reliability for Part 1 was 0.978, indicating excellent reliability. Split-half reliability for Part 2 was 0.970, indicating excellent reliability. Guttman split-half coefficient was 0.935, indicating excellent reliability. Inter-item correlation coefficients ranged from 0.931 to 0.972. The average measures (ICC) was 0.974, indicating excellent reliability. Single measures (ICC) reliability was 0.926, indicating excellent reliability. Dataset #4: Cronbach's alpha was 0.980 with 12 items, indicating excellent reliability. Cronbach's split-half reliability for Part 1 was 0.973, indicating excellent reliability. Split-half reliability for Part 2 was 0.996, indicating excellent reliability. Guttman split-half coefficient was 0.838, indicating good reliability. Inter-item correlation coefficients ranged from 0.692 to 0.999. The average measures (ICC) was 0.980, indicating excellent reliability. Single measures (ICC) reliability was 0.804, indicating good reliability. Conclusions These results suggest that Catalyst Datafinch demonstrates high internal consistency reliability when used with individuals with autism. This indicates that the application is reliable for collecting and analyzing behavioral data in this population. The ratings ranged from good to excellent, indicating a high consistency in the measurements., Competing Interests: The authors have declared that no competing interests exist., (Copyright © 2024, Peterson et al.)

Executive function is vital for normal social, cognitive, and motor functions. Executive function decline due to aging increases the risk of disability and falls in older adults, which has become an urgent public health issue. Fast and convenient neuropsychological tools are thus needed to identify high-risk groups as early as possible to conduct a timely intervention. Card sorting tasks, such as Wisconsin Card Sorting Task (WCST) and its variants, are popular tools for measuring executive function. This study investigated the reliability of an open-source, self-administered, online, short-version card sorting task with a sample of young (n = 107, 65 females, age: M = 30.1 years, SD = 5.5 years) and elderly Chinese (n = 113, 53 females, age: M = 64.0 years, SD = 6.7 years). We developed an automated scoring and visualization procedure following the recent recommendations on scoring perseverative responses to make the results comparable to the standardized WCST. Reliability estimates of commonly used measures were calculated using the split-half method. All task indices' reliabilities were reasonably good in both old and young groups except for "failure-to-maintain-set." Elderly Chinese adults showed compromised task performance on all measures compared with the young Chinese adults at the group level. The R script of automated scoring and estimation of reliability is publicly available., (© 2023. The Psychonomic Society, Inc.)

Self-administered computerized assessment has the potential to increase the reach of neuropsychological assessment. The present study reports the first split-half reliability estimates for a self-administered computerized variant of the Wisconsin Card Sorting Test (WCST), which is considered as a gold standard for the neuropsychological assessment of executive functions. We analyzed data from a large sample of young volunteers ( N = 375). Split-half reliability estimates for perseveration errors, set-loss errors, and inference errors were all above 0.90. Split-half reliability estimates for response time measures on switch and repeat trials exceeded 0.95. Our results indicated sufficient split-half reliability for a self-administered computerized WCST, paving the way for an advanced digital assessment of executive functions. We discuss potential effects of test formats, administration variants, and sample characteristics on split-half reliability.

The Wisconsin Card Sorting Test (WCST) represents the gold standard for the neuropsychological assessment of executive function. However, very little is known about its reliability. In the current study, 146 neurological inpatients received the Modified WCST (M-WCST). Four basic measures (number of correct sorts, categories, perseverative errors, set-loss errors) and their composites were evaluated for split-half reliability. The reliability estimates of the number of correct sorts, categories, and perseverative errors fell into the desirable range ( rel ≥ .90). The study therefore disclosed sufficiently reliable M-WCST measures, fostering the application of this eminent psychological test to neuropsychological assessment. Our data also revealed that the M-WCST possesses substantially better psychometric properties than would be expected from previous studies of WCST test-retest reliabilities obtained from non-patient samples. Our study of split-half reliabilities from discretionary construed and from randomly built M-WCST splits exemplifies a novel approach to the psychometric foundation of neuropsychology.

Background: In major depressive disorder (MDD), self-blame-related fMRI measures have shown the potential to be used as prognostic markers for recurrence risk. Like most potential fMRI markers, however, their reliability is unclear. Here, we probed the internal reliability of self-blame-related fMRI measures, as well as the impact of different modelling approaches on reliability metrics and validity., Methods: Internal consistency (i.e. split-half reliability) was calculated for blood oxygen level-dependent (BOLD) responses and psychophysiological interactions (PPI) related to self-blame-related biases in medication-free remitted MDD participants (n = 81) and healthy controls (n = 41). Trial-length was modelled using three durations (0, 2 and 5 s), which was convolved with the haemodynamic response function (HRF) with and without time and dispersion derivatives. Intraclass correlation coefficients (ICCs) were calculated for simple contrasts examining activation to self-blaming emotions and other-blaming emotions and the more complex contrast of the subtraction-based difference between self- and other-blaming emotions within the following a priori ROIs: right superior anterior temporal lobe seed region, anterior subgenual cingulate cortex, posterior subgenual cortex and right striatum / pallidum., Results: Across ROIs, we obtained fair reliability (ICC ≥ 0.40) for simple, but poor reliability (ICC < 0.40) for more complex fMRI measures related to self-blame. Despite this low internal consistency of complex measures at the individual level, we observed robust activation at the group-level, reproducing previously published results., Conclusions: While simple BOLD contrasts had fair reliability, previously employed PPI models had poor reliability and simple connectivity measures lacked predictive validity. This calls for the development of functional connectivity measures that strike a better balance between reliability and validity for future clinical applications, which require robust measures at the individual rather than group-level., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Many researchers have tried to predict semantic priming effects using a myriad of variables (e.g., prime-target associative strength or co-occurrence frequency). The idea is that relatedness varies across prime-target pairs, which should be reflected in the size of the priming effect (e.g., cat should prime dog more than animal does). However, it is only insightful to predict item-level priming effects if they can be measured reliably. Thus, in the present study we examined the split-half and test-retest reliabilities of item-level priming effects under conditions that should discourage the use of strategies. The resulting priming effects proved extremely unreliable, and reanalyses of three published priming datasets revealed similar cases of low reliability. These results imply that previous attempts to predict semantic priming were unlikely to be successful. However, one study with an unusually large sample size yielded more favorable reliability estimates, suggesting that big data, in terms of items and participants, should be the future for semantic priming research.

Reliabilities of scores for experimental tasks are likely to differ from one study to another to the extent that the task stimuli change, the number of trials varies, the type of individuals taking the task changes, the administration conditions are altered, or the focal task variable differs. Given that reliabilities vary as a function of the design of these tasks and the characteristics of the individuals taking them, making inferences about the reliability of scores in an ongoing study based on reliability estimates from prior studies is precarious. Thus, it would be advantageous to estimate reliability based on data from the ongoing study. We argue that internal consistency estimates of reliability are underutilized for experimental task data and in many applications could provide this information using a single administration of a task. We discuss different methods for computing internal consistency estimates with a generalized coefficient alpha and the conditions under which these estimates are accurate. We illustrate use of these coefficients using data for three different tasks.