Search

ABSTRACT Background Objective Methods Results Conclusions Accurate classification is essential for addressing childhood movement disorders (MD), but the common coexistence of multiple MDs complicates this process.The aim was to assess inter‐rater agreement on classifying hyperkinetic MDs among pediatric neurologists with expertise in MDs.Five pediatric neurologists were requested to examine 112 videos of 66 pediatric patients. Based on the Movement Disorder‐Childhood Rating Scale, 3 queries were posed: (Q1) Is there more than 1 MD? (Q2) What is the (predominant) MD? (Q3) What is the other MD (if present)?The final agreement rates were 57.5% for Q1, 66.6% for Q2, and 43.9% for absolute agreement. All videos with absolute agreement at the first evaluation featured 1 MD, whereas only 2 videos with multiple MDs could totally agree in the final review.This study reveals significant discordance in classification even among pediatric neurologists with expertise in MDs and highlights the necessity for a standardized approach. [ABSTRACT FROM AUTHOR]

Aim: In Germany, the CTC-EFF (Effectiveness of the Community Prevention System Communities That Care) study, a replication study of the US Community Youth Development Study (CYDS), was rolled out in 2021. This article aims to examine the validity of a translated and adapted version of an instrument that measures five constructs of community capacity for prevention. Subject and methods: Twenty-two a priori-matched intervention and comparison communities participated in the study. Two to ten community key informants (n = 182) were interviewed for each community. Exploratory (EFA) and confirmatory factor analyses (CFA) were conducted to assess structural validity. Reliability was assessed at the individual level using McDonald's omega and at the community level using a generalizability coefficient (GC). Inter-rater agreement was measured using an intraclass correlation coefficient (ICC). Concurrent construct validity was assessed using bivariate correlations and mean comparisons. Results: CFA showed good model fit (CFI = 0.964) for the hypothesized five constructs model. Reliability analyses showed good internal consistency at the individual level (omega = 0.86–0.87) and insufficient to moderate reliability at the community level (CG = 0.00–0.62). Inter-rater agreement ranged from insufficient to substantial (ICC = 0.02–0.32). Consistent with theory, three constructs correlated positively at the individual and community level (r = 0.24–0.42). There are no associations between these constructs and interviewer ratings about the respondents' cooperativeness, trustworthiness, and understanding. Conclusion: Three of the five constructs were measured with good validity. Two constructs show insufficient reliability. These will be revised until the next CTC-EFF data collection wave. Further validation is indicated. Trial registration: This study was registered with the German Clinical Trial Register, no. DRKS00022819, on August 18, 2021. [ABSTRACT FROM AUTHOR]

We previously developed a computer-assisted image analysis algorithm to detect and quantify the microscopic features of rodent progressive cardiomyopathy (PCM) in rat heart histologic sections and validated the results with a panel of five veterinary toxicologic pathologists using a multinomial logistic model. In this study, we assessed both the inter-rater and intra-rater agreement of the pathologists and compared pathologists' ratings to the artificial intelligence (AI)-predicted scores. Pathologists and the AI algorithm were presented with 500 slides of rodent heart. They quantified the amount of cardiomyopathy in each slide. A total of 200 of these slides were novel to this study, whereas 100 slides were intentionally selected for repetition from the previous study. After a washout period of more than six months, the repeated slides were examined to assess intra-rater agreement among pathologists. We found the intra-rater agreement to be substantial, with weighted Cohen's kappa values ranging from k = 0.64 to 0.80. Intra-rater variability is not a concern for the deterministic AI. The inter-rater agreement across pathologists was moderate (Cohen's kappa k = 0.56). These results demonstrate the utility of AI algorithms as a tool for pathologists to increase sensitivity and specificity for the histopathologic assessment of the heart in toxicology studies. [ABSTRACT FROM AUTHOR]

Aims Single-lead electrocardiograms (ECGs) can be recorded using widely available devices such as smartwatches and handheld ECG recorders. Such devices have been approved for atrial fibrillation (AF) detection. However, little evidence exists on the reliability of single-lead ECG interpretation. We aimed to assess the level of agreement on detection of AF by independent cardiologists interpreting single-lead ECGs and to identify factors influencing agreement. Methods and results In a population-based AF screening study, adults aged ≥65 years old recorded four single-lead ECGs per day for 1–4 weeks using a handheld ECG recorder. Electrocardiograms showing signs of possible AF were identified by a nurse, aided by an automated algorithm. These were reviewed by two independent cardiologists who assigned participant- and ECG-level diagnoses. Inter-rater reliability of AF diagnosis was calculated using linear weighted Cohen's kappa (⁠ κ w ⁠). Out of 2141 participants and 162 515 ECGs, only 1843 ECGs from 185 participants were reviewed by both cardiologists. Agreement was moderate: κ w = 0.48 (95% confidence interval, 0.37–0.58) at participant level and κ w = 0.58 (0.53–0.62) at ECG level. At participant level, agreement was associated with the number of adequate-quality ECGs recorded, with higher agreement in participants who recorded at least 67 adequate-quality ECGs. At ECG level, agreement was associated with ECG quality and whether ECGs exhibited algorithm-identified possible AF. Conclusion Inter-rater reliability of AF diagnosis from single-lead ECGs was found to be moderate in older adults. Strategies to improve reliability might include participant and cardiologist training and designing AF detection programmes to obtain sufficient ECGs for reliable diagnoses. [ABSTRACT FROM AUTHOR]

For decades, texts on methods in animal behavior research have stressed the need for observers of behavior to work to minimize potential unconscious biases in their coding of data. Two major ways of minimizing these biases are to carry out data coding blind to the key comparisons being made in the study and to have high inter‐observer reliability. Over 10 years ago, Burghardt et al. (2012, Ethology, 118, 511) reviewed five major journals in the field of animal behavior and coded randomly selected articles from five decadal volumes (1970 to 2010). That earlier article found poor rates of reporting these two common methods for minimizing potential biases. Here, we carried out similar coding for the 2020 volumes from those same five journals. We found that rates of reporting have increased in all five journals – some substantially. However, rates of reporting still lag behind the journal Infancy, which publishes research on human infant development and relies on many of the same behavioral observation and coding methods used by animal behavior researchers. Given increased calls for transparency and reproducibility in many different fields of scientific study, we argue that we – researchers, reviewers, and editors – can and need to do better at making sure we are actively conducting research in ways to minimize potential observer biases. [ABSTRACT FROM AUTHOR]

Background: Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts' quality and the quality of the peer review process. The goal of this pilot study was to evaluate structured peer review implementation by: 1) exploring whether and how reviewers answered structured peer review questions, 2) analysing reviewer agreement, 3) comparing that agreement to agreement before implementation of structured peer review, and 4) further enhancing the piloted set of structured peer review questions. Methods: Structured peer review consisting of nine questions was piloted in August 2022 in 220 Elsevier journals. We randomly selected 10% of these journals across all fields and IF quartiles and included manuscripts that received two review reports in the first 2 months of the pilot, leaving us with 107 manuscripts belonging to 23 journals. Eight questions had open-ended fields, while the ninth question (on language editing) had only a yes/no option. The reviews could also leave Comments-to-Author and Comments-to-Editor. Answers were independently analysed by two raters, using qualitative methods. Results: Almost all the reviewers (n = 196, 92%) provided answers to all questions even though these questions were not mandatory in the system. The longest answer (Md 27 words, IQR 11 to 68) was for reporting methods with sufficient details for replicability or reproducibility. The reviewers had the highest (partial) agreement (of 72%) for assessing the flow and structure of the manuscript, and the lowest (of 53%) for assessing whether interpretation of the results was supported by data, and for assessing whether the statistical analyses were appropriate and reported in sufficient detail (52%). Two thirds of the reviewers (n = 145, 68%) filled out the Comments-to-Author section, of which 105 (49%) resembled traditional peer review reports. These reports contained a Md of 4 (IQR 3 to 5) topics covered by the structured questions. Absolute agreement regarding final recommendations (exact match of recommendation choice) was 41%, which was higher than what those journals had in the period from 2019 to 2021 (31% agreement, P = 0.0275). Conclusions: Our preliminary results indicate that reviewers successfully adapted to the new review format, and that they covered more topics than in their traditional reports. Individual question analysis indicated the greatest disagreement regarding the interpretation of the results and the conducting and the reporting of statistical analyses. While structured peer review did lead to improvement in reviewer final recommendation agreements, this was not a randomized trial, and further studies should be performed to corroborate this. Further research is also needed to determine whether structured peer review leads to greater knowledge transfer or better improvement of manuscripts. [ABSTRACT FROM AUTHOR]

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment of major depression and binge eating disorder or bulimia nervosa (N = 212) to illustrate the limits of the intraclass coefficient. Randomly selected therapy sessions were rated for therapeutic alliance quality by independent observers using the well-validated Vanderbilt Psychotherapy Process Scale and Vanderbilt Therapeutic Alliance Scale. Scores on subscales related to therapist behavior were restricted, indicating consistent alliance-supportive actions. Inter-rater reliability estimates were low despite high agreement between raters, however. The use of Bland–Altman plots to visualize agreement and data spread is suggested as a useful tool for researchers, consistent with the ideal of exploring reliability from a number of perspectives. [ABSTRACT FROM AUTHOR]

In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects' prior distribution. A new nonparametric index λ is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed. [ABSTRACT FROM AUTHOR]

Background Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts’ quality and the quality of the peer review process. The goal of this pilot study was to evaluate structured peer review implementation by: 1) exploring whether and how reviewers answered structured peer review questions, 2) analysing reviewer agreement, 3) comparing that agreement to agreement before implementation of structured peer review, and 4) further enhancing the piloted set of structured peer review questions. Methods Structured peer review consisting of nine questions was piloted in August 2022 in 220 Elsevier journals. We randomly selected 10% of these journals across all fields and IF quartiles and included manuscripts that received two review reports in the first 2 months of the pilot, leaving us with 107 manuscripts belonging to 23 journals. Eight questions had open-ended fields, while the ninth question (on language editing) had only a yes/no option. The reviews could also leave Comments-to-Author and Comments-to-Editor. Answers were independently analysed by two raters, using qualitative methods. Results Almost all the reviewers (n = 196, 92%) provided answers to all questions even though these questions were not mandatory in the system. The longest answer (Md 27 words, IQR 11 to 68) was for reporting methods with sufficient details for replicability or reproducibility. The reviewers had the highest (partial) agreement (of 72%) for assessing the flow and structure of the manuscript, and the lowest (of 53%) for assessing whether interpretation of the results was supported by data, and for assessing whether the statistical analyses were appropriate and reported in sufficient detail (52%). Two thirds of the reviewers (n = 145, 68%) filled out the Comments-to-Author section, of which 105 (49%) resembled traditional peer review reports. These reports contained a Md of 4 (IQR 3 to 5) topics covered by the structured questions. Absolute agreement regarding final recommendations (exact match of recommendation choice) was 41%, which was higher than what those journals had in the period from 2019 to 2021 (31% agreement, P = 0.0275). Conclusions Our preliminary results indicate that reviewers successfully adapted to the new review format, and that they covered more topics than in their traditional reports. Individual question analysis indicated the greatest disagreement regarding the interpretation of the results and the conducting and the reporting of statistical analyses. While structured peer review did lead to improvement in reviewer final recommendation agreements, this was not a randomized trial, and further studies should be performed to corroborate this. Further research is also needed to determine whether structured peer review leads to greater knowledge transfer or better improvement of manuscripts.

Objective: This study aimed to evaluate the concordance of eating disorders (EDs) diagnoses within a multidisciplinary team in a specialized hospital unit dedicated to the medical care of ED. Methods: The study analyzed data from 608 female patients who sought consultation at the Eating Disorders Referral Center between 2017 and 2021. The diagnoses were established according to the DSM‐5 criteria by endocrinologists, psychiatrists, and finally confirmed or discussed within a monthly multidisciplinary consensus meeting (MCM). Fleiss' Kappa tests were conducted to assess inter‐raters' agreement. Results: Overall, substantial agreement was observed between endocrinologists and psychiatrists and the MCM. A more detailed analysis revealed variations in agreement across different disorders. Certain EDs demonstrated substantial agreement (e.g., anorexia nervosa restrictive subtype), while others approached near‐perfect agreement (e.g., binge‐eating disorder). In contrast, agreement was fair to poor for anorexia nervosa binge‐purge subtype (ANBP) and slight for other specified feeding and ED. A period of temporary disagreement was noted for ANBP, partially attributed to practitioner turnover. An improvement in interdisciplinary agreement was observed for all ED diagnoses by the end of the study period. Discussion: Variations or lower levels of inter‐rater agreement may stem from atypical cases that fall on the border between two diagnoses or complex cases, as well as fluctuating symptoms. The progress observed throughout the study can be attributed in part to interdisciplinary learning, particularly facilitated by the MCM. The findings underscore the significance of striving for optimal concordance among different medical specialties to enhance patient care in ED treatment. Public Significance: This study scrutinizes the agreement levels of ED diagnoses among endocrinologists and psychiatrists within a multidisciplinary team at an Eating Disorders Referral Center. While substantial overall agreement was achieved, disparities or lower agreement levels were evident for certain diagnoses such as anorexia nervosa binge‐purge subtype. However, collaborative meetings led to a progressive enhancement in agreement over time. This research underscores the crucial role of a multidisciplinary team working collectively to ensure precise diagnoses and improved care for patients with EDs. [ABSTRACT FROM AUTHOR]

Introduction: The Revised Organ Injury Scale (OIS) of the American Association for Surgery of Trauma (AAST) is the most widely accepted classification of splenic trauma. The objective of this study was to evaluate inter-rater agreement for CT grading of blunt splenic injuries. Methods: CT scans in adult patients with splenic injuries at a level 1 trauma centre were independently graded by 5 fellowship trained abdominal radiologists using the AAST OIS for splenic injuries – 2018 revision. The inter-rater agreement for AAST CT injury score, as well as low-grade (IIII) versus high-grade (IV-V) splenic injury was assessed. Disagreement in two key clinical scenarios (no injury versus injury, and high versus low grade) were qualitatively reviewed to identify possible sources of disagreement. Results: A total of 610 examinations were included. The inter-rater absolute agreement was low (Fleiss kappa statistic 0.38, P < 0.001), but improved when comparing agreement between low and high grade injuries (Fleiss kappa statistic of 0.77, P <.001). There were 34 cases (5.6%) of minimum two-rater disagreement about no injury vs injury (AAST grade ≥ I). There were 46 cases (7.5%) of minimum two-rater disagreement of low grade (AAST grade I-III) versus high grade (AAST grade IV-V) injuries. Likely sources of disagreement were interpretation of clefts versus lacerations, peri-splenic fluid versus subcapsular hematoma, application of adding multiple low grade injuries to higher grade injuries, and identification of subtle vascular injuries. Conclusion: There is low absolute agreement in grading of splenic injuries using the existing AAST OIS for splenic injuries. [ABSTRACT FROM AUTHOR]

Accurate segmentation of liver and tumor regions in medical imaging is crucial for the diagnosis, treatment, and monitoring of hepatocellular carcinoma (HCC) patients. However, manual segmentation is time-consuming and subject to inter- and intra-rater variability. Therefore, automated methods are necessary but require rigorous validation of high-quality segmentations based on a consensus of raters. To address the need for reliable and comprehensive data in this domain, we present LiverHccSeg, a dataset that provides liver and tumor segmentations on multiphasic contrast-enhanced magnetic resonance imaging from two board-approved abdominal radiologists, along with an analysis of inter-rater agreement.LiverHccSeg provides a curated resource for liver and HCC tumor segmentation tasks. The dataset includes a scientific reading and co-registered contrast-enhanced multiphasic magnetic resonance imaging (MRI) scans with corresponding manual segmentations by two board-approved abdominal radiologists and relevant metadata and offers researchers a comprehensive foundation for external validation, and benchmarking of liver and tumor segmentation algorithms. The dataset also provides an analysis of the agreement between the two sets of liver and tumor segmentations. Through the calculation of appropriate segmentation metrics, we provide insights into the consistency and variability in liver and tumor segmentations among the radiologists. A total of 17 cases were included for liver segmentation and 14 cases for HCC tumor segmentation. Liver segmentations demonstrates high segmentation agreement (mean Dice, 0.95 ± 0.01 [standard deviation]) and HCC tumor segmentations showed higher variation (mean Dice, 0.85 ± 0.16 [standard deviation]).The applications of LiverHccSeg can be manifold, ranging from testing machine learning algorithms on public external data to radiomic feature analyses. Leveraging the inter-rater agreement analysis within the dataset, researchers can investigate the impact of variability on segmentation performance and explore methods to enhance the accuracy and robustness of liver and tumor segmentation algorithms in HCC patients. By making this dataset publicly available, LiverHccSeg aims to foster collaborations, facilitate innovative solutions, and ultimately improve patient outcomes in the diagnosis and treatment of HCC.

Background: Posterior cortex epilepsy (PCE) primarily comprises seizures originating from the occipital, parietal, and/or posterior edge of the temporal lobe. Electroclinical dissociation and subtle imaging representation render the diagnosis of PCE challenging. Improved methods for accurately identifying patients with PCE are necessary. Objectives: To develop a novel voxel-based image postprocessing method for better visual identification of the neuroimaging abnormalities associated with PCE. Design: Multicenter, retrospective study. Methods: Clinical and imaging features of 165 patients with PCE were retrospectively reviewed and collected from five epilepsy centers. A total of 37 patients (32.4% female, 20.2 ± 8.9 years old) with magnetic resonance imaging (MRI)-negative PCE were finally included for analysis. Image postprocessing features were calculated over a neighborhood for each voxel in the multimodality data. The postprocessed maps comprised structural deformation, hyperintense signal, and hypometabolism. Five raters from three different centers were blinded to the clinical diagnosis and determined the neuroimaging abnormalities in the postprocessed maps. Results: The average accuracy of correct identification was 55.7% (range from 43.2 to 62.2%) and correct lateralization was 74.1% (range from 64.9 to 81.1%). The Cronbach's alpha was 0.766 for the correct identification and 0.683 for the correct lateralization with similar results of the interclass correlation coefficient, thus indicating reliable agreement between the raters. Conclusion: The image postprocessing method developed in this study can potentially improve the visual detection of MRI-negative PCE. The technique could lead to an increase in the number of patients with PCE who could benefit from the surgery. [ABSTRACT FROM AUTHOR]

Simple Summary: The objectives of this study were to evaluate the associations between the faecal egg count for strongylids and the clinical parameters in lactating dairy ewes. In addition, the agreement of three raters regarding the clinical examination was assessed. For this purpose, a total of 1195 dairy ewes from 16 farms were included. FAMACHA©, BCS and dag scores were determined and a Mini-FLOTAC was performed on individual samples, which were then pooled for larval culture at the farm level according to the number of lactations. Trichostrongylids were found in 95% of the investigated samples. The BCS was negatively correlated with the faecal egg count, while the FAMACHA© score showed a slightly positive correlation. The dag score did not show significant associations. A minority (25%) of the flock shed the majority of eggs (47% to 84%). It was concluded that clinical scoring does not allow us to extrapolate to faecal egg excretion. The agreement was moderate to good between different raters, showing that training on clinical parameters is recommended for assessment in sheep. From a clinical perspective, both faecal egg counts and clinical parameters should be used simultaneously as separate tools to detect major egg shedders (contaminating the pasture) and sheep suffering from strongylidosis. Gastrointestinal nematodes, most notably trichostrongylids, are known to cause significant losses in sheep production. Previous studies have shown that monitoring parameters (e.g., FAMACHA©, BCS, dag score) change with increasing egg excretion. These parameters are well known and frequently used for targeted selective treatment. Based on the willingness to participate in this study (based on a previous questionnaire distribution among sheep farmers in Austria) we investigated the associations between faecal egg counts and the FAMACHA©, BCS, and dag scores of 1195 dairy ewes. Faecal samples were analysed using the Mini-FLOTAC technique I and larval culture. Three raters assessed the FAMACHA©, BCS, and dag scores in sheep to calculate the inter-rater agreement and intraclass correlation coefficient. The responses to the questionnaire of 23 farms were used for the evaluation, of which 16 farms were visited. Trichostrongylid eggs were detected in 95% of the faecal samples. The BCS was negatively correlated with the eggs per gram of faeces (EpG) (r = −0.156; p < 0.001) and the FAMACHA© score was slightly positively correlated with EpG (r = 0.196; p < 0.001). A small proportion of sheep (25%) shed the majority of eggs (47% to 84%). A moderate to good agreement for the parameters was found between the raters. In conclusion, the clinical parameters showed only weak correlations with faecal egg counts, and we confirmed that a minority of the flock is responsible for the majority of the pasture contamination with trichostrongylid eggs. Clinical raters should be trained before a study to increase the agreement between them. [ABSTRACT FROM AUTHOR]

Τeachers' interpersonal behaviour is considered important for the quality of the student–teacher relationship. The present study examined teachers' and students' perceptions regarding teachers' interpersonal behaviour and the degree of their agreement at the class level. Moreover, this study investigated whether possible differences between them are associated with the teachers' and classes' characteristics. Participants were 1669 students and 43 secondary teachers within 85 secondary classrooms in Greece. The Greek version of the Questionnaire on Teacher Interaction was used to assess eight aspects of teachers' interpersonal behaviour. A high degree of divergence between teachers' self-perceptions and students' perceptions about their teachers' interpersonal behaviour was found. Multiple regression analyses showed that the differences among teachers' and students' perceptions can partially be explained by teachers' years of teaching experience and the degree level, as well as the class educational level. Findings of this study provide specific practical implications. [ABSTRACT FROM AUTHOR]

Our aim was to compare the inter-rater agreement about transvaginal ultrasonography (TVS) with magnetic resonance imaging (MRI) with regard to diagnosing adenomyosis and for assessing various predefined imaging features of adenomyosis, in the same set of women. The study cohort included 51 women, prospectively, consecutively recruited based on a clinical suspicion of adenomyosis. MRIs and TVS videoclips and 3D volumes were retrospectively assessed by four experienced radiologists and five experienced sonographers, respectively. Each rater subjectively evaluated the presence or absence of adenomyosis, as well as imaging features suggestive of adenomyosis. Fleiss kappa (κ) was used to reflect inter-rater agreement for categorical data, and the intraclass correlation coefficient (ICC) was used to reflect the reliability of quantitative data. Agreement between raters for diagnosing adenomyosis was higher for TVS than for MRI (κ = 0.42 vs. 0.28). MRI had a higher inter-rater agreement in assessing wall asymmetry, irregular junctional zone (JZ), and the presence of myometrial cysts, while TVU had a better agreement for assessing globular shape. MRI showed a moderate to good reliability for measuring the JZ (ICC = 0.57–0.82). For TVS, the JZ was unmeasurable in >50% of cases, and the remaining cases had low reliability (ICC = −0.31–0.08). We found that inter-rater agreement for diagnosing adenomyosis was higher for TVS than for MRI, despite the fact that MRI showed a higher inter-rater agreement in most specific features. Measurements of JZ in the coronal plane with 3D TVS were unreliable and thus unlikely to be useful for diagnosing adenomyosis. [ABSTRACT FROM AUTHOR]

Sleep dysregulation in Parkinson's disease (PD) has been hypothesized to occur, in part, from dysfunction in the basal ganglia-cortical circuit. Assessment of this relationship requires accurate sleep stage determination, a known challenge in this clinical population. Our objective was to optimize the consensus on the sleep staging process and reduce interrater variability in a cohort of advanced PD subjects. Fifteen PD subjects were enrolled from three sites in a clinical trial that involved recordings from subthalamic nucleus (STN) deep brain stimulation (DBS) leads (NCT04620551). Video polysomnography (vPSG) data for a total of 45 nights were analyzed. Four experienced scorers independently scored data on initial review. Epochs with less than 75% consensus were flagged for secondary review. In secondary review of discordant epochs, two of the original scorers re-assessed epochs, from which the final consensus stage was derived. Sleep stage classification agreement averaged 83.10% across all sleep stages on initial scoring (IS), and on secondary consensus scoring (CS) review, agreement reached 96.58%. Greatest disagreement was noted in determination of awake epochs (33.6% of discordant epochs) and non-rapid-eye-movement stage 2 (N2) epochs (31.8% of discordant epochs). Scoring discrepancy was resolved with direct measurement of cortical frequency and amplitudes, physiologic context of the epoch, and video review. Our method of multi-level initial and then secondary consensus review scoring resulted in consensus scoring agreement superior to conventional standards. This work features a custom-engineered vPSG software and review platform for integration of consensus sleep stage scoring in a multi-site clinical trial. • High sleep stage classification agreement across multi-site review for Parkinson's disease. • Consensus process was superior to conventional standard sleep staging. • Intra-day consistency was high across 3-night in-hospital sleep study. [ABSTRACT FROM AUTHOR]

Copyright of International Journal of Medicine & Science of Physical Activity & Sport / Revista Internacional de Medicina y Ciencias de la Actividad Física y del Deporte is the property of Revista Internacional de Medicina y Ciencias de la Actividad Fisica y del Deporte and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract Background Ear, Nose and Throat (ENT) primary contact models of care use audiologists as the first triage point for children referred to ENT for middle ear and hearing concerns; and have shown reduced waiting time, improved ENT surgical conversion rates and increased service capacity. This study aimed to investigate ‘safety and quality’ of the model by looking at agreement between audiologists’ and an ENT’s clinical decisions. Methods We performed an inter-rater agreement study on diagnosis and management decisions made by audiologists and an ENT for 50 children seen in an Australian hospital’s ENT primary contact service, and examined the nature and patterns of disagreements. Results Professionals agreed on at least one site-of-lesion diagnosis for all children (100%) and on the primary management for 74% (Gwet’s AC1 = 0.67). Management disagreements clustered around i) providing ‘watchful waiting’ versus sooner medical opinion (18%), and ii) providing monitoring versus discharge for children with no current symptoms (8%). There were no cases where the audiologist recommended discharge when the ENT recommended further medical opinion. Conclusions Our novel research provides further evidence that Audiologist-led primary contact models for children with middle ear and hearing concerns are safe as well as efficient.

Background: Magnetic resonance imaging (MRI)-based rotator cuff assessment is often qualitative and subjective; few studies have tried to validate such preoperative assessments. This study investigates relationships of preoperative MRI assessments made by conventional approaches to intraoperative findings of tear type, location, and size or MRI-assessed muscle occupation ratio. Methods: Intraoperatively, surgeons assessed tear type, location, anterior-posterior (AP) width, and medial-lateral length in 102 rotator cuff repair patients. Two musculoskeletal radiologists independently assessed the preoperative MRI scans for these same parameters and supraspinatus muscle atrophy by both Warner classification and quantitative occupation ratio. Exact agreement proportions, kappa statistics, and correlation coefficients were used to quantify agreement relationships. Results: Agreement between MRI readers’ and surgeons’ observations of tear status averaged 93% with κ = 0.38, and that of tear location averaged 77% with κ = 0.50. Concordance correlations of MRI and intraoperative measures of anterior-posterior and medial-lateral tear length averaged 0.59 and 0.56 across readers, respectively. Despite excellent interrater agreement on Warner classification (exact agreement proportion 0.91) and occupation ratio (concordance correlation 0.93) separately, correlations between these 2 measures were −0.54 and −0.64 for the 2 readers, respectively. Patients with Warner grade 0 had occupation ratios ranging from 0.5 to 1.5. Conclusion: Correlations of preoperative MRI tear dimensions and muscle atrophy assessed by conventional approaches with intraoperatively measured tear dimensions and quantitative occupation ratio, respectively, were only fair. Since tear size and muscle atrophy are known strong predictors of outcomes following rotator cuff repair that may influence treatment decisions, surgeons need to be aware of the limitations of MRI methods. Continued development and validation of quantitative preoperative imaging methods to accurately assess these parameters are needed to improve surgical planning and prognosis.

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC1 was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC1 ranging from 0.071 to 0.535). After training, the multi-rater AC1 ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC1: 0.405, 95%-CI 0.149–0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings. [ABSTRACT FROM AUTHOR]

ObjectiveTo assess and compare the inter-rater agreement of the CDC criteria and the ASEPSIS score in identifying surgical site infections after cesarean section.MethodsProspective observational study including 110 patients subjected to a cesarean section at our institution. Surgical wounds were managed according to standard care and were photographed on the third, seventh, and thirtieth postoperative day or during any evaluation in case of complications. Three expert surgeons reviewed the prospectively gathered data and photographs and classified each wound using CDC criteria and the ASEPSIS score. The inter-rater agreements of CDC criteria and ASEPSIS score were determined with Krippendorff's Alpha with linear weights and compared with a confidence interval approach.ResultsThe weighted α coefficient for CDC criteria was 0.587 (95%CI, 0.411–0.763, p

The extraction of patient signs and symptoms recorded as free text in electronic health records is critical for precision medicine. Once extracted, signs and symptoms can be made computable by mapping to signs and symptoms in an ontology. Extracting signs and symptoms from free text is tedious and time-consuming. Prior studies have suggested that inter-rater agreement for clinical concept extraction is low. We have examined inter-rater agreement for annotating neurologic concepts in clinical notes from electronic health records. After training on the annotation process, the annotation tool, and the supporting neuro-ontology, three raters annotated 15 clinical notes in three rounds. Inter-rater agreement between the three annotators was high for text span and category label. A machine annotator based on a convolutional neural network had a high level of agreement with the human annotators but one that was lower than human inter-rater agreement. We conclude that high levels of agreement between human annotators are possible with appropriate training and annotation tools. Furthermore, more training examples combined with improvements in neural networks and natural language processing should make machine annotators capable of high throughput automated clinical concept extraction with high levels of agreement with human annotators.

Objectives: Objective structured clinical examinations (OSCEs) are used to assess clinical competence in medical education. Evaluations using video-recorded OSCEs are effective in reducing costs in terms of time and human resources. To improve inter-rater reliability, these evaluations undergo moderation in the form of a discussion between the raters to obtain consistency in grading according to the rubric criteria. We examined the effect of moderation related to the rubric criteria on the inter-rater reliability of a video-recorded OSCE with real patients. Methods: Forty OSCE videos in which students performed range-of-motion tests at shoulder abduction on real patients were assessed by two raters. The two raters scored videos 1 to 10 without moderation and videos 11 to 40 with moderation each time. The inter-rater reliability of the OSCE was calculated using the weighted kappa coefficient. Results: The mean scores of the weighted kappa coefficients were 0.49 for videos 1 to 10, 0.57 for videos 11 to 20, 0.66 for videos 21 to 30, and 0.82 for videos 31 to 40. Conclusions: An assessment of video-recorded OSCEs was conducted with real patients in a real clinical setting. Repeated moderation improved the inter-rater reliability. This study suggests the effectiveness of moderation in OSCEs with real patients.

Both internalizing and externalizing psychopathologies interfere with the treatment of substance use disorders (SUD) in adolescents. Self-reports of psychopathologies are likely biased and may be validated with parental reports. We compared N = 70 standardized self-reports of adolescents entering outpatient SUD treatment (13.2–18.6 years old, 43% female) to parental reports on the same psychopathologies, and explored biases due to gender, age, SUD diagnoses and SUD severity. Bivariate bootstrapped Pearson correlation coefficients revealed several small to moderate correlations between both reporting sources (r = 0.29–0.49, all pcorrected ≤ 0.039). A repeated measures MANOVA revealed moderately stronger parental reports of adolescent psychopathologies compared to adolescent self-reports for most externalizing problems (dissocial and aggressive behaviors, p ≤ 0.016, η2part = 0.09–0.12) and social/attention problems (p ≤ 0.012, η2part = 0.10), but no differences for most internalizing problems (p ≥ 0.073, η2part = 0.02–0.05). Differences were not associated with other patient or parental characteristics including age, gender, number of co-occurring diagnoses or presence/absence of a certain SUD (all puncorrected ≥ 0.088). We concluded that treatment-seeking German adolescents with SUD present with a multitude of extensive psychopathologies. The relevant deviation between self- and parental reports indicate that the combination of both reports might help to counteract dissimulation and other reporting biases. The generalizability of results to inpatients, psychiatry patients in general, or adolescents without SUD, as well as the validity of self- and parental reports in comparison to clinical judgements remain unknown. [ABSTRACT FROM AUTHOR]

Background: Posterior cortex epilepsy (PCE) primarily comprises seizures originating from the occipital, parietal, and/or posterior edge of the temporal lobe. Electroclinical dissociation and subtle imaging representation render the diagnosis of PCE challenging. Improved methods for accurately identifying patients with PCE are necessary. Objectives: To develop a novel voxel-based image postprocessing method for better visual identification of the neuroimaging abnormalities associated with PCE. Design: Multicenter, retrospective study. Methods: Clinical and imaging features of 165 patients with PCE were retrospectively reviewed and collected from five epilepsy centers. A total of 37 patients (32.4% female, 20.2 ± 8.9 years old) with magnetic resonance imaging (MRI)-negative PCE were finally included for analysis. Image postprocessing features were calculated over a neighborhood for each voxel in the multimodality data. The postprocessed maps comprised structural deformation, hyperintense signal, and hypometabolism. Five raters from three different centers were blinded to the clinical diagnosis and determined the neuroimaging abnormalities in the postprocessed maps. Results: The average accuracy of correct identification was 55.7% (range from 43.2 to 62.2%) and correct lateralization was 74.1% (range from 64.9 to 81.1%). The Cronbach's alpha was 0.766 for the correct identification and 0.683 for the correct lateralization with similar results of the interclass correlation coefficient, thus indicating reliable agreement between the raters. Conclusion: The image postprocessing method developed in this study can potentially improve the visual detection of MRI-negative PCE. The technique could lead to an increase in the number of patients with PCE who could benefit from the surgery. [ABSTRACT FROM AUTHOR]

Purpose: Risk prediction models are widely used in the perioperative setting to identify high-risk patients who may benefit from additional care and to aid clinical decision-making. pPOSSUM is such a prediction model, however, little is known about the inter-rater agreement when scoring subjective parameters. This study assessed the inter-rater agreement between clinicians of different specialties and work-level when scoring 30 clinical case reports of geriatric hip fracture patients with pPOSSUM.Methods: Eighteen clinicians of the department of Surgery (three specialists, four residents), Anaesthesiology (four specialists, two residents) and Emergency Medicine (three specialists, two residents) who were familiar with the pPOSSUM scoring system were asked to calculate the scores. The kappa statistic and the statistical method of Fleiss were used to analyse inter-rater agreement.Results: The response rate was 100%. Among surgeons, Anaesthesiologists and Emergency department doctors (ED), the overall mean kappa values were 0.42, 0.08 and 0.20, respectively. Among surgery, anaesthesiology and ED residents the overall mean kappa values were 0.21, 0.33 and 0.37, respectively. Within the department of Surgery, Anaesthesiology and Emergency Medicine the overall mean kappa values were 0.23, 0.12 and 0.22, respectively. An overall mean kappa value of 0.19 was seen among all specialists. All residents had an overall mean kappa value of 0.21 and all clinicians had an overall mean kappa value of 0.21.Conclusion: The overall inter-rater agreement of clinicians and interdisciplinary agreement when scoring geriatric hip fracture patients with pPOSSUM was low and prone to subjectivity in our study. A higher work-experience level did not lead to better agreement. When pPOSSUM is calculated without clinical assessment by the same clinician, caution is advised to prevent over-reliance on the pPOSSUM risk prediction model.Level Of Evidence: III. [ABSTRACT FROM AUTHOR]

IoT edge computing is a new computing paradigm "in the IoT domain" for performing calculations and processing at the edge of the network, closer to the user and the source of the data. This paradigm is relatively recent, and, together with cloud and fog computing, there may be some confusion about its meaning and implications. This paper aims to help practitioners and researchers better understand what the industry thinks about what IoT edge computing is, and the expected benefits and challenges associated with this paradigm. We conducted a survey using a semi-structured in-depth questionnaire to collect qualitative data from relevant stakeholders from 29 multinational companies and qualitatively analyzed these data using the Constructivist Grounded Theory (Charmaz) method. Several researchers participated in the coding process (collaborative coding). To ensure consensus on the constructs that support the theory and thus improve the rigor of qualitative research, we conducted an intercoder agreement analysis. From the analysis, we have derived a substantive and analytic theory of what companies perceive about IoT edge computing, its benefits and challenges. The theory is substantive in that the scope of validity refers to the 29 surveys processed and analytic in that it analyzes "what is" rather than explaining causality or attempting predictive generalizations. A public repository with all the data related to the information capture process and the products resulting from the analysis of this information is publicly available. This study aims to strengthen the evidence and support practitioners in making better informed decisions about why companies are adopting edge computing and the current challenges they face. Additionally, the testing theory phase shows that the results are aligned with the ISO/IEC TR 30164 standard. [ABSTRACT FROM AUTHOR]

Background The treatment approach for proximal humeral fractures is determined by various factors, including patient age, sex, dominant arm, fracture pattern, presence of osteoporosis, preexisting arthritis, rotator cuff status, and medical comorbidities. However, there is a lack of consensus in the literature regarding the optimal treatment for displaced proximal humeral fractures. This study aimed to assess and quantify the decision-making process for either conservative or surgical treatment and the choice of surgical method among shoulder surgeons when treating proximal humeral fractures. Methods Forty sets of true anteroposterior view, scapular Y projection view, and three-dimensional computed tomography of proximal humeral fractures were provided to 12 shoulder surgeons along with clinical information. Surveys regarding Neer classification, decisions between conservative and surgical treatments, and chosen methods were conducted twice with an interval of 2 months. The factors affecting the treatment plans were also assessed. Results The inter-rater agreement was fair for Neer classification (kappa=0.395), moderate for the decision between conservative and surgical treatments (kappa=0.528), and substantial for the chosen method of surgical treatment (kappa=0.740). The percentage of agreement was 71.1% for Neer classification, 84.6% for the decision between conservative and surgical treatment, and 96.4% for the chosen method of surgical treatment. The fracture pattern was the most crucial factor in deciding between conservative and surgical treatments, followed by age and physical activity. Conclusions The decision between conservative and surgical treatment for proximal humeral fractures showed good agreement, while the chosen method between osteosynthesis and arthroplasty showed substantial agreement among shoulder surgeons.

Gastrointestinal nematodes, most notably trichostrongylids, are known to cause significant losses in sheep production. Previous studies have shown that monitoring parameters (e.g., FAMACHA©, BCS, dag score) change with increasing egg excretion. These parameters are well known and frequently used for targeted selective treatment. Based on the willingness to participate in this study (based on a previous questionnaire distribution among sheep farmers in Austria) we investigated the associations between faecal egg counts and the FAMACHA©, BCS, and dag scores of 1195 dairy ewes. Faecal samples were analysed using the Mini-FLOTAC technique I and larval culture. Three raters assessed the FAMACHA©, BCS, and dag scores in sheep to calculate the inter-rater agreement and intraclass correlation coefficient. The responses to the questionnaire of 23 farms were used for the evaluation, of which 16 farms were visited. Trichostrongylid eggs were detected in 95% of the faecal samples. The BCS was negatively correlated with the eggs per gram of faeces (EpG) (r = −0.156; p < 0.001) and the FAMACHA© score was slightly positively correlated with EpG (r = 0.196; p < 0.001). A small proportion of sheep (25%) shed the majority of eggs (47% to 84%). A moderate to good agreement for the parameters was found between the raters. In conclusion, the clinical parameters showed only weak correlations with faecal egg counts, and we confirmed that a minority of the flock is responsible for the majority of the pasture contamination with trichostrongylid eggs. Clinical raters should be trained before a study to increase the agreement between them.

Abstract Background Proximal humerus fractures (PHF) are frequent, however, several studies show low inter-rater agreement in the diagnosis and treatment of these injuries. Differences are usually related to the experience of the evaluators and/or the diagnostic methods used. This study was designed to investigate the hypothesis that shoulder surgeons and diagnostic imaging specialists using 3D printing models and shoulder CT scans in assessing proximal humerus fractures. Methods We obtained 75 tomographic exams of PHF to print three-dimensional models. After, two shoulder surgeons and two specialists in musculoskeletal imaging diagnostics analyzed CT scans and 3D models according to the Neer and AO/OTA group classification and suggested a treatment recommendation for each fracture based on the two diagnostic methods. Results The classification agreement for PHF using 3D printing models among the 4 specialists was moderate (global k = 0.470 and 0.544, respectively for AO/OTA and Neer classification) and higher than the CT classification agreement (global k = 0.436 and 0.464, respectively for AO/OTA and Neer). The inter-rater agreement between the two shoulder surgeons were substantial. For the AO/OTA classification, the inter-rater agreement using 3D printing models was higher (k = 0.700) than observed for CT (k = 0.631). For Neer classification, inter-rater agreement with 3D models was similarly higher (k = 0.784) than CT images (k = 0.620). On the other hand, the inter-rater agreement between the two specialists in diagnostic imaging was moderate. In the AO/OTA classification, the agreement using CT was higher (k = 0.532) than using 3D printing models (k = 0.443), while for Neer classification, the agreement was similar for both 3D models (k = 0.478) and CT images (k = 0.421). Finally, the inter-rater agreement in the treatment of PHF by the 2 surgeons was higher for both classifications using 3D printing models (AO/OTA—k = 0.818 for 3D models and k = 0.537 for CT images). For Neer classification, we saw k = 0.727 for 3D printing models and k = 0.651 for CT images. Conclusion The insights from this diagnostic pilot study imply that for shoulder surgeons, 3D printing models improved the diagnostic agreement, especially the treatment indication for PHF compared to CT for both AO/OTA and Neer classifications On the other hand, for specialists in diagnostic imaging, the use of 3D printing models was similar to CT scans for diagnostic agreement using both classifications. Trial registration Brazil Platform under no. CAAE 12273519.7.0000.5505.

European languages represent 3% of the world 7,000+ languages, and yet, a recent survey shows that they amount for 84% of the studies in metaphor publications. The disproportionate reliance on such an infinitely small sample of the world diversity of languages seriously undermines the representativeness of generalizations about a phenomenon that is claimed to be found across languages and cultures. This paper is a call for more diversity in metaphor studies and a discussion about the methodological benefits and challenges posed by such ambition. Particularly relevant to increasing diversity in metaphor studies is the role of culture and context in metaphor identification procedures. This methodological question is addressed with a specific case study that shows how providing annotators with cultural and contextual information are key to increasing inter-rater agreements.

Background: Ear, Nose and Throat (ENT) primary contact models of care use audiologists as the first triage point for children referred to ENT for middle ear and hearing concerns; and have shown reduced waiting time, improved ENT surgical conversion rates and increased service capacity. This study aimed to investigate 'safety and quality' of the model by looking at agreement between audiologists' and an ENT's clinical decisions.  METHODS: We performed an inter-rater agreement study on diagnosis and management decisions made by audiologists and an ENT for 50 children seen in an Australian hospital's ENT primary contact service, and examined the nature and patterns of disagreements.Results: Professionals agreed on at least one site-of-lesion diagnosis for all children (100%) and on the primary management for 74% (Gwet's AC1 = 0.67). Management disagreements clustered around i) providing 'watchful waiting' versus sooner medical opinion (18%), and ii) providing monitoring versus discharge for children with no current symptoms (8%). There were no cases where the audiologist recommended discharge when the ENT recommended further medical opinion.Conclusions: Our novel research provides further evidence that Audiologist-led primary contact models for children with middle ear and hearing concerns are safe as well as efficient. [ABSTRACT FROM AUTHOR]

Background:Head computed tomography (CT) is used to predict neurological outcome after cardiac arrest (CA). The current reference standard includes quantitative image analysis by a neuroradiologist to determine the Gray-White-Matter Ratio (GWR) which is calculated via the manual measurement of radiodensity in different brain regions. Recently, automated analysis methods have been introduced. There is limited data on the Inter-rater agreement of both methods. Methods: Three blinded human raters (neuroradiologist, neurologist, student) with different levels of clinical experience retrospectively assessed the Gray-White-Matter Ratio (GWR) in head CTs of 95 CA patients. GWR was also quantified by a recently published computer algorithmthat uses coregistration with standardized brain spaces to identify regions of interest (ROIs). We calculated intraclass correlation (ICC) for inter-rater agreement between human and computer raters as well as area under the curve (AUC) and sensitivity/specificity for poor outcome prognostication. Results: Inter-rater agreement on GWR was very good (ICC 0.82-0.84) between all three human raters across different levels of expertise and between the computer algorithm and neuroradiologist (ICC 0.83; 95% CI 0.78-0.88). Despite high overall agreement, we observed considerable, clinically relevant deviations of GWR measurements (up to 0.24) in individual patients. In our cohort, at a GWR threshold of 1.10, this did not lead to any false poor neurological outcome prediction. Conclusion: Human and computer raters demonstrated high overall agreement in GWR determination in head CTs after CA. The clinically relevant deviations of GWR measurement in individual patients underscore the necessity of additional qualitative evaluation and integration of head CT findings into a multimodal approach to prognostication of neurological outcome after CA. [ABSTRACT FROM AUTHOR]

This study aims to compare the reliability and acceptability of psychiatric interviews using telepsychiatry and face-to-face modalities in the emergency room setting. In this prospective observational feasibility study, psychiatric patients (n = 38) who presented in emergency rooms between April and June 2020, went through face-to-face and videoconference telepsychiatry interviews in a non-randomised varying order. Interviewers and a senior psychiatry resident who observed both interviews determined diagnosis, recommended disposition and indication for involuntary admission. Patients and psychiatrists completed acceptability post-assessment surveys. Agreement between raters on recommended disposition and indication for involuntary admission as measured by Cohen's kappa was 'strong' to 'almost perfect' (0.84/0.81, 0.95/0.87 and 0.89/0.94 for face-to-face vs. telepsychiatry, observer vs. face-to-face and observer vs. telepsychiatry, respectively). Partial agreement between the raters on diagnosis was 'strong' (Cohen's kappa of 0.81, 0.85 and 0.85 for face-to-face vs. telepsychiatry, observer vs. face-to-face and observer vs. telepsychiatry, respectively).Psychiatrists' and patients' satisfaction rates, and psychiatrists' perceived certainty rates, were comparably high in both face-to-face and telepsychiatry groups. Telepsychiatry is a reliable and acceptable alternative to face-to-face psychiatric assessments in the emergency room setting. Implementing telepsychiatry may improve the quality and accessibility of mental health services. Telepsychiatry and face-to-face psychiatric assessments in the emergency room setting have comparable reliability. Patients and providers report a comparable high level of satisfaction with telepsychiatry and face-to-face modalities in the emergency room setting. Providers report a comparable level of perceived certainty in their clinical decisions based on telepsychiatry and face-to-face psychiatric assessments in the emergency room setting. [ABSTRACT FROM AUTHOR]

Objective: The Swiss guidelines for driving with epilepsy require that electroencephalogram (EEG) findings must be compatible with the fitness to drive (FTD) without specifying any criteria. This nationwide survey investigated how Swiss neurologists implement this requirement in clinical practice., Methods: An online survey, including 19 EEG examples and statements about the compatibility of the EEG with FTD, was distributed to all members of the Swiss Society of Clinical Neurophysiology and all Swiss neurological clinics with residency programs. Descriptive statistics and Fleiss' Kappa for inter-rater agreement were performed., Results: 102 participants (37% female, 45% <45 years) completed the survey, with 15% primarily working in a specialized epileptology center (EPI), 55% in a hospital setting without epileptological focus (HOS), and 30% in private practice (PP). Most participants of all three groups regarded EEG normal variants as compatible with FTD, while hyperventilation-induced rhythmic slowing and some pathological patterns (e.g., frontal and temporal intermittent rhythmic delta activity [FIRDA, TIRDA], focal interictal epileptiform discharges [IED], focal seizure) were evaluated more heterogeneously. The EEG inter-rater agreement for EPI was 0.4; 0.31 for PP; and 0.24 for HOS. No consensus was found for acceptable duration for generalized or focal IEDs. Among all participants, evaluation by an epilepsy center (85%) and reaction-time testing (67%) were evaluated as the most useful additional examinations. However, reaction-time testing is rarely or never performed by 58%. Most supported EEG results as part of the FTD assessment and demanded more research (both 82%)., Significance: Our survey indicates considerable heterogeneity among Swiss neurologists when evaluating the EEG findings for FTD. Inter-rater agreement in all three groups was fair, with highest agreement among epileptologists. We noted a discrepancy between the usefulness and actual application of reaction-time testing. More training and research are warranted to achieve rater-independent consistency for FTD evaluation., Plain Language Summary: In Switzerland, neurologists must consider the findings from EEG (brain waves) exams to evaluate driving safety in people with epilepsy. We surveyed Swiss neurologists, asking their opinions on this matter. We found that opinions differ among individual doctors, with the highest agreement among epilepsy specialists. We also found that some additional tests, such as reaction-time testing, are perceived as useful but are rarely performed. The participating neurologists supported considering EEG results for driving fitness assessments but favored more precise guidelines and research., (© 2024 The Author(s). Epilepsia Open published by Wiley Periodicals LLC on behalf of International League Against Epilepsy.)

Abstract Background Vast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers. Methods A total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018. Results A total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups. Conclusions We did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed. Trial registration The study was preregistered at OSF.io/n4fq3 .

BackgroundHead computed tomography (CT) is used to predict neurological outcome after cardiac arrest (CA). The current reference standard includes quantitative image analysis by a neuroradiologist to determine the Gray-White-Matter Ratio (GWR) which is calculated via the manual measurement of radiodensity in different brain regions. Recently, automated analysis methods have been introduced. There is limited data on the Inter-rater agreement of both methods.MethodsThree blinded human raters (neuroradiologist, neurologist, student) with different levels of clinical experience retrospectively assessed the Gray-White-Matter Ratio (GWR) in head CTs of 95 CA patients. GWR was also quantified by a recently published computer algorithm that uses coregistration with standardized brain spaces to identify regions of interest (ROIs). We calculated intraclass correlation (ICC) for inter-rater agreement between human and computer raters as well as area under the curve (AUC) and sensitivity/specificity for poor outcome prognostication.ResultsInter-rater agreement on GWR was very good (ICC 0.82–0.84) between all three human raters across different levels of expertise and between the computer algorithm and neuroradiologist (ICC 0.83; 95% CI 0.78–0.88). Despite high overall agreement, we observed considerable, clinically relevant deviations of GWR measurements (up to 0.24) in individual patients. In our cohort, at a GWR threshold of 1.10, this did not lead to any false poor neurological outcome prediction.ConclusionHuman and computer raters demonstrated high overall agreement in GWR determination in head CTs after CA. The clinically relevant deviations of GWR measurement in individual patients underscore the necessity of additional qualitative evaluation and integration of head CT findings into a multimodal approach to prognostication of neurological outcome after CA.

Our aim was to compare the inter-rater agreement about transvaginal ultrasonography (TVS) with magnetic resonance imaging (MRI) with regard to diagnosing adenomyosis and for assessing various predefined imaging features of adenomyosis, in the same set of women. The study cohort included 51 women, prospectively, consecutively recruited based on a clinical suspicion of adenomyosis. MRIs and TVS videoclips and 3D volumes were retrospectively assessed by four experienced radiologists and five experienced sonographers, respectively. Each rater subjectively evaluated the presence or absence of adenomyosis, as well as imaging features suggestive of adenomyosis. Fleiss kappa (κ) was used to reflect inter-rater agreement for categorical data, and the intraclass correlation coefficient (ICC) was used to reflect the reliability of quantitative data. Agreement between raters for diagnosing adenomyosis was higher for TVS than for MRI (κ = 0.42 vs. 0.28). MRI had a higher inter-rater agreement in assessing wall asymmetry, irregular junctional zone (JZ), and the presence of myometrial cysts, while TVU had a better agreement for assessing globular shape. MRI showed a moderate to good reliability for measuring the JZ (ICC = 0.57–0.82). For TVS, the JZ was unmeasurable in >50% of cases, and the remaining cases had low reliability (ICC = −0.31–0.08). We found that inter-rater agreement for diagnosing adenomyosis was higher for TVS than for MRI, despite the fact that MRI showed a higher inter-rater agreement in most specific features. Measurements of JZ in the coronal plane with 3D TVS were unreliable and thus unlikely to be useful for diagnosing adenomyosis.

Background: Magnetic resonance imaging (MRI) is used to detect degenerative changes of the lumbar spine. SpineNet (SN), a computer vision-based system, performs an automated analysis of degenerative features in MRI scans aiming to provide high accuracy, consistency and objectivity. This study evaluated SN's ratings compared with those of an expert radiologist.Method: MRIs of 882 patients (mean age, 72 ± 8.8 years) with degenerative spinal disorders from two previous trials carried out in our spine center between 2011 and 2019, were analyzed by an expert radiologist. Lumbar segments (L1/2-L5/S1) were graded for Pfirrmann Grades (PG), Spondylolisthesis (SL) and Central Canal Stenosis (CCS). SN's analysis for the equivalent parameters was generated. Agreement between methods was analyzed using kappa (κ), Spearman correlation (ρ) and Lin's concordance correlation (ρc) coefficients and class average accuracy (CAA).Results: 4410 lumbar segments were analyzed. κ statistics showed moderate to substantial agreement in PG between the radiologist and SN depending on spinal level (range κ 0.63-0.77, all levels together 0.72; range CAA 45-68%, all levels 55%), slight to substantial agreement for SL (range κ 0.07-0.60, all levels 0.63; range CAA 47-57%, all levels 56%) and CCS (range κ 0.17-0.57, all levels 0.60; range CAA 35-41%, all levels 43%). SN tended to record more pathological features in PG than did the radiologist whereas the contrary was the case for CCS. SL showed an even distribution between methods.Conclusion: SN is a robust and reliable tool with the ability to grade degenerative features such as PG, SL or CCS in lumbar MRIs with moderate to substantial agreement compared to the current gold-standard, the radiologist. It is a valuable alternative for analyzing MRIs from large cohorts for diagnostic and research purposes. [ABSTRACT FROM AUTHOR]

The Swanson, Nolan, and Pelham scale version IV (SNAP-IV) is widely used to assess symptoms of attention deficit hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD) in children and adolescents. Nevertheless, there is insufficient data to support its use in preschool children. The study had three goals: First, to test the factorial validity of the three correlated-factors model of ADHD and ODD items of the SNAP-IV. Second, to investigate the measurement invariance of the items over time (6-month longitudinal interval) and by sex. Third, to investigate the convergent validity and method-specific influences on ADHD/ODD assessments with respect to multiple raters (parents/teachers) of children's symptoms. Participants were 618 preschool children (3.5-6 years) at baseline and 6-month follow-up. For model testing, we used confirmatory factor analysis for categorical observed variables. Method and trait effects were examined using the CT-C(M-1) model. The analyses showed partial measurement invariance over time and according to sex. Moreover, strong rater-specific effects were detected. The implication of the results for construct validation of the instrument and clinical assessment of ADHD and ODD traits are discussed. [ABSTRACT FROM AUTHOR]

Here we describe the development and evaluation of a survey instrument to assess the research suitability of veterinary electronic medical records (EMRs) through the conduct of two studies as part of the Dog Aging Project (DAP). In study 1, four reviewers used the instrument to score a total of 218 records in an overlapping matrix of pairs to assess inter-rater agreement with respect to appropriate format (qualification), identification match (verification), and record quality. Based upon the moderate inter-rater agreement with respect to verification and the relatively large number of records that were incorrectly rejected the instrument was modified and more specific instructions were provided. In study 2, a modified instrument was again completed by four reviewers to score 100 different EMRs. The survey scores were compared to a gold standard of board-certified specialist review to determine receiver operating curve statistics. The refined survey had substantial inter-rater agreement across most qualification and verification questions. The cut-off value identified had a sensitivity of 95 and 96% (by reviewer 1 and reviewer 2, respectively) and a specificity of 82% and 91% (by reviewer 1 and reviewer 2, respectively) to predict gold standard acceptance or rejection of the record. Using just qualification and verification questions within the instrument (as opposed to full scoring) minimally impacted sensitivity and specificity and resulted in substantial time savings in the review process.

BackgroundThe COVID-19 pandemic prompted the scientific community to share timely evidence, also in the form of pre-printed papers, not peer reviewed yet.PurposeTo develop an artificial intelligence system for the analysis of the scientific literature by leveraging on recent developments in the field of Argument Mining.MethodologyScientific quality criteria were borrowed from two selected Cochrane systematic reviews. Four independent reviewers gave a blind evaluation on a 1–5 scale to 40 papers for each review. These scores were matched with the automatic analysis performed by an AM system named MARGOT, which detected claims and supporting evidence for the cited papers. Outcomes were evaluated with inter-rater indices (Cohen's Kappa, Krippendorff's Alpha, s* statistics).ResultsMARGOT performs differently on the two selected Cochrane reviews: the inter-rater indices show a fair-to-moderate agreement of the most relevant MARGOT metrics both with Cochrane and the skilled interval scores, with larger values for one of the two reviews.Discussion and conclusionsThe noted discrepancy could rely on a limitation of the MARGOT system that can be improved; yet, the level of agreement between human reviewers also suggests a different complexity between the two reviews in debating controversial arguments. These preliminary results encourage to expand and deepen the investigation to other topics and a larger number of highly specialized reviewers, to reduce uncertainty in the evaluation process, thus supporting the retraining of AM systems.

Abstract Background In an inter-rater agreement study, if two raters tend to rate considering different aspects of the subject of interest or have different experience levels, a grey zone occurs among the levels of a square contingency table showing the inter-rater agreement. These grey zones distort the degree of agreement between raters and negatively impact the decisions based on the inter-rater agreement tables. In this sense, it is important to know how the existence of a grey zone impacts the inter-rater agreement coefficients to choose the most reliable agreement coefficient against the grey zones to reach out with more reliable decisions. Methods In this article, we propose two approaches to create grey zones in simulations setting and conduct an extensive Monte Carlo simulation study to figure out the impact of having grey zones on the weighted inter-rater agreement measures for ordinal tables over a comprehensive simulation space. Results The weighted inter-rater agreement coefficients are not reliable against the existence of grey zones. Increasing sample size and the number of categories in the agreement table decreases the accuracy of weighted inter-rater agreement measures when there is a grey zone. When the degree of agreement between the raters is high, the agreement measures are not significantly impacted by the existence of grey zones. However, if there is a medium to low degree of inter-rater agreement, all the weighted coefficients are more or less impacted. Conclusions It is observed in this study that the existence of grey zones has a significant negative impact on the accuracy of agreement measures especially for a low degree of true agreement and high sample and tables sizes. In general, Gwet’s AC2 and Brennan-Prediger’s κ with quadratic or ordinal weights are reliable against the grey zones.

This study combines language assessment processes and interlanguage analysis techniques to determine rater agreement and disagreement in assessing English article acquisition. Employing native English speaking and non-native English speaking raters, picture sequence narratives that were written by English as a Foreign Language (EFL) learners (n=97) were coded and scored for suppliance-in-obligatory context (SOC) and target-like utterance (TLU). Although the kappa statistic revealed a fair agreement between raters (0.17 – 0.33), content analysis methods revealed much higher agreement (88.29% - 94.07%). Furthermore, language background effects between the raters could not be substantiated however the results demonstrated a discernable disagreement pattern between them. Thus, the study recommends the inclusion of a foreign language teaching background as a factor for rater selection to minimize language background effects on rating language assessments.

The degree of inter‐rater agreement is usually assessed through κ‐type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some κ‐type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four κ‐type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies. [ABSTRACT FROM AUTHOR]

Objectives: We have developed two Dutch questionnaires to assess the shared decision-making (SDM) process in oncology; the iSHAREpatient and iSHAREphysician. In this study, we aimed to determine: scores, construct validity, test-retest agreement (iSHAREpatient), and inter-rater (iSHAREpatient-iSHAREphysician) agreement.Methods: Physicians from seven Dutch hospitals recruited cancer patients, and completed the iSHAREphysician and SDM-Questionnaire-physician version. Their patients completed the: iSHAREpatient, nine-item SDM-Questionnaire, Decisional Conflict Scale, Combined Outcome Measure for Risk communication And treatment Decision-making Effectiveness, and five-item Perceived Efficacy in Patient-Physician Interactions. We formulated, respectively, one (iSHAREphysician) and 10 (iSHAREpatient) a priori hypotheses regarding correlations between the iSHARE questionnaires and questionnaires assessing related constructs. To assess test-retest agreement patients completed the iSHAREpatient again 1-2 weeks later.Results: In total, 151 treatment decision-making processes with unique patients were rated. Dimension and total iSHARE scores were high both in patients and physicians. The hypothesis on the iSHAREphysician and 9/10 hypotheses on the iSHAREpatient were confirmed. Test-retest and inter-rater agreement were>.60 for most items.Conclusions: The iSHARE questionnaires show high scores, have good construct validity, substantial test-retest agreement, and moderate inter-rater agreement.Practice Implications: Results from the iSHARE questionnaires can inform both physician- and patient-directed efforts to improve SDM in clinical practice. [ABSTRACT FROM AUTHOR]