Search

Background The modified Thomas test (MTT) is commonly used to assess the flexibility of hip musculature, including the iliopsoas, rectus femoris, and tensor fascia latae. This measurement is important to include in a comprehensive musculoskeletal examination. However, existing research shows conflicting results regarding its reliability, particularly due to variations in controlling pelvic tilt during testing, which may lead to inaccurate measurements of hip extension when quantifying the test outcomes. Purpose/Hypothesis This study aimed to evaluate the intra- and inter-rater reliability of the Modified Thomas Test (MTT) in assessing hip flexor length using a goniometer. It was hypothesized that controlling for pelvic tilt would enhance the reliability of these measurements. Study Design Intra- and inter-rater reliability study Methods Sixty-four healthy individuals were recruited to participate in this study. The MTT was performed twice on each leg by both an experienced and a student physical therapist. Blinded goniometric measurements for hip extension range of motion (ROM) in the MTT position were taken with neutral pelvic tilt being enforced via palpation. A double-blind protocol was used where both examiners were unaware of each other's measurements and the goniometer was covered to blind the measuring therapist to the values as well. ROM values were entered into a Microsoft Excel spreadsheet and quantified using SPSS software. Statistical analysis included calculating Intraclass Correlation Coefficients (ICCs) and Standard Errors of Measurement (SEMs) using SPSS software. Results The study included 64 participants (mean age = 23.7 ± 4.34 years). The MTT demonstrated high intra-rater reliability (ICC = 0.911) and inter-rater reliability (ICC = 0.851). The SEMs indicated minimal variability around the mean scores. The average hip extension ROM measured was 5.43± 9.73 degrees. Conclusion These results suggest that the MTT is a reliable tool for assessing hip flexor length in clinical practice, particularly when pelvic tilt is controlled. These results have important implications for accurately testing orthopedic limitations that can contribute to low back, hip, and knee pain. Level of Evidence 3 [ABSTRACT FROM AUTHOR]

Transect surveys are widely used to quantify mismanaged plastic waste discarded to the environment. However, very few studies have quantified inter-observer reliability in environmental waste surveys. The aim of this study was to assess inter-observer reliability for a set of environmental waste indicators derived from transect surveys in slum areas of two sub-Saharan African cities. Pairs of observers independently recorded counts of scattered waste items, large waste piles, and burnt waste along 64 environmental transects in Kisumu, Kenya, and Greater Accra, Ghana. Concordance correlation coefficients were used to measure inter-observer reliability for derived indicators measuring mismanaged waste density, waste composition, waste origins of policy concern, and waste disposal practices. Evidence across all observers consistently showed that single-use disposable diapers, discarded Personal Protective Equipment, and bagged or bottled water are locally important constituents of mismanaged waste in both cities that should be addressed through urban waste management strategies. In both cities, there was generally excellent inter-observer reliability for density of burnt waste and density of large waste piles (concordance correlation coefficient > 0.9 for all but one observer pair), but weak to moderate inter-observer reliability for scattered waste density and densities of specific waste items such as disposable nappies and discarded Personal Protective Equipment. Our study shows that beach litter survey protocols can be adapted for use in slums. However, to generate robust estimates of scattered waste, its composition, and waste origins of policy concern in slums, assessment of inter-observer reliability should be incorporated into field team training. [ABSTRACT FROM AUTHOR]

Background: The study examined long-term lip appearance after Skoog's lip repair in unilateral cleft, lip, and palate (UCLP) patients. To include different views, three different panels respectively, rated level of lip appearance and desire for improvement. The secondary aim was to investigate the correlation between the self-reported satisfaction of UCLP patients and controls with the panel assessments. Methods: Cropped frontal photographs of the lips of 74 UCLP-treated individuals and 61 non-cleft individuals were assessed twice by three panels of five cleft surgeons (Professional panel), nine laymen (Layman panel), and five individuals with CLP (Cleft panel) using a web-based platform, at a mean follow-up of 37 years post-lip surgery. Results: All three panels rated the lip appearance of UCLP patients worse than that of the controls (p < 0.001). The Professional panel reported higher satisfaction levels and less desire to change lip appearance than the Layman and Cleft panels. Intra-rater agreements were, in general, moderate in the Professional panel ICC (0.57–0.82) and Laymen panel ICC (0.63–0.73) and poor in the Cleft panel ICC (0.21–0.36). The inter-rater agreement was, in general, poor in all panels. No significant correlation was found between the patient's Self-assessments and any of the panel assessments. Conclusions: All panels' rated lip appearance of adults born with UCLP treated with Skoog's technique as worse than that of non-cleft controls. The study highlights a significant discrepancy between the Professional panel's more favorable assessment and the more critical views of the laymen and cleft panels, contradicting the initial hypothesis of uniform perception across panels. The lack of correlation between patient self-assessment and panel assessments underscores the subjective nature of lip satisfaction, emphasizing the need for personalized patient care strategies in CLP treatment outcomes. Level of evidence: Not gradable. [ABSTRACT FROM AUTHOR]

Introduction/Aims Methods Results Discussion Limitations exist in evaluating mechanical detection thresholds (MDTs) due to a lack of dependable electronic instruments designed to assess Aβ fibers and measure MDTs across different body areas. This study aims to evaluate the test–retest and inter‐rater reliability of the cutaneous mechanical stimulator (CMS), an electronic tactile stimulator, in quantifying MDTs.Using a test–retest design, participants underwent assessments of MDTs using Semmes–Weinstein monofilaments (SWM) and the CMS. This study included 27 healthy volunteers (mean age 24.07 ± 3.76 years). Two raters assessed MDTs using SWM and the CMS at two stimulation sites (the left hand and foot) in two experimental sessions approximately 2 weeks apart.MDTs using SWM and the CMS showed excellent reliability on the hand (intraclass correlation coefficient [ICC] = .84) and foot (ICC = .90). A comparison of results obtained at the two sessions showed that MDTs on the hand displayed good reliability for both SWM (ICC = .63) and the CMS (ICC = .73), whereas MDTs on the foot displayed fair reliability for SWM (ICC = .50) and the CMS (ICC = .42). MDTs exhibited good inter‐rater reliability with SWM (ICC = .66) and excellent inter‐rater reliability with the CMS (ICC = .82) on the hand, as well as showing fair inter‐rater reliability with SWM (ICC = .53) and good inter‐rater reliability with the CMS (ICC = .60) on the foot.The CMS showed superior inter‐rater reliability, indicating its potential as a valuable tool for assessing tactile sensitivity in research and clinical settings. [ABSTRACT FROM AUTHOR]

Autistic youth are more likely to experience depression than their non-autistic peers, yet research on risk and protective factors to depression in this population is limited. Behavioral activation (i.e., prioritizing and engaging in meaningful activities), including social activities, is an important mechanism in the pathway to depression in non-autistic youth that is understudied in autism. Ratings of youth depressive symptoms and behavioral and social activation at one timepoint from 100 autistic youth without intellectual disability and 100 of their caregivers were analyzed. The study aims were to examine caregiver and youth ratings of youth internalizing symptoms and behavioral and social activation, inter-rater reliability on study variables, and associations between depressive symptoms and behavioral and social activation in autistic youth by rater. Results revealed significant differences in youth and caregiver ratings on all variables and inter-rater reliability ranged from poor to moderate. Across both raters, more severe anxiety symptoms and lower behavioral activation were associated with more severe depressive symptoms; social activation, specifically the number of friends youth have, was significant in caregiver ratings only. Findings can be leveraged to enhance risk stratification and intervention efforts for autistic youth experiencing depression. [ABSTRACT FROM AUTHOR]

In this study, using data from evaluations of 20 companies by seven raters in a corporate award on the theme of human resource management, I examined the consistency of company evaluations by a small number of experts in different fields and the criterion-related validity of the mean evaluations per company. The evaluation was conducted in the order of documents and interviews, and each was rated on five items. The intraclass correlations showed that the evaluations of "philosophy," "innovation," "leadership," and "motivation" were consistent across raters and that the interview evaluations were more consistent than the document evaluations. Regarding "personnel" evaluations, significant consistency was found only for document evaluations. The correlations between the mean evaluations and indices of application documents showed that companies with more text in their application documents tended to be evaluated more highly. It was confirmed that the few experts' evaluations of the companies had a moderate degree of consistency and validity; however, it was also suggested that some combinations of evaluation items and evaluation materials may not have resulted in appropriate evaluations. [ABSTRACT FROM AUTHOR]

Objectives. This study aimed to investigate the consistency between results of the American Conference for Governmental Occupational Hygienists (ACGIH) threshold limit value (TLV) for hand activity and proposed action levels of objective measurements in risk assessments of work-related musculoskeletal disorders. Methods. Wrist velocities and forearm muscular load were measured for 11 assemblers during one working day. Simultaneously, each assembler's hand activity level (HAL) during three sub-cycles was rated twice on two separate occasions by two experts, using a HAL scale. Arm/hand exertion was also rated by the assemblers themselves using a Borg scale. In total, 66 sub-cycles were assessed and assigned to three exposure categories: A) below ACGIH action limit (AL) (green); B) between AL and TLV (yellow); and C) above TLV (red). The median wrist velocity and the 90th percentile of forearm muscular load obtained from the objective measurements corresponding to the sub-cycles were calculated and assigned to two exposure categories: A) below or C) above the proposed action level. Results. The agreement between ACGIH TLV for hand activity and the proposed action level for wrist velocity was 87%. Conclusions. The proposed action level for wrist velocity is highly consistent with the TLV. Additional studies are needed to confirm the results. [ABSTRACT FROM AUTHOR]

Background: Fixed-frame dynamometry systems are used worldwide to assess isometric strength in both general and athletic populations. There is currently a paucity of published work where reliability estimates for fixed-frame dynamometry systems have been estimated. The aim of this study was to determine the inter-and intra-rater reliability of the KangaTech (KT360) fixed frame dynamometry system when measuring maximal isometric strength of the knee flexor muscles. Study Design: Inter and intra-rater reliability single cohort study. Methods: Twenty healthy university-level athletes (age= 21.65 ± 3 years, weight= 74.465 ± 30kg, height= 170.1 ± 7.0cm) took part in two testing sessions where two raters collected data during a 90° hip and knee flexion protocol. Participants performed each test twice, building to a maximal isometric contraction holding over a 5 second period with 30 second rest between sets. Data were checked for normality using a Shapiro-Wilk test. Intraclass correlation coefficient (ICC), standard error of measurement (SEM) and minimal detectable change (MDC) were calculated. Finally, a Bland-Altman analysis was used to determine the levels of agreement for intra-and inter-rater measurements. Results: High levels of agreement were demonstrated between left and right knee flexion as 95% of the differences were less than two standard deviations away from the mean. 'Almost perfect' intraclass correlation coefficient (ICC) values were demonstrated (Knee flexion: Inter-rater: Left, 0.99; Right, 0.99; Intra-rater: Left, 0.99; Right:0.99). Standard error of measurement (SEM) for inter-and intra-rater strength ranged from 0.26-0.69 kg, SEM% ranged from 1.34-2.71% and minimal detectable change (MDC) ranged from 1.14-2.31kg. Conclusion: Overall, high level of inter-and intra-rater reliability were demonstrated when testing maximal isometric knee flexion. Therefore, the KT360 fixed frame dynamometry system may be considered a viable tool for measuring maximal isometric contraction of the knee flexors when repeat measures are required in clinical settings. Level of Evidence: 3b [ABSTRACT FROM AUTHOR]

Introduction: Obsessive-compulsive disorder (OCD) is characterized by underlying obsessions and yielding/nonyielding compulsions, which often leads to significant levels of distress for the individual. Recently, obsessive-compulsive symptoms have been grouped into various symptom dimensions. The Dimensional Yale-Brown Obsessive-Compulsive Scale (DY-BOCS) gives a comprehensive picture regarding symptom severity in each individual dimension of OCD, while also providing a global severity score. Aims and Objectives: The psychometric properties of the Bengali adaptation of DY-BOCS have been assessed in this study. Material and Methods: A cross-sectional study was performed on 80 individuals suffering from OCD as per the ICD-10 DCR criteria. The Bengali version of the DY-BOCS and other scales for OCD, anxiety, and depression were administered to all of them. Inter-rater reliability was assessed for 34 individuals, and agreement between self and clinician-rated version of DY-BOCS was studied for 60 individuals. Internal consistency for items in a particular dimension of DY-BOCS, convergent validity, and divergent validity were also assessed. Results: Internal consistency was found to be excellent (Cronbach’s α > 0.90 in all domains), and inter-rater reliability was found to be good to excellent for all the six dimensions of the DY-BOCS (intraclass correlation values >0.80 for aggressive and miscellaneous domains and >0.90 in all other domains). High correlation was found between self and clinician-rated versions of DY-BOCS for the age group 18–35 years, the highest correlations being found for the Contamination (r = 0.82) and Aggressive domains (r = 0.75), while it was found to decrease in the older age group. Convergent validity and divergent validity were found to be satisfactory. Conclusion: The psychometric attributes of the Bengali version of DY-BOCS appear to be satisfactory and thus can be reasonably administered to the population having Bengali as their mother tongue.

Prechtl’s General Movement Assessment (GMA) is a qualitative video assessment that is an early predictor of cerebral palsy (CP) in infants. GMA raters undergo two levels of training: basic and advanced. Using a retrospective study design, we evaluated the impact of the GMA training level on three key measures: inter-rater reliability (IRR), predictability for a later CP diagnosis, and generalizability for both medically trained and non-medically trained raters. As part of a larger study on former level-IV neonatal intensive care unit (NICU) infants, participants had a GMA video at 3–4 months of corrected age with parental consent. Four GMA raters with basic training interpreted the videos. Subsequently, two raters underwent advanced training and reinterpreted the videos. We evaluated IRR with Gwet’s AC1 coefficient, CP prediction with logistic regression, and generalizability with Gwet’s Linearization method and McNemar’s test. Following basic GMA training, all four raters had moderate to almost perfect agreement. The CP predictability of the raters’ individual GMA scores ranged from 73% to 81%. Advanced GMA training significantly improved IRR and enhanced CP predictability. GMA rater proficiency was similar regardless of medical training. Advanced GMA training improved readers’ capabilities to correctly identify abnormal movement patterns and aid in early CP diagnosis.

Introduction: Painful Diabetic Peripheral Neuropathy (PDPN) is associated with pain and disturbed sensory symptoms. Altered Pressure Pain Threshold (PPT) in PDPN often leads to complications of diabetic foot and consequent amputations. Early detection of altered PPT can prevent future complications and reduce mortality rates. PPT may be determined with a pressure algometer, which measures the pressure and/or force at which the first perception of pain begins. The cost of algometers frequently prevents them from being used in clinical and research settings. An affordable and dependable algometer would be a valuable tool in PDPN, where health costs are already 20% higher than those of diabetic controls. Aim: To evaluate the test-retest and inter-rater reliability of a low-cost digital pressure algometer in individuals suffering from PDPN. Materials and Methods: This quantitative cross-sectional study was conducted for four months at Metro Heart Institute with Multispeciality Hospital, Faridabad, Haryana, India. PPT of 30 patients with PDPN aged 50-70 years (mean age 61.53±5.84 years) was collected twice by one rater (R1) after a gap of 24 hours. Another rater (R2) repeated the first reading at similar points on both feet. PPT was noted at the dorsum, 2nd, and 3rd metatarsal on the plantar surface of the foot. The main outcome measurements were the Intraclass Correlation Coefficient (ICC), Standard Error of Measurement (SEM), Minimal Detectable Change (MDC), and using the Bland-Altman approach, measurement bias was evaluated. Results: The ICC for test-retest reliability for the dorsal right and left foot was 0.85 and 0.83, respectively. The ICC for Plantar 2nd metatarsal right and left was 0.86 and 0.89, respectively. The ICC for the plantar third metatarsal right and left foot was 0.85 and 0.81, respectively. The inter-rater reliability ICC values varied from 0.63 to 0.87. Bland-Altman plots showed acceptable levels of agreement. Conclusion: The digital algometer showed good test-retest and moderate inter-rater reliability in patients with PDPN.

OBJECTIVES: The use of analytic rubrics in assessing and grading students' performance has become more prominent among instructors due to its reliability and validity in ensuring consistency in student evaluation. However, there is limited evidence demonstrating the consistency of examiner judgments between analytic marking rubrics and holistic marking rubrics. METHODS: Therefore, we aimed to compare the consistency of marks given using holistic marking methods and analytic rubrics at an Australian university by analyzing the mean mark differences and number of adjudications between two rubric types as well as the inter-rater reliability between two assessors. We analyzed all scores for project manuscripts between 2016 and 2021 for Honours medical students. We compared the mean mark differences graded using the Kruskal–Wallis test and Welch t -test. We used chi-squared tests to compare the frequency of adjudications for each rubric type. We assessed interrater reliability by comparing the marks between the two examiners utilizing Pearson correlation. RESULTS: We found that analytic rubrics have lower mean mark differences and fewer adjudicators are required. We showed a strong positive association between the consistency of marks given and the use of analytic rubrics when compared to holistic marking. Pearson correlation showed a low but stronger correlation between marks awarded by the two assessors when analytic rubrics were used (r = 0.36), compared to holistic marking rubrics (r = 0.24). CONCLUSIONS: Our findings suggest that the use of analytic rubrics may increase the consistency and reliability between two independent examiners in marking medical students' work. [ABSTRACT FROM AUTHOR]

Introduction: Obsessive-compulsive disorder (OCD) is characterized by underlying obsessions and yielding/nonyielding compulsions, which often leads to significant levels of distress for the individual. Recently, obsessive-compulsive symptoms have been grouped into various symptom dimensions. The Dimensional Yale-Brown Obsessive-Compulsive Scale (DY-BOCS) gives a comprehensive picture regarding symptom severity in each individual dimension of OCD, while also providing a global severity score. Aims and Objectives: The psychometric properties of the Bengali adaptation of DY-BOCS have been assessed in this study. Material and Methods: A cross-sectional study was performed on 80 individuals suffering from OCD as per the ICD-10 DCR criteria. The Bengali version of the DY-BOCS and other scales for OCD, anxiety, and depression were administered to all of them. Inter-rater reliability was assessed for 34 individuals, and agreement between self and clinician-rated version of DY-BOCS was studied for 60 individuals. Internal consistency for items in a particular dimension of DY-BOCS, convergent validity, and divergent validity were also assessed. Results: Internal consistency was found to be excellent (Cronbach's α > 0.90 in all domains), and inter-rater reliability was found to be good to excellent for all the six dimensions of the DY-BOCS (intraclass correlation values >0.80 for aggressive and miscellaneous domains and >0.90 in all other domains). High correlation was found between self and clinician-rated versions of DY-BOCS for the age group 18-35 years, the highest correlations being found for the Contamination (r = 0.82) and Aggressive domains (r = 0.75), while it was found to decrease in the older age group. Convergent validity and divergent validity were found to be satisfactory. Conclusion: The psychometric attributes of the Bengali version of DY-BOCS appear to be satisfactory and thus can be reasonably administered to the population having Bengali as their mother tongue. [ABSTRACT FROM AUTHOR]

Background: Dysphagia is commonly evaluated using videofluoroscopy (VFS). As its ratings are usually subjective normal‐abnormal ratings, objective measurements have been developed. We compared the inter‐rater reliability of the usual VFS ratings to the objective measurement VFS ratings and evaluated their clinical relevance. Methods: Two blinded raters analyzed the subjective normal‐abnormal ratings of 77 patients' VFS. Two other blinded raters analyzed the objective measurements of pharyngeal aerated area with bolus held in the oral cavity (PAhold), the pharyngeal area of residual bolus during swallowing (PAmax), the pharyngeal constriction ratio (PCR), the maximum pharyngoesophageal segment opening (PESmax), pharyngoesophageal segment opening duration (POD), airway closure duration (ACD), and total pharyngeal transit time (TPT). We evaluated the inter‐rater agreement in the subjective ratings and the objective measurements. Clinical utility analysis compared the measurements with the VFS findings of pharyngeal phase abnormality, penetration/aspiration, and cricopharyngeal relaxation. Results: In the pharyngeal findings, the subjective analysis inter‐rater agreement was mainly moderate to strong. The strongest agreements were on the pharyngeal residues and penetration/aspiration findings. The objective measurements had fair to good inter‐rater agreement. Clinical utility analysis found statistically significant connections between TPT and pharyngeal phase abnormality, normal PCR and lack of penetration/aspiration, and normal PESmax and normal cricopharyngeal relaxation. Conclusions: The subjective analysis had moderate to strong inter‐rater agreement in the pharyngeal VFS findings, especially concerning pharyngeal residues and penetration/aspiration detection, reflecting the efficacy and safety of swallowing. The objective measurements had fair to good inter‐observer reproducibility and could thus improve the reliability of VFS diagnostics. Level of evidence: 4. [ABSTRACT FROM AUTHOR]

AbstractObjectiveMethodResultsConclusions\nHIGHLIGHTSSuicide risk assessments are obligatory when patients express a death wish in clinical practice. Yet, suicide risk estimates based on unguided risk assessments have been shown to be of low reliability. Since generalizability of previous studies is limited, the current study aimed to assess inter-rater and intra-rater reliability of risk estimates conducted by psychotherapists and psychology students using written case vignettes.In total, N = 256 participants (psychology students, psychotherapists) were presented with 24 case vignettes describing patients at either low, moderate, severe or extreme risk of suicide. Participants were asked to assign a level of risk to each single vignette at a baseline assessment and again at a follow-up assessment two weeks later.Risk estimates showed a low inter-rater reliability, both for students (AC1 = .35) and for psychotherapists (AC1 = .44). Intra-rater reliability was moderate for psychotherapists (AC1 = .59) and rather low for psychology students (AC1 = .47). In general, intra- and intra-rater reliability were highest for vignettes displaying “low” and “extreme” risk.The results highlight that the reliability of unguided suicide risk assessments is questionable. Standardized risk assessment protocols are therefore recommended. Nonetheless, even reliable risk estimation does not imply predictive validity of risk estimates for future suicidal behavior.Suicide risk estimates have been shown to be of low reliabilitySuicide risk estimates by psychotherapists and students also showed low inter-rater and intra-rater reliability in the current studyReliable risk estimation does not imply predictive validity of risk estimates for future suicidal behavior [ABSTRACT FROM AUTHOR]

Background: Arthroscopy of the temporomandibular joint (TMJ) plays a long-established role in the diagnostics and therapy of patients suffering from arthrogenic temporomandibular disorders (TMDs), which do not respond adequately to conservative/non-invasive therapy. However, the interpretation of arthroscopic findings remains challenging. This study investigates the reliability and variability of assessing arthroscopic views of pathologies in patients with TMDs by non-specialists in arthroscopy and whether a standardized assessment tool may improve correctness. Methods: Following a comprehensive one-semester lecture, dental students in the clinical stage of education were asked to rate 25 arthroscopic views (freeze images and corresponding video clips) regarding the severity of synovitis, adhesions, and degenerative changes on a scale of 0–10 (T1). The results were compared to ratings stated by two European-board-qualified academic OMF surgeons. In a second round (T2), the students were asked to repeat the ratings using a 10-point rating scheme. Results: With regard to all three subcategories, congruency with the surgeons' results at T1 was at a low level (p < 0.05 in 19/75 cases) and even decreased at T2 after the implementation of the TMDs-SevS (p < 0.05 in 38/75 cases). For both T1 and T2, therefore, the inter-rater agreement was at a low level, showing only a slight agreement for all three subcategories (Fleiss' Kappa (κ) between 0.014 and 0.099). Conclusions: The judgement of the arthroscopic pathologies of the TMJ remains an area of temporomandibular surgery that requires wide experience and training in TMDs to achieve expertise in TMJ arthroscopic assessments, which cannot be transferred by theoretical instruction alone. [ABSTRACT FROM AUTHOR]

AbstractAimMethodsResultsConclusionThe Test of Gross Motor Development Third Edition (TGMD-3) is used to assess the development of fundamental movement skills in children from 3 to 10 years old. This study aimed to evaluate the intra-rater, inter-rater, and test-retest reliability and to determine the minimal detectable change (MDC) value of the TGMD-3 in children with developmental coordination disorder (DCD).The TGMD-3 was administered to 20 children with DCD. The child’s fundamental movement skills were recorded using a digital video camera. Reliability was assessed at two occasions by three raters using the generalizability theory.The TGMD-3 demonstrates good inter-rater reliability for the locomotor skills subscale, the ball skills subscale, and the total score (φ = 0.77 − 0.91), while the intra-rater reliability was even higher (φ = 0.94 − 0.97). Test-retest reliability was also shown to be good (φ = 0.79-0.93). The MDC95 was determined to be 10 points.This study provides evidence that the TGMD-3 is a reliable test when used to evaluate fundamental movement skills in children with DCD and suggests that an increase of 10 points represents a significant change in the motor function of a child with DCD. [ABSTRACT FROM AUTHOR]

An accurate assessment of intimate partner violence (IPV) is crucial to guide public policy and intervention. The Conflict Tactic Scales Revised (CTS-2) is one of the most widely used instruments to do so. Despite its good psychometric properties, research on interpartner agreement has pointed to low-to-moderate estimates, which generated some concerns about the validity of the results obtained through single-partner reports. This cross-sectional study introduces indexes that have not previously been used to assess interpartner agreement. Both partners' reports on perpetration and victimization were analyzed in a community sample of 268 different-sex couples. Our results generally pointed to better agreement levels on IPV occurrence than frequency, suggesting that the proxy method (i.e., using a single-partner report) could be a reliable method for assessing IPV occurrence but not its frequency in this population. Findings are discussed as well as the advantages and constraints of different IPV assessment practices. [ABSTRACT FROM AUTHOR]

Background: Chronic leg ulcers present a global challenge in healthcare, necessitating precise wound measurement for effective treatment evaluation. This study is the first to validate the "split-wound design" approach for wound studies using objective measures. We further improved this relatively new approach and combined it with a semi-automated wound measurement algorithm. Method: The algorithm is capable of plotting an objective halving line that is calculated by splitting the bounding box of the wound surface along the longest side. To evaluate this algorithm, we compared the accuracy of the subjective wound halving of manual operators of different backgrounds with the algorithm-generated halving line and the ground truth, in two separate rounds. Results: The median absolute deviation (MAD) from the ground truth of the manual wound halving was 2% and 3% in the first and second round, respectively. On the other hand, the algorithm-generated halving line showed a significantly lower deviation from the ground truth (MAD = 0.3%, p < 0.001). Conclusions: The data suggest that this wound-halving algorithm is suitable and reliable for conducting wound studies. This innovative combination of a semi-automated algorithm paired with a unique study design offers several advantages, including reduced patient recruitment needs, accelerated study planning, and cost savings, thereby expediting evidence generation in the field of wound care. Our findings highlight a promising path forward for improving wound research and clinical practice. [ABSTRACT FROM AUTHOR]

Bone marrow edema-like lesions (BMEL) in the knee have been linked to the symptoms and progression of osteoarthritis (OA), a highly prevalent disease with profound public health implications. Manual and semi-automatic segmentations of BMELs in magnetic resonance images (MRI) have been used to quantify the significance of BMELs. However, their utilization is hampered by the labor-intensive and time-consuming nature of the process as well as by annotator bias, especially since BMELs exhibit various sizes and irregular shapes with diffuse signal that lead to poor intra- and inter-rater reliability. In this study, we propose a novel unsupervised method for fully automated segmentation of BMELs that leverages conditional diffusion models, multiple MRI sequences that have different contrast of BMELs, and anomaly detection that do not rely on costly and error-prone annotations. We also analyze BMEL segmentation annotations from multiple experts, reporting intra-/inter-rater variability and setting better benchmarks for BMEL segmentation performance. [ABSTRACT FROM AUTHOR]

Introduction: Painful Diabetic Peripheral Neuropathy (PDPN) is associated with pain and disturbed sensory symptoms. Altered Pressure Pain Threshold (PPT) in PDPN often leads to complications of diabetic foot and consequent amputations. Early detection of altered PPT can prevent future complications and reduce mortality rates. PPT may be determined with a pressure algometer, which measures the pressure and/or force at which the first perception of pain begins. The cost of algometers frequently prevents them from being used in clinical and research settings. An affordable and dependable algometer would be a valuable tool in PDPN, where health costs are already 20% higher than those of diabetic controls. Aim: To evaluate the test-retest and inter-rater reliability of a low-cost digital pressure algometer in individuals suffering from PDPN. Materials and Methods: This quantitative cross-sectional study was conducted for four months at Metro Heart Institute with Multispeciality Hospital, Faridabad, Haryana, India. PPT of 30 patients with PDPN aged 50-70 years (mean age 61.53±5.84 years) was collected twice by one rater (R1) after a gap of 24 hours. Another rater (R2) repeated the first reading at similar points on both feet. PPT was noted at the dorsum, 2nd, and 3rd metatarsal on the plantar surface of the foot. The main outcome measurements were the Intraclass Correlation Coefficient (ICC), Standard Error of Measurement (SEM), Minimal Detectable Change (MDC), and using the Bland-Altman approach, measurement bias was evaluated. Results: The ICC for test-retest reliability for the dorsal right and left foot was 0.85 and 0.83, respectively. The ICC for Plantar 2nd metatarsal right and left was 0.86 and 0.89, respectively. The ICC for the plantar third metatarsal right and left foot was 0.85 and 0.81, respectively. The inter-rater reliability ICC values varied from 0.63 to 0.87. Bland-Altman plots showed acceptable levels of agreement. Conclusion: The digital algometer showed good test-retest and moderate inter-rater reliability in patients with PDPN. [ABSTRACT FROM AUTHOR]

Age assessment of the living is a fundamental procedure in the process of human identification, in order to guarantee fair treatment of individuals, which has ethical, civil, legal, and medical repercussions. The careful selection of the appropriate methods requires evaluation of several parameters: accuracy, precision of the method, as well as its reproducibility. The approach proposed by Mincer et al. adapted from Demirjian et al. exploring third molar mineralisation, is one of the most frequently considered for age estimation of the living. Thus, this work aims to assess potential bias in the data collection when applying the classification stages for dental mineralisation adapted by Mincer et al. A total of 102 orthopantomographs, of clinical origin, belonging to individuals aged between 12 and 25 years (⁠|$ \bar{\textit x} $| = 20.12 years, SD = 3.49 years; 65 females, 37 males, all of Portuguese nationality) were included and a retrospective analysis performed by five observers with different levels of experience (high, average, and basic). The performance and agreement between five observers were evaluated using Weighted Cohen's Kappa and the Intraclass Correlation Coefficient. To access the influence of impaction on third molar classification, variables were tested using ordinal logistic regression Generalised Linear Model. It was observed that there were variations in the number of teeth identified among the observers, but the agreement levels ranged from moderate to substantial (0.4–0.8). Upon closer examination of the results, it was observed that although there were discernible differences between highly experienced observers and those with less experience, the gap was not as significant as initially hypothesised, and a greater disparity between the classifications of the upper (0.24–0.49) and lower third molars (>0.55) was observed. When bone superimposition is present, the classification process is not significantly influenced; however, variation in teeth angulation affects the assessment. The results suggest that with an efficient preparation, the level of experience as a factor can be overcome. Mincer and colleague's classification system can be replicated with ease and consistency, even though the classification of upper and lower third molars presents distinct challenges. [ABSTRACT FROM AUTHOR]

Purpose: Relative performance evaluation (RPE) is a widely practiced employee appraisal process in the services industry. In a global delivery model, teams are spread across different geographical locations. The team members work on various tasks under the guidance of different managers and at times under more than one manager for performing the same task. Such complexities make RPE of the team members quite challenging. The paper proposes a methodical step-by-step approach to simplify the evaluation process without compromising on the rigour. Design/methodology/approach: RPE has followed three different approaches. First is the traditional way, wherein evaluators had a common meeting to discuss and arrive at relative evaluation and ranking of members of the peer group employees. In the second, the number of evaluators and employees in a peer group were split in to 2 subgroups. The evaluators provided independent ratings and rankings. Simple mathematical tool then derived the combined ranking. In the third approach, each evaluator evaluated each employee in the peer group and provided the relative ranking for each employee. Again, mathematical tools provided the final ranking considering inputs from all evaluators. All the three evaluation approaches were analysed through an inter-rater agreement method. Findings: All the three approaches for evaluation provided similar results giving confidence that less time-consuming methods could be adopted by evaluators without compromising on the rigour of the evaluation. The outcome of the exercise proved effective as the complaints reaching the ombudsmen reduced as compared to the earlier years. Considerable evaluation time was also saved. The study described in this paper is carried out in a non-unionized, Indian private sector services firm. Its effectiveness in other set ups is yet to be tested. Research limitations/implications: The research is carried out in the Indian Engineering services firm operating in the Knowledge based sector. Though study results are encouraging, the adaptability of methodology across different sectors and geographies is yet to be tested. More broad based studies are needed to evaluate suitability across firms and regions. Practical implications: Relative evaluation exercise is challenging for evaluators. Although openness in evaluation is desired, it also makes evaluators uncomfortable in appearing to be taking sides or being opposing a candidate's ranking. The proposed approach brings in anonymity to each evaluator without scarifying individual evaluation. Social implications: The proposed methodology can be deployed across different services industries as the proposed methodology is business domain agnostic. It can be easily ported and tailored to align with an individual organization's evaluation philosophy. The suitability and effectiveness of the method can be studied under various types of firms like manufacturing, private, public, NGO, labour oriented, etc. As the proposed method reduces efforts, the stake holders can focus on understanding the relation between employee performance measurement, employee engagement, and long-term outcomes related to employee performance evaluation. Originality/value: The proposed employee evaluation method leverages inter-rater reliability and agreement tool as a consensus approach to the relative performance ranking exercise. Such an approach to relative performance ranking is original as no prior studies with such an approach are found in the existing Literature. [ABSTRACT FROM AUTHOR]

Most measures of agreement are chance-corrected. They differ in three dimensions: their definition of chance agreement, their choice of disagreement function, and how they handle multiple raters. Chance agreement is usually defined in a pairwise manner, following either Cohen's kappa or Fleiss's kappa. The disagreement function is usually a nominal, quadratic, or absolute value function. But how to handle multiple raters is contentious, with the main contenders being Fleiss's kappa, Conger's kappa, and Hubert's kappa, the variant of Fleiss's kappa where agreement is said to occur only if every rater agrees. More generally, multi-rater agreement coefficients can be defined in a g-wise way, where the disagreement weighting function uses g raters instead of two. This paper contains two main contributions. (a) We propose using Fréchet variances to handle the case of multiple raters. The Fréchet variances are intuitive disagreement measures and turn out to generalize the nominal, quadratic, and absolute value functions to the case of more than two raters. (b) We derive the limit theory of g-wise weighted agreement coefficients, with chance agreement of the Cohen-type or Fleiss-type, for the case where every item is rated by the same number of raters. Trying out three confidence interval constructions, we end up recommending calculating confidence intervals using the arcsine transform or the Fisher transform. [ABSTRACT FROM AUTHOR]

Background and Objectives: Muscle properties are critical for performance and injury risk, with changes occurring due to physical exertion, aging, and neurological conditions. The MyotonPro device offers a non-invasive method to comprehensively assess muscle biomechanical properties. This systematic review evaluates the reliability of MyotonPro across various muscles for diagnostic purposes. Materials and Methods: Following PRISMA guidelines, a comprehensive literature search was conducted in Medline (PubMed), Ovid (Med), Epistemonikos, Embase, Cochrane Library, Clinical trials.gov, and the WHO International Clinical Trials platform. Studies assessing the reliability of MyotonPro across different muscles were included. A methodological quality assessment was performed using established tools, and reviewers independently conducted data extraction. Statistical analysis involved summarizing intra-rater and inter-rater reliability measures across muscles. Results: A total of 48 studies assessing 31 muscles were included in the systematic review. The intra-rater and inter-rater reliability were consistently high for parameters such as frequency and stiffness in muscles of the lower and upper extremities, as well as other muscle groups. Despite methodological heterogeneity and limited data on specific parameters, MyotonPro demonstrated promising reliability for diagnostic purposes across diverse patient populations. Conclusions: The findings suggest the potential of MyotonPro in clinical assessments for accurate diagnosis, treatment planning, and monitoring of muscle properties. Further research is needed to address limitations and enhance the applicability of MyotonPro in clinical practice. Reliable muscle assessments are crucial for optimizing treatment outcomes and improving patient care in various healthcare settings. [ABSTRACT FROM AUTHOR]

Radiologists today play a central role in making diagnostic decisions and labeling images for training and benchmarking artificial intelligence (AI) algorithms. A key concern is low inter-reader reliability (IRR) seen between experts when interpreting challenging cases. While team-based decisions are known to outperform individual decisions, inter-personal biases often creep up in group interactions which limit nondominant participants from expressing true opinions. To overcome the dual problems of low consensus and interpersonal bias, we explored a solution modeled on bee swarms. Two separate cohorts, three board-certified radiologists, (cohort 1), and five radiology residents (cohort 2) collaborated on a digital swarm platform in real time and in a blinded fashion, grading meniscal lesions on knee MR exams. These consensus votes were benchmarked against clinical (arthroscopy) and radiological (senior-most radiologist) standards of reference using Cohen's kappa. The IRR of the consensus votes was then compared to the IRR of the majority and most confident votes of the two cohorts. IRR was also calculated for predictions from a meniscal lesion detecting AI algorithm. The attending cohort saw an improvement of 23% in IRR of swarm votes (k = 0.34) over majority vote (k = 0.11). Similar improvement of 23% in IRR (k = 0.25) in 3-resident swarm votes over majority vote (k = 0.02) was observed. The 5-resident swarm had an even higher improvement of 30% in IRR (k = 0.37) over majority vote (k = 0.07). The swarm consensus votes outperformed individual and majority vote decision in both the radiologists and resident cohorts. The attending and resident swarms also outperformed predictions from a state-of-the-art AI algorithm.

Abstract Background The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses. Methods This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters. Results The Cohen’s Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa. Conclusion Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.’s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries.

Introduction: The British Thyroid Association Ultrasound-classification is a risk stratification model which grades thyroid nodules in U2–5 based on their sonographic appearance. Existence of variability between the ultrasound operators when U-scoring is reported in the literature with some evidence found in the author's department. The aim of this study was to investigate whether there is significant disagreement in the department and identify potential reasons for variability. Methods: Eight operators, radiologists and sonographers, were recruited to grade 33 TNs and answer a tick box questionnaire using the British Thyroid Association lexicon. The inter-operator variability for the U-categories, indication for fine-needle aspiration biopsy and ultrasound features was assessed using Fleiss' kappa and Gwet-AC1. The operators' accuracy was measured against the most experienced operator in the department using Cohen's kappa and percentage agreement. Results: Fair agreement (Fleiss' K = 0.21) was obtained between the participants when U-scoring (U2–5). Fair-to-moderate agreement was noted between sonographers (K = 0.40). Significant variability was demonstrated between radiologists (p > 0.05). Indication for fine-needle aspiration biopsy reached fair to almost substantial agreement (radiologists' AC1 = 0.34, sonographers' AC1 = 0.58, overall AC1 = 0.41). No significant variability measured for echogenicity (K = 0.29), composition (K = 0.33), shape (K = 0.58), margin (K = 0.45), halo (K = 0.34) and vascularity (K = 0.44). Accuracy reached fair agreement (mean Cohen's K = 0.29) and moderate agreement (mean AC1 = 0.53) for the U-categories and fine-needle aspiration biopsy, respectively. Radiologists demonstrated lower accuracy. Conclusion: No significant inter-rater variability in U-scoring or recommending fine-needle aspiration biopsy was demonstrated between all the operators in the department. Radiologists showed significant variability in U-scoring and lower accuracy. Reliability and accuracy could be improved by addressing those problematic categories and features identified with this study. [ABSTRACT FROM AUTHOR]

The objective of this study is to determine the inter-rater reliability of the Pizzi Health and Wellness Assessment (PHWA) by comparing the consistency in scores between clients and their caregivers in the following areas of participation: social, physical, family, occupational, mental/emotional, and spiritual. A retrospective inter-rater correlational design was used to analyze the agreement of scores from a convenience sample consisting of two groups: clients with disabilities (n = 19) and their healthy caregivers (n = 19). Inter-rater reliability was calculated using correlations for the PHWA as a whole, and for the current level of participation and wishing to improve participation subsections. Inter-rater reliability as calculated by an Intraclass Correlation Coefficient, and either the Pearson or Spearman rho correlation and found to be reliable between clients and caregivers (rICC =.636, p <.001; rho =.642, p <.001). More specifically, current level of participation demonstrated acceptable reliability (rICC =.513, p <.001; r =.521, p <.001) as did wishing to improve participation (rICC =.689, p <.001; r =.725, p <.001). This supports the PHWA as a clinically relevant health and wellness occupational therapy assessment. [ABSTRACT FROM AUTHOR]

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment of major depression and binge eating disorder or bulimia nervosa (N = 212) to illustrate the limits of the intraclass coefficient. Randomly selected therapy sessions were rated for therapeutic alliance quality by independent observers using the well-validated Vanderbilt Psychotherapy Process Scale and Vanderbilt Therapeutic Alliance Scale. Scores on subscales related to therapist behavior were restricted, indicating consistent alliance-supportive actions. Inter-rater reliability estimates were low despite high agreement between raters, however. The use of Bland–Altman plots to visualize agreement and data spread is suggested as a useful tool for researchers, consistent with the ideal of exploring reliability from a number of perspectives. [ABSTRACT FROM AUTHOR]

Purpose: The STAT (Safety Threats and Adverse Events in Trauma) taxonomy was developed through expert consensus, and groups 65 identified trauma resuscitation adverse events (AEs) into nine distinct categories. It provides a framework for standardized analysis of trauma resuscitations and creates a foundation for targeted quality improvement and patient safety initiatives. This study aims to evaluate the reliability of the STAT taxonomy in identifying AEs during video-recorded trauma resuscitations. Methods: High-definition audiovisual data from 30 trauma resuscitations were reviewed. Videos were assessed and scored by four independent reviewers (two trainees and two staff). The STAT taxonomy was used to identify AEs based on binary responses: yes and no. Inter-rater reliability was calculated using Gwet's AC1. The frequencies of AEs were tallied and reported as counts and percentages. Results: The most common AEs identified in the videos were failure to measure temperature (86.7%) and inadequate personal protective equipment (86.7%), followed by inability to use closed-loop communication (76.7%). The agreement on all AEs between reviewers was 0.94 (95% CI: 0.93–0.95). The Gwet's AC1 agreement across the 9 AE categories was paramedic handover (0.82), airway and breathing (0.99), circulation (0.95), assessment of injuries (0.91), management of injuries (0.96), procedure-related (0.97), patient monitoring and IV access (0.99), disposition (0.98), team communication and dynamics (0.87). Conclusion: The STAT taxonomy demonstrated excellent inter-rater reliability between reviewers and can be used to identify AEs in video-recorded trauma resuscitations. These results provide a foundation for adapting video review to objectively quantify and assess AEs in the trauma bay. [ABSTRACT FROM AUTHOR]

With two-thirds of adults presenting for a videofluoroscopy swallow study (VFSS) with oesophageal abnormalities, it seems prudent to include visualisation of the oesophagus, in the context of the entire swallow process, to provide further information to the diagnostic team. This study aims to evaluate the ability of speech-language pathologists (SLPs) to interpret oesophageal sweep on VFSS and the relative improvement in that ability with additional training. One hundred SLPs attended training in oesophageal visualisation during VFSS, based on a previous study. Ten oesophageal sweep videos (five normal, five abnormal) with one 20 ml thin fluid barium bolus (19% w/v) were presented at baseline and following training. Raters were blinded to patient information other than age. Binary ratings were collected for oesophageal transit time (OTT), presence of stasis, redirection, and referral to other specialists. Inter-rater reliability as measured by Fleiss' kappa improved for all parameters, reaching statistical significance for OTT (pre-test kappa = 0.34, post-test kappa = 0.73; p < 0.01) and redirection (pre-test kappa = 0.38, post-test kappa = 0.49; p < 0.05). Overall agreement improved significantly (p < 0.001) for all parameters except stasis, where improvement was only slight. Interaction between pre-post and type of video (normal/abnormal) was statistically significant (p < 0.001) for redirection, with a large pre-post increase in positive accuracy compared with a slight pre-post decrease in negative accuracy. Findings indicate that SLPs require training to accurately interpret an oesophageal sweep on VFSS. This supports the inclusion of education and training on both normal and abnormal oesophageal sweep patterns, and the use of standardised protocols for clinicians using oesophageal visualisation as part of the VFSS protocol. [ABSTRACT FROM AUTHOR]

Background: The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses. Methods: This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters. Results: The Cohen's Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa. Conclusion: Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.'s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries. [ABSTRACT FROM AUTHOR]

The accurate comparison of prevalence rates of disease between different groups/populations is vital if we seek to contextualize our understanding of the impact of risk factors on health in the past. The majority of bioarchaeological studies of maxillary sinusitis employ the methods and "diagnostic criteria" outlined by Boocock and colleagues in 1995. However, until now, few attempts have been made to assess the inter‐rater reliability of these methods. This study presents the results of the analysis of interobserver variability in the recording of bone changes related to maxillary sinusitis among three observers within three human osteoarchaeological populations. The results of Cohen's kappa coefficient tests indicated variability in agreement between different observers. The agreement on the presence/absence of maxillary sinusitis in different osteoarchaeological populations ranged from κ = 0.433 ("moderate" agreement) to κ = 0.629 ("substantial" agreement). The agreement on the type of bone change present within affected sinuses was often poor, with almost no to negative agreement for pitting (Observers 1 and 2) and remodeled spicules (Observers 1 and 3). Methodological problems that can impact consistency of results between researchers include variability in sinus preservation, duration of observer experience, the use of different endoscopic equipment, lack of clarity in the original method descriptions, and a deficit in clinical corroborations for bone changes. The results of this study highlight the need to improve standards for the recording of bone changes related to sinusitis to allow for meaningful comparisons of past maxillary sinusitis frequency rates. Further investigations of interobserver variability, incorporating a greater range of variables, are also required. [ABSTRACT FROM AUTHOR]

The process of identification and annotation of evaluation has received a lot of attention in recent years. However, given the complexity of the topic, the discussion of some of the central issues is still ongoing. The present article contributes to this debate by presenting an annotation scheme that is designed for the identification and annotation of evaluative stance in a corpus of four English genres, namely, newspaper discourse, political discourse, newspaper scientific popularization and fora. A 4,862-word corpus was sampled from a larger 400,000-word corpus compiled within the research project STANCEDISC on the study of stance in discourse varieties. The scheme posits a series of ad hoc categories designed to optimise the transparency, reliability and replicability of the identification, annotation and analysis of evaluative stance. The categories are as follows: parts of speech (Noun Phrase, NP; Adjectival Phrase, AP; Adverbial Phrase, ADVP; Verbal Phrase, VP), function (classifying, predicational and attitude), metaphoricity (metaphoric and non-metaphoric), and value (positive and negative). The aim of this paper is to explain the scheme together with the theoretical justification of the categories and the methodological procedure adopted, and to illustrate the implementation of the scheme by discussing examples taken from different genres. [ABSTRACT FROM AUTHOR]

Purpose: Inter-rater reliability is a critical aspect of stroke image interpretation. This study aims to investigate inter-rater reliability between neurologists and neuro-radiologist when assessing Alberta Stroke Program Early CT Score (ASPECTS) and Intracerebral Haemorrhage (ICH) scores using a mobile application (SMART INDIA App – by neurologists) and the Picture Archiving and Communication System (PACS – by neuro-radiologist). Methods: Adult patients diagnosed with ischemic or haemorrhagic stroke were included in this study. Two Neurologists (R1 and R2) assessed the ASPECTS and ICH scores by viewing the SMART INDIA App. A neuroradiologist expert (R3) assessed the same using PACS. Kappa statistics are presented for agreement between the Raters. Results: 100 consecutive patients each of Acute Ischemic stroke (AIS) and ICH were included. A significant agreement in the total ASPECTS between the Raters 1 and 2 (0.85(95% CI—0.775,0.926)), Raters 1 and 3(0.76(95% CI—0.671,0.857)) and Raters 2 and 3(0.76(95% CI—0.673,0.857)) was noted. A good agreement was ascertained between Rater 1 and 3, even though the devices were different. A similar excellent agreement was also noted in assessing the ICH score. The study's findings indicate that neurologists using different devices and platforms demonstrated good to excellent agreement with the neuro-radiologist (considered the gold standard) when estimating ASPECTS and ICH scores. Conclusion: The interpretations of neurologists using the SMART INDIA App can be deemed valid and reliable from the neuroimaging viewpoint. These results contribute to our understanding of the feasibility and reliability of app-based image evaluation in the assessment of stroke. [ABSTRACT FROM AUTHOR]

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen's kappa and Gwet's AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments. [ABSTRACT FROM AUTHOR]

Abstract Background Dysphagia is commonly evaluated using videofluoroscopy (VFS). As its ratings are usually subjective normal‐abnormal ratings, objective measurements have been developed. We compared the inter‐rater reliability of the usual VFS ratings to the objective measurement VFS ratings and evaluated their clinical relevance. Methods Two blinded raters analyzed the subjective normal‐abnormal ratings of 77 patients' VFS. Two other blinded raters analyzed the objective measurements of pharyngeal aerated area with bolus held in the oral cavity (PAhold), the pharyngeal area of residual bolus during swallowing (PAmax), the pharyngeal constriction ratio (PCR), the maximum pharyngoesophageal segment opening (PESmax), pharyngoesophageal segment opening duration (POD), airway closure duration (ACD), and total pharyngeal transit time (TPT). We evaluated the inter‐rater agreement in the subjective ratings and the objective measurements. Clinical utility analysis compared the measurements with the VFS findings of pharyngeal phase abnormality, penetration/aspiration, and cricopharyngeal relaxation. Results In the pharyngeal findings, the subjective analysis inter‐rater agreement was mainly moderate to strong. The strongest agreements were on the pharyngeal residues and penetration/aspiration findings. The objective measurements had fair to good inter‐rater agreement. Clinical utility analysis found statistically significant connections between TPT and pharyngeal phase abnormality, normal PCR and lack of penetration/aspiration, and normal PESmax and normal cricopharyngeal relaxation. Conclusions The subjective analysis had moderate to strong inter‐rater agreement in the pharyngeal VFS findings, especially concerning pharyngeal residues and penetration/aspiration detection, reflecting the efficacy and safety of swallowing. The objective measurements had fair to good inter‐observer reproducibility and could thus improve the reliability of VFS diagnostics. Level of evidence 4.

Abstract Ensuring consistent and reliable scoring is paramount in education, especially in performance-based assessments. This study delves into the critical issue of marking consistency, focusing on speaking proficiency tests in English language learning, which often face greater reliability challenges. While existing literature has explored various methods for assessing marking reliability, this study is the first of its kind to introduce an alternative statistical tool, namely the gauge repeatability and reproducibility (GR&R) approach, to the educational context. The study encompasses both intra- and inter-rater reliabilities, with additional validation using the intraclass correlation coefficient (ICC). Using a case study approach involving three examiners evaluating 30 recordings of a speaking proficiency test, the GR&R method demonstrates its effectiveness in detecting reliability issues over the ICC approach. Furthermore, this research identifies key factors influencing scoring inconsistencies, including group performance estimation, work presentation order, rubric complexity and clarity, the student’s chosen topic, accent familiarity, and recording quality. Importantly, it not only pinpoints these root causes but also suggests practical solutions, thereby enhancing the precision of the measurement system. The GR&R method can offer significant contributions to stakeholders in language proficiency assessment, including educational institutions, test developers and policymakers. It is also applicable to other cases of performance-based assessments. By addressing reliability issues, this study provides insights to enhance the fairness and accuracy of subjective judgements, ultimately benefiting overall performance comparisons and decision making.

Abstract Background Worldwide, there is a large and growing group of older adults. Frailty is known as an important discriminatory factor for poor outcomes. The Clinical Frailty Scale (CFS) has become a frequently used frailty instrument in different clinical settings and health care sectors, and it has shown good predictive validity. The aims of this study were to describe and validate the translation and cultural adaptation of the CFS into Swedish (CFS-SWE), and to test the inter-rater reliability (IRR) for registered nurses using the CFS-SWE. Methods An observational study design was employed. The ISPOR principles were used for the translation, linguistic validation and cultural adaptation of the scale. To test the IRR, 12 participants were asked to rate 10 clinical case vignettes using the CFS-SWE. The IRR was assessed using intraclass correlation and Krippendorff’s alpha agreement coefficient test. Results The Clinical Frailty Scale was translated and culturally adapted into Swedish and is presented in its final form. The IRR for all raters, measured by an intraclass correlation test, resulted in an absolute agreement value among the raters of 0.969 (95% CI: 0.929–0.991) and a consistency value of 0.979 (95% CI: 0.953–0.994), which indicates excellent reliability. Krippendorff’s alpha agreement coefficient for all raters was 0.969 (95% CI: 0.917–0.988), indicating near-perfect agreement. The sensitivity of the reliability was examined by separately testing the IRR of the group of specialised registered nurses and non-specialised registered nurses respectively, with consistent and similar results. Conclusion The Clinical Frailty Scale was translated, linguistically validated and culturally adapted into Swedish following a well-established standard technique. The IRR was excellent, judged by two established, separately used, reliability tests. The reliability test results did not differ between non-specialised and specialised registered nurses. However, the use of case vignettes might reduce the generalisability of the reliability findings to real-life settings. The CFS has the potential to be a common reference tool, especially when older adults are treated and rehabilitated in different care sectors.

Introduction: Gastroesophageal reflux disease affects a significant portion of the Australian and world population. Minimally invasive laparoscopic fundoplication is a highly effective treatment in appropriately selected patients, with a 90% satisfaction rate. However, up to 5% will undergo revisional surgery. Endoscopy is an important investigation in the evaluation of persistent or new symptoms after fundoplication. Our study sought to evaluate the inter-rater reliability and variability in assessing fundoplication with endoscopy. Methods: Upper gastrointestinal (UGI) surgeons and gastroenterologists were invited to join the cohort study through their professional membership with two societies based in Australia. Participants completed a two part 25-item multiple choice questionnaire, involving the analysis of ten static endoscopic images post-fundoplication. Results: A total of 101 participants were included in the study (64 UGI surgeons and 37 gastroenterologists). Over 95% of participants were consultant level, working in non-rural tertiary hospitals. Total accuracy for all 10 cases combined was 76% for UGI surgeons and 69.9% for gastroenterologists. In three of the 10 cases, UGI surgeons performed significantly better than gastroenterologists (p < 0.05). When assessing performance across each of the 4 questions for each case, UGI surgeons were more accurate than gastroenterologists in describing the integrity of the wrap (p = 0.014). Inter-rater reliability was low across both groups for most domains (kappa < 1). Conclusion: Our study confirms low inter-rater reliability between endoscopists and large variations in reporting. UGI surgeons performed better than gastroenterologists in certain cases, usually when describing the integrity of the fundoplication. Our study provides further support for the use of a standardized reporting system in post-fundoplication patients. [ABSTRACT FROM AUTHOR]

Aim: The aim of this Swedish study was to evaluate the assessment of clinical signs of perceptual disorder in children with cerebral palsy (CP). Methods: Three experienced raters assessed 56 videos of 19 children from 1 to 18 years of age with bilateral spastic CP, which were recorded by colleagues at an Italian hospital. Six signs were evaluated for inter‐rater reliability and criterion validity. Clinical applicability was evaluated by assessing inter‐rater reliability between 47 Swedish clinicians, who examined 15 of the videos during face‐to‐face and online education seminars. There were 41 physiotherapists, two occupational therapists and four doctors, with 1–37 years of clinical experience and a median of 10 years. Results: The experienced raters demonstrated moderate to almost perfect inter‐rater reliability (kappa 0.54–0.81) and criterion validity (0.54–0.87) for startle reaction, upper limbs in startle position, averted eye gaze and eye blinking. The clinicians recognised these signs with at least moderate reliability (0.56–0.88). Grimacing and posture freezing were less reliable (0.22–0.35) and valid (0.09–0.50). Conclusion: Four of the six signs of perceptual disorder were reliably recognised by experienced raters and by clinicians after education seminars. Extended education and larger study samples are needed to recognise all the signs. [ABSTRACT FROM AUTHOR]

Ensuring consistent and reliable scoring is paramount in education, especially in performance-based assessments. This study delves into the critical issue of marking consistency, focusing on speaking proficiency tests in English language learning, which often face greater reliability challenges. While existing literature has explored various methods for assessing marking reliability, this study is the first of its kind to introduce an alternative statistical tool, namely the gauge repeatability and reproducibility (GR&R) approach, to the educational context. The study encompasses both intra- and inter-rater reliabilities, with additional validation using the intraclass correlation coefficient (ICC). Using a case study approach involving three examiners evaluating 30 recordings of a speaking proficiency test, the GR&R method demonstrates its effectiveness in detecting reliability issues over the ICC approach. Furthermore, this research identifies key factors influencing scoring inconsistencies, including group performance estimation, work presentation order, rubric complexity and clarity, the student's chosen topic, accent familiarity, and recording quality. Importantly, it not only pinpoints these root causes but also suggests practical solutions, thereby enhancing the precision of the measurement system. The GR&R method can offer significant contributions to stakeholders in language proficiency assessment, including educational institutions, test developers and policymakers. It is also applicable to other cases of performance-based assessments. By addressing reliability issues, this study provides insights to enhance the fairness and accuracy of subjective judgements, ultimately benefiting overall performance comparisons and decision making. [ABSTRACT FROM AUTHOR]

The rapid development of artificial intelligence and large language models (LLMs) has led to significant advancements in applying machine learning techniques across diverse disciplines, including educational science research. This study investigates the potential of LLMs like ChatGPT for qualitative data analysis, focusing on open, axial, selective coding, theme or pattern identification, and inter-rater reliability. Our findings indicate promising capabilities of ChatGPT in open coding, demonstrating accurate categorization of qualitative data. However, axial coding posed challenges due to the model's limited understanding, which was partially addressed by refining prompts based on ChatGPT's interpretation. ChatGPT also showed competence in selective coding and theme or pattern identification, providing additional insights. For inter-rater reliability, ChatGPT's performance varied across datasets, with improvements observed when providing contextual information. It is important to note the limitations and variability of LLMs such as ChatGPT, which is in public beta and subject to potential limitations in usage and reliability. Our study demonstrates ChatGPT's potential for coding and inter-rater reliability. Improved results are achieved with refined prompts and utilising ChatGPT's own definitions. The adoption of LLMs for qualitative analysis requires further exploration, including addressing algorithmic bias and the potential for inaccurate responses. Validation techniques are crucial in mitigating these risks. [ABSTRACT FROM AUTHOR]

Copyright of Rigakuryoho Kagaku is the property of International Press Editing Center Incorporation and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Introduction: The Wessex Head Injury Matrix (WHIM) was developed to assess patients with disorders of consciousness (DOC) and was tested in terms of inter-rater reliability (IRR) and test–retest reliability (TRR) in the year 2000. The American Congress of Rehabilitation and Medicine reported that IRR and TRR were unproven. We aim to assess the reliability of the WHIM in prolonged DOC patients (PDOC). Methods: A total of 51 PDOC patients (32 unresponsive wakefulness syndrome (UWS/VS) and 19 minimally conscious state (MCS)) who were hosted in a dedicated unit for long-term brain injury care were enrolled. The time from injury ranged from 182 to 3325 days. Two raters administered the Coma Recovery Scale-Revised (CRS-R) and the WHIM to test the IRR and TRR. The TRR was administered two weeks after the first assessment. Results: For the CRS-R, the agreement in IRR and TRR was perfect between the two raters. The agreement for the WHIM ranged from substantial to almost perfect for IRR and from fair to substantial for the TRR. Conclusions: The WHIM showed a strong IRR when administered by expert raters and strongly correlated with the CRS-R. This study provides further evidence of the psychometric qualities of the WHIM and the importance of its use in PDOC patients. [ABSTRACT FROM AUTHOR]

Background: Worldwide, there is a large and growing group of older adults. Frailty is known as an important discriminatory factor for poor outcomes. The Clinical Frailty Scale (CFS) has become a frequently used frailty instrument in different clinical settings and health care sectors, and it has shown good predictive validity. The aims of this study were to describe and validate the translation and cultural adaptation of the CFS into Swedish (CFS-SWE), and to test the inter-rater reliability (IRR) for registered nurses using the CFS-SWE. Methods: An observational study design was employed. The ISPOR principles were used for the translation, linguistic validation and cultural adaptation of the scale. To test the IRR, 12 participants were asked to rate 10 clinical case vignettes using the CFS-SWE. The IRR was assessed using intraclass correlation and Krippendorff's alpha agreement coefficient test. Results: The Clinical Frailty Scale was translated and culturally adapted into Swedish and is presented in its final form. The IRR for all raters, measured by an intraclass correlation test, resulted in an absolute agreement value among the raters of 0.969 (95% CI: 0.929–0.991) and a consistency value of 0.979 (95% CI: 0.953–0.994), which indicates excellent reliability. Krippendorff's alpha agreement coefficient for all raters was 0.969 (95% CI: 0.917–0.988), indicating near-perfect agreement. The sensitivity of the reliability was examined by separately testing the IRR of the group of specialised registered nurses and non-specialised registered nurses respectively, with consistent and similar results. Conclusion: The Clinical Frailty Scale was translated, linguistically validated and culturally adapted into Swedish following a well-established standard technique. The IRR was excellent, judged by two established, separately used, reliability tests. The reliability test results did not differ between non-specialised and specialised registered nurses. However, the use of case vignettes might reduce the generalisability of the reliability findings to real-life settings. The CFS has the potential to be a common reference tool, especially when older adults are treated and rehabilitated in different care sectors. [ABSTRACT FROM AUTHOR]