Search

Background: Numerous cardiovascular risk prediction models (RPM) have been developed, however, agreement studies between these models are scarce. We aimed to assess the inter-model agreement between eight RPMs: assessing cardiovascular risk using SIGN, the Australian CVD risk score (AusCVDRisk), the Framingham Risk Score for Hard Coronary Heart Disease, the Multi-Ethnic Study of Atherosclerosis risk score, the Pooled Cohort Equation (PCE), the QRISK3 cardiovascular risk calculator, the Reynolds Risk Score, and Systematic Coronary Risk Evaluation-2 (SCORE2). Methods: A cross-sectional study was conducted on 11,174 40–65-year-old individuals with diagnosed metabolic syndrome from a single tertiary university hospital in Lithuania. Cardiovascular risk was calculated using the eight RPMs, and the results were categorized into high, intermediate, and low-risk groups. Inter-model agreement was quantified using Cohen's Kappa coefficients. Results: The study revealed significant heterogeneity in risk categorizations with only 1.49% of cases where all models agree on the risk category. SCORE2 predominantly categorized participants as high-risk (67.39%), while the PCE identified the majority as low-risk (62.03%). Cohen's Kappa coefficients ranged from −0.09 to 0.64, indicating varying degrees of inter-model agreement. Conclusions: The choice of RPM can substantially influence clinical decision-making and patient management. The PCE and AusCVDRisk models exhibited the highest degree of agreement while the SCORE2 model consistently exhibited low agreement with other models. [ABSTRACT FROM AUTHOR]

Background: In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing. Methods: This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved. Results: The Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure. Conclusion: Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research. [ABSTRACT FROM AUTHOR]

Scientific studies of the elements that influence the box office performance of Indian films have generally concentrated on post-production elements, such as those discovered after a film has been completed or released, and notably for Bollywood films. Only fewer studies have looked at regional film industries and pre-production factors, which are elements that are known before a decision to greenlight a film is made. This study looked at Indian films using natural language processing and machine learning approaches to see if they would be profitable in the pre-production stage. We extract movie data and English subtitles (as an approximation to the screenplay) for the top five Indian regional film industries: Bollywood, Kollywood, Tollywood, Mollywood, and Sandalwood, as they make up a major portion of the Indian film industry's revenue. Subtitle Vector (Sub2Vec), a Paragraph Vector model trained on English subtitles, was used to embed subtitle text into 50 and 100 dimensions. The proposed approach followed a two-stage pipeline. In the first stage, Return on Investment (ROI) was calculated using aggregated subtitle embeddings and associated movie data. Classification models used the ROI calculated in the first step to predicting a film's verdict in the second step. The optimal regressor–classifier pair was determined by evaluating classification models using F 1 -score and Cohen's Kappa scores on various hyperparameters. When compared to benchmark methods, our proposed methodology forecasts box office success more accurately. [ABSTRACT FROM AUTHOR]

Cohen's and Fleiss's kappa are popular estimators for assessing agreement among two and multiple raters, respectively, for a binary response. While additional methods have been developed to account for multiple raters and covariates, they are not always applicable, rarely used, and none simplify to Cohen's kappa. Furthermore, there are no methods to simulate Bernoulli observations under the kappa agreement structure such that the developed methods could be adequately assessed. This manuscript overcomes these shortfalls. First, we developed a model‐based estimator for kappa that accommodates multiple raters and covariates through a generalized linear mixed model and encompasses Cohen's kappa as a special case. Second, we created a framework to simulate dependent Bernoulli observations that upholds all 2‐tuple pair of rater's kappa agreement structure and includes covariates. We used this framework to assess our method when kappa was nonzero. Simulations showed that Cohen's and Fleiss's kappa estimates were inflated unlike our model‐based kappa. We analyzed an Alzheimer's disease neuroimaging study and the classic cervical cancer pathology study. The proposed model‐based kappa and advancement in simulation methodology demonstrates that the popular approaches of Cohen's and Fleiss's kappa are poised to yield invalid conclusions while our work overcomes shortfalls, leading to improved inferences. [ABSTRACT FROM AUTHOR]

INTRODUCTION: The Workplace Safety Culture Questionnaire (WSCQ) was developed to assess the level of knowledge, attitudes and practices on Workplace Safety Culture (WSC) among public sector office workers in Nigeria. The main objective of this study was to determine the validity and reliability of the new Workplace Safety Culture Questionnaire (WSCQ) using the Information Motivation and Behavior (IMB) model. MATERIALS AND METHODS: The WSCQ questionnaire was completed twice by the respondents themselves, with an interval of ten days between sessions, to assess the accuracy of the initial results with the retest. This study involved 44 participants. RESULTS: Cronbach's alpha showed significant item consistency for each construct in the second pilot study. The information construct (Cronbach's a=0.929 and minimum corrected item-total correlation (CITC)=0.399). The motivation construct (Cronbach's a=0.932 and minimum corrected item-total correlation (CITC)=0.450). The behavior construct (Cronbach's a=0.812 and minimum corrected item-total correlation (CITC)= 0.401). In test-retest reliability, Cohen's kappa coefficient for construct of information for all items was almost 70% between (kappa k=0.689-1000, p<0.001). The intraclass correlation coefficient (ICC) for the construct of motivation for all items is 90% between (ICC=0.810-1000, p<0.001) and (kappa k=0.944-1000, p<0.001) for the construct of behavior was 71% for all items. CONCLUSION: This study introduces the WSCQ using the IMB model among public sector office workers as the first official validation of the WSCQ. [ABSTRACT FROM AUTHOR]

Background: In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table. Methods: In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone. Results: The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for 3 × 3 and 4 × 4 tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies. Conclusions: The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables. [ABSTRACT FROM AUTHOR]

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters. [ABSTRACT FROM AUTHOR]

To evaluate inter observer agreement between Radiologist and Otorhinolaryngologists in identifying important structures and pathologies on pre operative computed tomography (CT) scans of paranasal sinus (PNS) in chronic rhinosinusitis (CRS). A retrospective review of CT scans PNS of CRS patients. Archived CT scans were evaluated by 3 observers, similarly experienced radiologist and otorhinolaryngologist and a less experienced otorhinolarygologist. The degree of intra- as well as inter observer agreement was assessed by Cohen's kappa statistics. A moderate and a fair inter observer agreement was noted among experienced radiologist and otorhinolaryngologist; & experienced experts and less experienced otorhinolaryngologist respectively. The greatest disagreements among all observers were found in superior attachment of uncinate process, optic nerve's relationship with sphenoid sinus and dehiscence of anterior ethmoid canal. The present study demonstrated a considerable inter observer variability among radiologists and otorhinolaryngologists, irrespective of their experience in the field of speciality. A few critically important structures which showed an unsatisfactory level of agreement and thus need to be sought after with more training and practice, were also identified. These included Keros type, optic nerve's location, dehiscence of anterior ethmoid artery, vidian and maxillary nerves, lamina papyracia defect, sphenoid sinus pneumatization and attachment of uncinate process. The study also emphasized the importance of inter disciplinary conferences for pre operative evaluation of sinus CT scans of CRS patients. [ABSTRACT FROM AUTHOR]

The paper investigates changing levels of online concern about the Kashmiri Pandit migration of the 1990s on Twitter. Although decades old, this movement of people is an ongoing issue in India, with no current resolution. Analysing changing reactions to it on social media may shed light on trends in public attitudes to the event. Tweets were downloaded from Twitter using the academic version of its application programming interface (API) with the aid of the free social media analytics software Mozdeh. A set of 1000 tweets was selected for content analysis with a random number generator in Mozdeh. The results show that the number of tweets about the issue has increased over time, mainly from India, and predominantly driven by the release of films like Shikara and The Kashmir Files. The tweets show apparent universal support for the Pandits but often express strong emotions or criticize the actions of politicians, showing that the migration is an ongoing source of anguish and frustration that needs resolution. The results also show that social media analysis can give insights even into primarily offline political issues that predate the popularity of the web, and can easily incorporate international perspectives necessary to understand complex migration issues. [ABSTRACT FROM AUTHOR]

Background: Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.Methods: We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.Results: The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.Conclusion: The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement. [ABSTRACT FROM AUTHOR]

The need to measure the degree of agreement among R raters who independently classify n subjects within K nominal categories is frequent in many scientific areas. The most popular measures are Cohen's kappa (R = 2), Fleiss' kappa, Conger's kappa and Hubert's kappa (R ≥ 2) coefficients, which have several limitations. In 2004, the delta coefficient was defined for the case of R = 2, which did not have the limitations of Cohen's kappa coefficient. This article extends the coefficient delta from R = 2 raters to R ≥ 2 (coefficient multi-rater delta), demonstrating that it can be expressed in the kappa format and has the same advantages as the coefficient delta with regard to the type kappa classic coefficients: (i) it refers to the proportion of replies that are concordant not by chance; (ii) allow to obtain a parameter that faithfully measures the degree of agreement in each category; and (iii) it is not affected by the marginal imbalance. [ABSTRACT FROM AUTHOR]

Background: The reliability of Criteria for Assessing Prescription Quality in Chinese Hospitals (CAPQCH) has never been rigorously verified. This study was designed to verify the reliability of the CAPQCH among pharmacists in China.Methods: Fourteen pharmacists, 5 from hospitals and 9 from the communities were recruited. We randomly selected 200 prescriptions, and made the testing prescriptions including appropriate and inappropriate testing prescriptions. Pharmacists assessed these testing prescriptions according to criteria in CAPQCH. Three test sets (Set 1, Set 2, and Set 3) were evaluated at 6-month intervals. Before administration of Set 3, pharmacists were informed that achievement on Set 3 would be reflected in their performance appraisal. We also evaluated the performance based on prescription comments before and after combining several confusing criteria. Cohen's Kappa statistic, Fleiss' Kappa statistic, and accuracy were employed to evaluate reliability among pharmacists.Results: Median values of Cohen's Kappa were 0.61 in Set 1, 0.66 in Set 2, and 0.80 in Set 3; reliability is thus substantial. Our data indicate no significant differences between Set 1 and Set 2, whereas Set 3 indicates significantly improved performance. Moreover, combinations of confusing criteria contributed little to improvement of performance in prescription comments.Conclusion: Our results verified the reliability of CAPQCH application by working pharmacists. Adding performance based on prescription comments to personal appraisals was effective in improving the quality of prescription comments. These findings may be useful when future modification of the CAPQCH is considered. Moreover, this study contributes to improving the understanding of the prescription assessment situation in China. [ABSTRACT FROM AUTHOR]

Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen's kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson's correlation, Spearman's rho, and Kendall's tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study. [ABSTRACT FROM AUTHOR]

Background: Vitamin A deficiency (VAD) and anemia are the most prevalent nutritional deficiency in children globally. The dried blood spot (DBS) method has been used in prevalence studies of VAD and anemia in different age groups. However, it has not yet been validated for children.Objectives: This study aimed to assess the reproducibility and validity of DBS in the diagnosis of VAD and anemia in preschoolers.Methods: Venous and capillary blood samples were collected from a representative sample of children <5 y old who attended the public health system in Rio de Janeiro. Serum retinol and hemoglobin were measured in 235 and 182 children, respectively. Serum retinol was measured with HPLC and hemoglobin was measured with spectrophotometry in samples of venous (gold standard) and capillary blood (test method, DBS). DBS reproducibility was assessed with the intraclass correlation coefficient (ICC), κ, and prevalence-adjusted and bias-adjusted κ (PABAK). DBS validity was assessed with sensitivity, specificity, accuracy index (AI), positive predictive value (PPV), and negative predictive value (NPV).Results: DBS showed very good reproducibility for serum retinol (ICC = 0.94, κ = 0.83, PABAK = 0.76) and very good/good reproducibility for hemoglobin (ICC = 0.86, κ = 0.69, PABAK = 0.69). Prevalence rates for VAD by the reference and test methods were 11.5% and 11.9%, respectively, whereas the anemia rates were 19.2% and 46.2%. The test method showed low sensitivity (33%) and PPV (32%) and high specificity (91%) and NPV (92%) for serum retinol. For hemoglobin, the test method showed fair sensitivity (71%), low PPV (30%), fair specificity (60%), and high NPV (90%). AI was 83% for VAD and 62% for anemia.Conclusions: The results suggest that DBS is adequate for the diagnosis of VAD in preschool children, but not for anemia. [ABSTRACT FROM AUTHOR]

We read with great interest the article by Hendershot et al. (Parasit Vectors 14:473, 2021). The authors compared a PCR method for detecting Plasmodium vivax's mitochondrial (mt) cytochrome oxidase I (COX-I) gene with the current "gold standard" circumsporozoite (CSP) ELISA for detecting circumsporozoite protein for identification of different life stages of Plasmodium vivax during development within Anopheles arabiensis. We found that Cohen's kappa value for measuring the agreement between mt COX-I PCR and CSP ELISA was questionable. In addition, we recommend a more appropriate statistical method in this article. In short, any scientific conclusion requires support by the reasonable application of methodological and statistical methods. [ABSTRACT FROM AUTHOR]

A unified treatment of agreement coefficients for multiple raters is shown, where the chance-expected proportions of the Bennett et al.-type, Scott-type, its new variation, and Cohen-type are dealt with using full or lower-order agreement among raters. When only pairwise agreement is used for multiple raters, chance corrections of the Gwet-type and its new variation are also considered. For the unified treatment, Conger's two formulas of the ratio of means and the mean of ratios are combined into a parent formula of the weighted mean of weighted ratios. The corresponding unified expressions of their new asymptotic results are presented. [ABSTRACT FROM AUTHOR]

Machine learning (ML) has been increasingly employed in science assessment to facilitate automatic scoring efforts, although with varying degrees of success (i.e., magnitudes of machine-human score agreements [MHAs]). Little work has empirically examined the factors that impact MHA disparities in this growing field, thus constraining the improvement of machine scoring capacity and its wide applications in science education. We performed a meta-analysis of 110 studies of MHAs in order to identify the factors most strongly contributing to scoring success (i.e., high Cohen's kappa [κ]). We empirically examined six factors proposed as contributors to MHA magnitudes: algorithm, subject domain, assessment format, construct, school level, and machine supervision type. Our analyses of 110 MHAs revealed substantial heterogeneity in κ (m e a n =.64; range =.09-.97 , taking weights into consideration). Using three-level random-effects modeling, MHA score heterogeneity was explained by the variability both within publications (i.e., the assessment task level: 82.6%) and between publications (i.e., the individual study level: 16.7%). Our results also suggest that all six factors have significant moderator effects on scoring success magnitudes. Among these, algorithm and subject domain had significantly larger effects than the other factors, suggesting that technical features and assessment external features might be primary targets for improving MHAs and ML-based science assessments. [ABSTRACT FROM AUTHOR]

In this study, Standardized Precipitation Index (SPI) derived from parametric and nonparametric methods using 0.25∘ gridded rainfall data from 1901 to 2013 (113 years) generated by India Meteorological Department (IMD) was compared for understanding drought conditions over the Indian region. The parametric SPI was computed using a three-parameter Gamma distribution function, whereas nonparametric SPI was computed using Gringorten, Weibull, and Hazen plotting positions, on a 4-month cumulative rainfall data of June–September (SPI-4) representing the southwest monsoon season. Nonnormality is a major concern if equal-sized intervals are drawn for interpretation, and SPI being a normalized index wherein classes are standard deviations from normal, its impact on drought assessment needs to be understood. Accordingly, in our study, normality tests were performed using the Shapiro-Wilk method on SPI derived from both parametric and nonparametric methods. The SPI showed 100% of grid cells conforming to normality in the case of nonparametric methods, whereas in the case of parametric approach it was only 80%. The remaining 20% of nonnormality in parametric SPI is spread over montane, tropical wet, and semi-arid regions of India. Furthermore, differences in the estimation of dryness are observed in the range of 1.0 to 2.5% between nonparametric and parametric SPI for the drought years considered this study. The quantile analysis on all grid cells for the drought year 2002 showed an important fact that at 0.025 quantile only 2.6% of grid cells are in the extremely dry condition as per parametric SPI, whereas in the case of nonparametric SPI it is 6.9%. For the drought year 1939 in grid cells where normality is not followed in parametric SPI, Cohen's kappa (κ = 0.15) under extreme dryness category indicates large disagreements between parametric and nonparametric SPI. The temporal analysis of Cohen's kappa computed for each grid cell over drought years shows that in 22.5% of cases the drought category between nonparametric and parametric SPI is not in perfect agreement. Hence, the nonparametric SPI can better categorize the drought classes, representing well the extent of dryness and normality conditions, it is highly recommended for drought assessment over India. [ABSTRACT FROM AUTHOR]

We present a methodology motivated by a controlled trial designed to validate SPOT GRADE, a novel surgical bleeding severity scale. Briefly, the study was designed to quantify inter- and intra-surgeon agreement for characterizing the severity of surgical bleeds via a Kappa statistic. Multiple surgeons were presented with a randomized sequence of controlled bleeding videos and asked to apply the rating system to characterize each wound. Each video was shown multiple times to quantify intra-surgeon reliability, creating clustered data. In addition, videos within the same category may have had different classification probabilities due to changes in blood flow rates and wound sizes. In this work, we propose a new variance estimator for the Kappa statistic, for use in clustered data as well as heterogeneity among items within the same classification category. We then apply this methodology to data from the SPOT GRADE trial. [ABSTRACT FROM AUTHOR]

There is a frequent need to measure the degree of agreement among R observers who independently classify n subjects within K nominal or ordinal categories. The most popular methods are usually kappa‐type measurements. When R = 2, Cohen's kappa coefficient (weighted or not) is well known. When defined in the ordinal case while assuming quadratic weights, Cohen's kappa has the advantage of coinciding with the intraclass and concordance correlation coefficients. When R > 2, there are more discrepancies because the definition of the kappa coefficient depends on how the phrase 'an agreement has occurred' is interpreted. In this paper, Hubert's interpretation, that 'an agreement occurs if and only if all raters agree on the categorization of an object', is used, which leads to Hubert's (nominal) and Schuster and Smith's (ordinal) kappa coefficients. Formulae for the large‐sample variances for the estimators of all these coefficients are given, allowing the latter to illustrate the different ways of carrying out inference and, with the use of simulation, to select the optimal procedure. In addition, it is shown that Schuster and Smith's kappa coefficient coincides with the intraclass and concordance correlation coefficients if the first coefficient is also defined assuming quadratic weights. [ABSTRACT FROM AUTHOR]

In this article, some misuses of Kappa statistic in the original paper [Chemosphere, 307, 135831] are discussed. By using DRASTIC and Analytic Hierarchy Process (AHP) models, the authors have assessed the groundwater vulnerability of Totko, India. High nitrate concentrations in groundwater have been found in highly vulnerable areas, and the accuracy of the models has been assessed through Pearson's correlation coefficient and Kappa coefficient. However, using Cohen's Kappa to estimate the intra-rater reliabilities (IRRs) of the two models is not appropriate on the condition of ordinal categorical variables in five categories in the original paper. We briefly introduce the Kappa statistic and propose to use weighted Kappa to compute IRRs under such conditions. To conclude, we recognize that this does not significantly alter the conclusions of the original paper, but it is necessary to ensure that the appropriate statistical tools are used. [Display omitted] • Cohen's Kappa assesses IRR for 2 or ≥3 unordered categorical variables categories. • Weighted Kappa measures IRR for ordered categorical variables with ≥3 categories. • Fleiss' Kappa evaluates IRR for three or more raters. [ABSTRACT FROM AUTHOR]

Cohen's kappa coefficient is commonly used for assessing agreement between classifications of two raters on a nominal scale. Three variants of Cohen's kappa that can handle missing data are presented. Data are considered missing if one or both ratings of a unit are missing. We study how well the variants estimate the kappa value for complete data under two missing data mechanisms--namely, missingness completely at random and a form of missingness not at random. The kappa coefficient considered in Gwet (Handbook of Inter-rater Reliability, 4th ed.) and the kappa coefficient based on listwise deletion of units with missing ratings were found to have virtually no bias and mean squared error if missingness is completely at random, and small bias and mean squared error if missingness is not at random. Furthermore, the kappa coefficient that treats missing ratings as a regular category appears to be rather heavily biased and has a substantial mean squared error in many of the simulations. Because it performs well and is easy to compute, we recommend to use the kappa coefficient that is based on listwise deletion of missing ratings if it can be assumed that missingness is completely at random or not at random. [ABSTRACT FROM AUTHOR]

Reliability, or repeatability, of permanent tooth staging techniques is usually expressed as Cohen's Kappa. This single value obscures information about the quantity and allocation of disagreements. In this study we assess and compare intra-observer reliability of permanent tooth staging techniques described by Nolla, Moorrees et al. and Demirjian et al. The sample was panoramic radiographs of healthy dental patients made up of 100 males and 100 females aged 6–15 years. All permanent teeth on the left side (excluding third molars) were scored twice. Weighted Kappa and percentage agreement were calculated. Results show Kappa values for all teeth combined as 0.918, 0.922 and 0.938 for Demirjian (number of teeth N = 2682), Nolla (N = 2698) and Moorrees (N = 2674) respectively. A comparison of Kappa values between upper and lower teeth showed marginally higher values for upper incisors and lower molar for all three scoring methods. Small differences in Kappa values were noted between tooth types with the upper first molar having smaller values than other teeth. Percentage agreement ranged from 81 % (Moorrees), 86 % (Nolla) to 87 % (Demirjian). Tooth stage differences between first and second assessments were not more than one stage. Our findings show that Demirjian scoring is marginally more reliable than Nolla or Moorrees. We suggest that (1) data for reliability are tabulated in full to show the quantity and allocation of disagreement between first and second readings, and (2) that the reliability sample is sufficiently large with a wide age range to include multiple different tooth stages. • Forensic age estimation using developing teeth relies on reliability of tooth stage scoring, usually expressed as a single weighted Kappa value based on a ratio. • A positive finding of this study was excellent reliability using tooth stages of Nolla, Moorrees and Demirjian. • A negative finding of this study is that and quantity and allocation of disagreement between readings are hidden when a single value expresses reliability. More information of the location and quantity of disagreement are provided in pivot tables of 1st and 2nd readings. • We suggest that results for reliability are reported more fully than a single Kappa value. [ABSTRACT FROM AUTHOR]

dos Santos Teixeira et al. evaluated the effectiveness of three different sensors, portable X-ray fluorescence (pXRF) spectrometry, visible near-infrared (Vis-NIR), and NixPro TM color sensor, in predicting six soil properties including sum of bases (SB), effective cation exchange capacity (eCEC), potential cation exchange capacity (CEC), base saturation (BS), soil organic matter (SOM) content, and soil pH, and compared their performance when used alone or in combination. Preprocessing methods and other variables were also analyzed. In this comment paper, some misuses of Kappa statistic in the original paper by dos Santos Teixeira et al. are discussed. The authors calculated Cohen's kappa coefficient to make categorical predictions for soil fertility classes. However, using Cohen's Kappa to estimate the intra-rater reliabilities (IRRs) is not appropriate on the condition of ordinal categorical variables in four or five categories in the original paper. We briefly introduce the Kappa statistic for evaluation of IRR between two raters and propose to use weighted Kappa to compute IRRs under such conditions. To conclude, we recognize that this does not significantly alter the conclusions of this paper, but it is necessary to ensure that the appropriate statistical tools are used. • Cohen's Kappa assesses IRR for 2 or ≥3 unordered categorical variables categories. • Weighted Kappa measures IRR for ordered categorical variables with ≥3 categories. • QWK reveals more inconsistencies than LWK. [ABSTRACT FROM AUTHOR]

Psychiatric and psychological diagnoses are imperfect. Unlike somatic medicine, most psychological and psychiatric phenomena have no gold standard to establish their presence beyond reasonable doubt. Consequently, prevalence estimates are based on the average agreement of imperfect evaluators. Küchenhoff, Augustin, and Kunz (2012) provided a statistical method for estimating confidence intervals of the prevalence based on the well-known kappa coefficient of interrater agreement. We expand this method and derive confidence intervals for the probability of a diagnosis being true (i.e., the positive predictive value). We illustrate the method and its results with empirical data for a particular type of paraphilia (pedophilia) in sexual offenders. The findings indicate that up to 1 in 3 diagnoses of pedophilia may be wrong. Given the similar rates of prevalence and interrater agreement reported for diagnoses in general psychiatry (such as schizophrenia or affective disorders), the results likely apply to other diagnostic domains as well. (PsycINFO Database Record [ABSTRACT FROM AUTHOR]

A variety of metrics are commonly employed by map producers and users to assess and compare thematic maps'; quality, but their use and interpretation is inconsistent. This problem is exacerbated by a shortage of tools to allow easy calculation and comparison of metrics from different maps or as a map';s legend is changed. In this paper, we introduce a new website and a collection of R functions to facilitate map assessment. We apply these tools to illustrate some pitfalls of error metrics and point out existing and newly developed solutions to them. Some of these problems have been previously noted, but all of them are under-appreciated and persist in published literature. We show that binary and categorical metrics, including information about true-negative classifications, are inflated for rare categories, and more robust alternatives should be chosen. Most metrics are useful to compare maps only if their legends are identical. We also demonstrate that combining land-cover classes has the often-neglected consequence of apparent improvement, particularly if the combined classes are easily confused (e.g., different forest types). However, we show that the average mutual information (AMI) of a map is relatively robust to combining classes, and reflects the information that is lost in this process; we also introduce a modified AMI metric that credits only correct classifications. Finally, we introduce a method of evaluating statistical differences in the information content of competing maps, and show that this method is an improvement over other methods in more common use. We end with a series of recommendations for the meaningful use of accuracy metrics by map users and producers. [ABSTRACT FROM AUTHOR]

Cohen's kappa coefficient is a commonly used method for estimating interrater agreement for nominal and/or ordinal data; thus agreement is adjusted for that expected by chance. The weighted kappa statistic is used as an agreement index for ordinal data. The weights quantify the degree of discrepancy between the two categories. The choice of this particular set of weights affects the value of kappa. The common scores are Cicchetti-Allison and Fleiss-Cohen weights. In this article, we discuss the use of ridit type and exponential scores to compute kappa statistics in general. [ABSTRACT FROM AUTHOR]

Cohen's κ, a similarity measure for categorical data, has since been applied to problems in the data mining field such as cluster analysis and network link prediction. In this paper, a new application is examined: community detection in networks. A new algorithm is proposed that uses Cohen's κ as a similarity measure for each pair of nodes; subsequently, the κ values are then clustered to detect the communities. This paper defines and tests this method on a variety of simulated and real networks. The results are compared with those from eight other community detection algorithms. Results show this new algorithm is consistently among the top performers in classifying data points both on simulated and real networks. Additionally, this is one of the broadest comparative simulations for comparing community detection algorithms to date. [ABSTRACT FROM PUBLISHER]

This letter documents some problems in Ancaiani et al. (2015). Namely the evaluation of concordance, based on Cohen's kappa, reported by Ancaiani et al. was not computed on the whole random sample of 9,199 articles, but on a subset of 7,597 articles. The kappas relative to the whole random sample were in the range 0.07-0.15, indicating an unacceptable agreement between peer review and bibliometrics. The subset was obtained by non-random exclusion of all articles for which bibliometrics produced an uncertain classification; these raw data were not disclosed, so that concordance analysis is not reproducible. The VQR-weighted kappa for Area 13 reported by Ancaiani et al. is higher than that reported by Area 13 panel and confirmed by Bertocchi et al. (2015), a difference explained by the use, under the same name, of two different set of weights. Two values of kappa reported by Ancaiani et al. differ from the corresponding ones published in the official report. Results reported by Ancaiani et al. do not support a good concordance between peer review and bibliometrics. As a consequence, the use of both techniques introduced systematic distortions in the final results of the Italian research assessment exercise. The conclusion that it is possible to use both technique as interchangeable in a research assessment exercise appears to be unsound, by being based on a misinterpretation of the statistical significance of kappa values. [ABSTRACT FROM AUTHOR]

The study was conducted to evaluate influence of the true within-herd prevalence of small ruminant lentivirus (SRLV) infection on agreement beyond chance between three different types of commercial serological ELISAs. Blood samples were collected from 865 goats from 12 dairy goat herds. Serum samples were tested using three commercial ELISA kits: whole-virus indirect ELISA (wELISA), indirect ELISA based on recombined TM and CA antigens (TM/CA-ELISA), and competitive-inhibition ELISA based on SU antigen (SU-ELISA). Herds were classed into three prevalence strata of high (>50%), moderate (10–50%) and low (<10%) true within-herd prevalence of SRLV infection. The latter was estimated on the basis of results of wELISA adjusted by its sensitivity and specificity. Agreement beyond chance between the three ELISAs was assessed at two levels. First, the general agreement was determined using two coefficients corrected for chance-agreement: Cohen’s kappa and Gwet’s AC 1 . Then, agreement between tests was evaluated using Gwet’s AC 1 separately in the three prevalence strata and compared between them by computing 95% confidence intervals for differences with a Bonferroni correction for multiple comparisons. The general agreement between the three tests was very good: wELISA and TM/CA-ELISA – Cohen’s kappa of 81.8% (CI 95%: 77.9% to 85.7%), Gwet’s AC 1 of 82.7% (CI 95%: 79.0% to 86.4%); wELISA and SU-ELISA – Cohen’s kappa of 83.2% (CI 95%: 79.4% to 86.9%), Gwet’s AC 1 of 83.9% (CI 95%: 80.4% to 87.5%); TM/CA-ELISA and SU-ELISA – Cohen’s kappa of 86.0% (CI 95%: 82.6% to 89.5%), Gwet’s AC 1 of 86.9% (CI 95%: 83.6% to 90.1%). However, agreement between ELISAs was significantly related to the within-herd true prevalence – it was significantly lower (although still high) when within-herd true prevalence was moderate (Gwet’s AC 1 between 67.2% and 78.7%), whereas remained very high, when true within-herd prevalence was either >50% (Gwet’s AC 1 between 91.9% and 98.8%) or <10% (Gwet’s AC 1 between 94.7% and 98.4%). Concluding, the three different commercial ELISAs for SRLV infection in goats available on the market yield highly consistent results. However, their agreement is affected by the true within-herd prevalence in a tested population, and the worse (although still high) agreement should be expected, when the percentage of infected goats is moderate. [ABSTRACT FROM AUTHOR]

Background: Dengue is a high incidence arboviral disease in tropical countries around the world. Colombia is an endemic country due to the favourable environmental conditions for vector survival and spread. Dengue surveillance in Colombia is based in passive notification of cases, supporting monitoring, prediction, risk factor identification and intervention measures. Even though the surveillance network works adequately, disease mapping techniques currently developed and employed for many health problems are not widely applied. We select the Colombian city of Bucaramanga to apply Bayesian areal disease mapping models, testing the challenges and difficulties of the approach. Methods: We estimated the relative risk of dengue disease by census section (a geographical unit composed approximately by 1-20 city blocks) for the period January 2008 to December 2015. We included the covariates normalized difference vegetation index (NDVI) and land surface temperature (LST), obtained by satellite images. We fitted Bayesian areal models at the complete period and annual aggregation time scales for 2008-2015, with fixed and space-varying coefficients for the covariates, using Markov Chain Monte Carlo simulations. In addition, we used Cohen's Kappa agreement measures to compare the risk from year to year, and from every year to the complete period aggregation. Results: We found the NDVI providing more information than LST for estimating relative risk of dengue, although their effects were small. NDVI was directly associated to high relative risk of dengue. Risk maps of dengue were produced from the estimates obtained by the modeling process. The year to year risk agreement by census section was sligth to fair. Conclusion: The study provides an example of implementation of relative risk estimation using Bayesian models for disease mapping at small spatial scale with covariates. We relate satellite data to dengue disease, using an areal data approach, which is not commonly found in the literature. The main difficulty of the study was to find quality data for generating expected values as input for the models. We remark the importance of creating population registry at small spatial scale, which is not only relevant for the risk estimation of dengue but also important to the surveillance of all notifiable diseases. [ABSTRACT FROM AUTHOR]

The kappa coefficient is a widely used measure for assessing agreement on a nominal scale. Weighted kappa is an extension of Cohen's kappa that is commonly used for measuring agreement on an ordinal scale. In this article, it is shown that weighted kappa can be computed as a function of unweighted kappas. The latter coefficients are kappa coefficients that correspond to smaller contingency tables that are obtained by merging categories. [ABSTRACT FROM AUTHOR]

Aim The earth's land cover is often represented by discrete classes, and predicting shifts between these classes is a major goal in the field. One increasingly common approach is to build models that predict land cover classes with probabilities rather than discrete outcomes. Current assessment approaches have drawbacks when applied to these types of models. In this paper we present a new metric, which assesses agreement between model predictions and observations, while correcting for chance agreement. Location Global. Methods [ABSTRACT FROM AUTHOR]

It has long been a concern that performance measures of species distribution models react to attributes of the modeled entity arising from the input data structure rather than to model performance. Thus, the study of Allouche et al. ( Journal of Applied Ecology, 43, 1223, 2006) identifying the true skill statistics (TSS) as being independent of prevalence had a great impact. However, empirical experience questioned the validity of the statement. We searched for technical reasons behind these observations. We explored possible sources of prevalence dependence in TSS including sampling constraints and species characteristics, which influence the calculation of TSS. We also examined whether the widespread solution of using the maximum of TSS for comparison among species introduces a prevalence effect. We found that the design of Allouche et al. ( Journal of Applied Ecology, 43, 1223, 2006) was flawed, but TSS is indeed independent of prevalence if model predictions are binary and under the strict set of assumptions methodological studies usually apply. However, if we take realistic sources of prevalence dependence, effects appear even in binary calculations. Furthermore, in the widespread approach of using maximum TSS for continuous predictions, the use of the maximum alone induces prevalence dependence for small, but realistic samples. Thus, prevalence differences need to be taken into account when model comparisons are carried out based on discrimination capacity. The sources we identified can serve as a checklist to safely control comparisons, so that true discrimination capacity is compared as opposed to artefacts arising from data structure, species characteristics, or the calculation of the comparison measure (here TSS). [ABSTRACT FROM AUTHOR]

Summary: The present study tested if the accuracy of the VFA reading reproducibility is more affected by the statistical tool used or by the reader's level of expertise in 50 VFA from a population-based cohort, the OstéoLaus study. We found that uniform kappa and instruction reading with the ISCD/IOF VFA reading course both increased the accuracy of the reproducibility. Introduction: Vertebral fractures (VF) due to osteoporosis are under diagnosed. Screening osteoporosis in the general population allows improving management of fragility fracture. It consists to perform a dual X-ray absorptiometry and a spine X-ray to look at a VF. To reduce the dosage of radiation, prevalent or incident VF could be detected by DXA image. The aim of the present study was to test the reproducibility of vertebral fracture assessment (VFA) readings in a population-based cohort and to explore if the accuracy of the reproducibility is more affected by the statistical tool used or by the reader's level of expertise. Methods: We calculated the reproducibility of VFA reading by uniform and Cohen's kappa, comparing one expert and one non-expert, before and after an instructional on-line International Society of Clinical Densitometry (ISCD) /International Osteoporosis Foundation (IOF) course on VFA reading. We performed the analysis on 50 VFA from a population-based cohort, the OstéoLaus study. Results: Before the VFA reading course, reproducibility with Cohen's kappa was moderate to poor (0 to 0.520), good with the uniform kappa (0.796 to 0.958). After the course, both Cohen's kappa and uniform kappa statistically increased, ranging from 0.524 to 1.000. Conclusions: For female population-based cohort studies, we recommend using the uniform kappa and instructing a non-expert reader using the ISCD/IOF VFA reading course to correctly read and evaluate the reproducibility of the VFA reading. [ABSTRACT FROM AUTHOR]

Reliability assessment of a measure: the kappa statistic The Kappa coefficient is a measure of inter-rater agreement. This statistic measures observed agreement greater than chance and can range from -1 to 1. A value of zero indicates statistical independence and a value of 1 indicates a perfect agreement between observers. The value of Kappa is influenced by the prevalence of the evaluated condition: two observers can have a high observed agreement but low Kappa if the prevalence is very high or very low (paradox of the Kappa statistic). [ABSTRACT FROM AUTHOR]

Circular classifications are classification scales with categories that exhibit a certain periodicity. Since linear scales have endpoints, the standard weighted kappas used for linear scales are not appropriate for analyzing agreement between two circular classifications. A family of kappa coefficients for circular classifications is defined. The kappas differ only in one parameter. It is studied how the circular kappas are related and if the values of the circular kappas depend on the number of categories. It turns out that the values of the circular kappas can be strictly ordered in precisely two ways. The orderings suggest that the circular kappas are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for circular kappas that tend to produce higher values. [ABSTRACT FROM AUTHOR]

Background: Data collection techniques that routinely provide health system information at the local level are in demand and needed. LQAS is intended for use by local health teams to collect data at the district and sub-district levels. Our question is whether local health staff produce biased results as they are responsible for implementing the programs they also assess.Methods: This test-retest study replicates on a larger scale an earlier LQAS reliability assessment in Uganda. We conducted in two districts an LQAS survey using 15 local health staff as data collectors. A week later, the data collectors swapped districts, where they acted as disinterested non-local data collectors, repeating the LQAS survey with the same respondents. We analysed the resulting two data sets for agreement using Cohens' Kappa.Results: The average Kappa score for the knowledge indicators was k = 0.43 (SD = 0.16) and for practice indicators k = 0.63 (SD = 0.17). These scores show moderate agreement for knowledge indicators and substantial agreement for practice indicators. Analyses confirm that respondents were more knowledgeable on retest; no evidence of bias was found for practice indicators.Conclusion: The findings of this study are remarkably similar to those produced in the first reliability study. There is no evidence that using local healthcare staff to collect LQAS data biases data collection in an LQAS study. The bias observed in the knowledge indicators was most likely due to a 'practice effect', whereby respondents increased their knowledge as a result of completing the first survey; no corresponding effect was seen in the practice indicators. [ABSTRACT FROM AUTHOR]

Since an important goal of the youth welfare system is to prevent troubled adolescents from committing acts of delinquency in future, professional caregivers need to possess accurate knowledge about past behaviors in order to implement appropriate interventions. As part of a comprehensive study on youth in state care, adolescents at 30 residential care facilities in Switzerland were surveyed about past acts of delinquency, and their responses were compared to those of their professional caregivers to see how well they correlated. A sample of 267 male and female adolescents aged 11-18years completed questionnaires about the frequency, nature, and seriousness of different types of offenses, while a designated caregiver for each resident completed a corresponding questionnaire. The majority of residents (86.1%) reported having committed at least one offense, which confirms the prevalence of problem behaviors in this population and the need for strategies to prevent it. The overall rate of agreement between the residents and their caregivers was 77.2%, with both parties reporting that the resident had committed at least one offense in 69.7% of cases, and both reporting that no offense had been committed in 7.5% of cases. Agreement was substantially higher for offenses that were serious than for those that were minor or moderate. Cohen's kappa reached slight to moderate values with regard to individual and categorized offenses. Seriousness scales of delinquency for self-reports and caregiver reports were moderately associated. While the overall rate of agreement between the residents and their caregivers was high, increasing it still further might lead to improvements in strategies for the prevention of recidivism. [ABSTRACT FROM AUTHOR]

• Applying overall concordance rate alone is unreliable in evaluating consistency. • The agreement depends on the prevalence of each observer. • Combining various statistical methods in consistency analysis should be considered. [ABSTRACT FROM AUTHOR]

Autoantibodies against extractable nuclear antigens (ENA) play a pivotal role in the diagnosis and classification of systemic autoimmune rheumatic diseases (SARD). In recent years, newly developed methods have enabled the simultaneous and quantitative detection of multiple anti-ENA reactivities. However, data regarding the comparability of results obtained using different technologies across different platforms are scarce. In this study we compared eight different immunoassays, commonly used in current laboratory practice for detection of anti-ENA antibodies. Sixty patients suffering from different SARD, 10 inflammatory arthritis patients (disease controls) and 10 healthy blood donors were included in this comparative study. Sera were collected in 15 centers belonging to the Study Group on Autoimmune Diseases of the Italian Society of Clinical Pathology and Laboratory Medicine. We evaluated the analytical sensitivity, specificity and diagnostic accuracy of each method for antibodies to Sm, RNP, Ro60, Ro52, Scl70, CENP-B and Jo1. Cohen's kappa was used to analyze the agreement among methods. Average agreement among methods was 0.82, ranging from substantial (k = 0.72) to almost perfect (k = 0.92). However, while the specificity was very good for all methods, some differences emerged regarding the analytical sensitivity. Diagnostic performance of current technologies for anti-ENA antibody detection showed good comparability. However, as some differences exist among methods, laboratory scientists and clinicians must be aware of the diagnostic accuracy of the testing method in use. • Antibodies to ENA are important in the diagnosis of autoimmune rheumatic diseases. • Many different immunological methods are available to the clinical laboratory. • Diagnostic performance of current immunoassays show good comparability. • Though agreement among assays is good, differences exist in diagnostic accuracy. • Clinicians and laboratory scientists should be aware of these differences. [ABSTRACT FROM AUTHOR]

Observational studies in the field of sport are complicated by the added difficulty of having to analyse multiple, complex events or behaviours that may last just a fraction of a second. In this study, we analyse three aspects related to the reliability of data collected in such a study. The first aim was to analyse and compare the reliability of data sets assessed quantitatively (calculation of kappa statistic) and qualitatively (consensus agreement method). The second aim was to describe how, by ensuring the alignment of events, we calculated the kappa statistic for the order parameter using SDIS-GSEQ software (version 5.1) for data sets containing different numbers of sequences. The third objective was to describe a new consultative procedure designed to remove the confusion generated by discordant data sets and improve the reliability of the data. The procedure is called "consultative" because it involves the participation of a new observer who is responsible for consulting the existing observations and deciding on the definitive result. [ABSTRACT FROM AUTHOR]

In the 1880s, American scholars developed measures of association and chance for cross-classification tables that anticipated the more widely known work of Galton, Pearson, Yule, and Fisher. Three of the measures form the historical backdrop for the earliest known use of a joint probability measure that mirrored Bayes’ theorem long before the latter gained general interest among statisticians. The joint probability measure, which served as a foundational step in M. H. Doolittle's development of the first of the two “association ratios,” has not previously been reviewed in the statistical literature. It was reintroduced as if newly developed in a subfield of experimental psychology more than a century after Doolittle's work was published. It has flourished there, but it has not seen use in other academic venues. The article describes its properties and limitations and proposes that it be disseminated and debated beyond its current narrow application. The article notes that Doolittle's first association ratio can be expressed as another joint probability and that prior treatments in the literature are inconsistent with Doolittle's understanding of its purpose. The article also demonstrates that the equivalent of Cohen's kappa (κ)was developed by Doolittle in 1887, as his second association measure. [Received December 2014. Revised August 2015.] [ABSTRACT FROM AUTHOR]

Screening and diagnostic procedures often require a physician's subjective interpretation of a patient's test result using an ordered categorical scale to define the patient's disease severity. Because of wide variability observed between physicians' ratings, many large-scale studies have been conducted to quantify agreement between multiple experts' ordinal classifications in common diagnostic procedures such as mammography. However, very few statistical approaches are available to assess agreement in these large-scale settings. Many existing summary measures of agreement rely on extensions of Cohen's kappa. These are prone to prevalence and marginal distribution issues, become increasingly complex for more than three experts, or are not easily implemented. Here we propose a model-based approach to assess agreement in large-scale studies based upon a framework of ordinal generalized linear mixed models. A summary measure of agreement is proposed for multiple experts assessing the same sample of patients' test results according to an ordered categorical scale. This measure avoids some of the key flaws associated with Cohen's kappa and its extensions. Simulation studies are conducted to demonstrate the validity of the approach with comparison with commonly used agreement measures. The proposed methods are easily implemented using the software package R and are applied to two large-scale cancer agreement studies. Copyright © 2015 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Reliable measurements are fundamental for the empirical sciences. In observational research, measurements often consist of observers categorizing behavior into nominal-scaled units. Since the categorization is the outcome of a complex judgment process, it is important to evaluate the extent to which these judgments are reproducible, by having multiple observers independently rate the same behavior. A challenge in determining interrater agreement for timed-event sequential data is to develop clear objective criteria to determine whether two raters' judgments relate to the same event (the linking problem). Furthermore, many studies presently report only raw agreement indices, without considering the degree to which agreement can occur by chance alone. Here, we present a novel, free, and open-source toolbox (EasyDIAg) designed to assist researchers with the linking problem, while also providing chance-corrected estimates of interrater agreement. Additional tools are included to facilitate the development of coding schemes and rater training. [ABSTRACT FROM AUTHOR]

Bovine tuberculosis is a complex disease that is difficult to diagnose, control and eradicate and negatively impacts many farms. The objective of this study was to assess the degree of concordance between the results of histopathological examination and bacterial culture in the diagnosis of bovine tuberculosis lesions obtained in the laboratories certified by the Secretaría de Agricultura, Ganadería, Desarrollo Rural, Pesca y Alimentación (Ministry of Agriculture, Animal Production, Rural Development, Fisheries and Feeding) at the national level between January 2009 and December 2012 in Mexico. Tissue samples (10,818) from regular slaughter cattle that did not have tuberculin tests but had lesions suspected of having been caused by tuberculosis were sent to 10 authorized laboratories. Using Cohen's kappa to measure the reliability of the diagnosis, a general concordance was obtained between the histopathological examination and bacterial culture results with a kappa of 0.634 and a 95% (0.618 - 0.650) confidence interval (CI), which shows good concordance between the two techniques at the national level. The laboratory in Chihuahua had the highest kappa [k=0.784, 95% CI (0.754- 0.814)] and the laboratory of the La Laguna Region in the state of Coahuila had a low value of global concordance [k= 0.334, 95% CI (0.257-0.412)]. The number of positive samples in bacterial culture was low in Tamaulipas (21 samples), Sonora (41 samples), and Yucatán (45 samples) because tuberculosis prevalence in those states is ὄ 0.04%. The reason for the disagreement between the two tests among some laboratories includes factors such as a lack of economic resources, infrastructure or personnel training. Correct sampling procedures, storage facilities, and shipping and sample processes are important in optimizing the bacteriological and histopathological results and obtaining correct diagnoses and differential diagnoses of bovine tuberculosis. [ABSTRACT FROM AUTHOR]

Background: Various measures of observer agreement have been proposed for 2x2 tables. We examine the behavior of alternative measures of observer agreement for 2x2 tables. Methods: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. Results: Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. Conclusions: The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2x2 tables. [ABSTRACT FROM AUTHOR]

It is shown that if cell weights may be calculated from the data the chance-corrected Zegers-ten Berge coefficients for metric scales are special cases of Cohen's weighted kappa. The corrected coefficients include Pearson's product-moment correlation, Spearman's rank correlation and the intraclass correlation ICC(3 , 1). [ABSTRACT FROM AUTHOR]

Background Identification of cut-off values for key biomarkers of clinical risk is a useful clinical tool. The kappa coefficient is a popular descriptive statistical measure for summarising the cross classification of two nominal variables with identical classes. On the basis of this definition, I propose that the kappa coefficient can also be used to capture discrimination, in the same way that the receiver operating characteristic (ROC) curve is used in preventive epidemiology studies. Methods The statistics were determined using Cohen's kappa statistics for a gold standard and a continuous biomarker. The proposed design is compared with the ROC curve by applying it to articles on the metabolic syndrome and a colon cancer clinical trial. Results The two methods gave similar results. Moreover, Monte Carlo simulation results confirm that, from a power perspective, the proposed method is to be preferred. In general, the proposed method has higher power than the area under the ROC curve (AUC) for a study of positively correlative design. Conclusion Overall, the power performance of the proposed method is better that that of the AUC. [ABSTRACT FROM AUTHOR]