Search

Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F1 score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications.

The aim of this study was to identify school motivation profiles of Dutch 9th grade students in a four-dimensional motivation space, including mastery, performance, social and extrinsic motivation. Multiple clustering methods (K-means, K-medoids, restricted latent profile analysis) and multiple indices for selecting the optimal number of clusters were applied. The statistical selection methods did not completely concur on the optimal number of clusters, but a clear common denominator was provided by the Calinski-Harabasz index and the minimum and mean Silhouette values. All three indices indicated two clusters as the optimal number, regardless of the clustering method used: one cluster of 9th graders with high average scores on all dimensions and one cluster with low mean scores on all dimensions. In addition, we explored the substantive interpretation of multiple cluster solutions. It was discovered that most students are in clusters that can be classified into one of three profile types that may differ in level: (1) approximately equal mean scores on all dimensions, (2) relative high mean scores on mastery and social motivation, and (3) a relatively low mean score on performance motivation. The latter profile type is believed to be a new discovery.

This article reports outcomes of a quasi-experimental evaluation of Success for All (SfA), a comprehensive school reform program that has recently been introduced in the Netherlands. The Response to Intervention framework is used to describe how SfA supports students at different Tiers. The effects of SfA on five reading subskills were investigated for first-grade students at risk of reading problems. 299 students from two different cohorts were involved. Multilevel analyses demonstrated a significant effect of SfA on reading comprehension (ES = +0.26) in the first cohort. For the second cohort and the other reading subskills, mostly small positive effects of SfA were found, though these effects were not statistically significant. Furthermore, the relationship between tutoring intensity and reading achievement was examined. In the second cohort, a significant negative association of tutoring intensity with word and text reading skills was found. Implementation issues that may have impacted the outcomes are discussed.

In unsupervised machine learning, agreement between partitions is commonly assessed with so-called external validity indices. Researchers tend to use and report indices that quantify agreement between two partitions for all clusters simultaneously. Commonly used examples are the Rand index and the adjusted Rand index. Since these overall measures give a general notion of what is going on, their values are usually hard to interpret. The goal of this study is to provide a thorough understanding of the adjusted Rand index as well as many other partition comparison indices based on counting object pairs. It is shown that many overall indices based on the pair-counting approach can be decomposed into indices that reflect the degree of agreement on the level of individual clusters. The decompositions (1) show that the overall indices can be interpreted as summary statistics of the agreement on the cluster level, (2) specify how these overall indices are related to the indices for individual clusters, and (3) show that the overall indices are affected by cluster size imbalance: if cluster sizes are unbalanced these overall measures will primarily reflect the degree of agreement between the partitions on the large clusters, and will provide much less information on the agreement on smaller clusters. Furthermore, the value of Rand-like indices is determined to a large extent by the number of pairs of objects that are not joined in either of the partitions.

Kappa coefficients are commonly used for quantifying reliability on a categorical scale, whereas correlation coefficients are commonly applied to assess reliability on an interval scale. Both types of coefficients can be used to assess the reliability of ordinal rating scales. In this study, we compare seven reliability coefficients for ordinal rating scales: the kappa coefficients included are Cohen’s kappa, linearly weighted kappa, and quadratically weighted kappa; the correlation coefficients included are intraclass correlation ICC(3,1), Pearson’s correlation, Spearman’s rho, and Kendall’s tau-b. The primary goal is to provide a thorough understanding of these coefficients such that the applied researcher can make a sensible choice for ordinal rating scales. A second aim is to find out whether the choice of the coefficient matters. We studied to what extent we reach the same conclusions about inter-rater reliability with different coefficients, and to what extent the coefficients measure agreement in a similar way, using analytic methods, and simulated and empirical data. Using analytical methods, it is shown that differences between quadratic kappa and the Pearson and intraclass correlations increase if agreement becomes larger. Differences between the three coefficients are generally small if differences between rater means and variances are small. Furthermore, using simulated and empirical data, it is shown that differences between all reliability coefficients tend to increase if agreement between the raters increases. Moreover, for the data in this study, the same conclusion about inter-rater reliability was reached in virtually all cases with the four correlation coefficients. In addition, using quadratically weighted kappa, we reached a similar conclusion as with any correlation coefficient a great number of times. Hence, for the data in this study, it does not really matter which of these five coefficients is used. Moreover, the four correlation coefficients and quadratically weighted kappa tend to measure agreement in a similar way: their values are very highly correlated for the data in this study.

The purpose of crying has recently become a topic of interest, with evidence supporting its interpersonal functions. The assumption that tears not only express a need for help, but in reaction also foster willingness to help in an observer, has received preliminary empirical support. The current study replicated previous work using a within-subject design with 140 subjects (50% female) who were exposed to images depicting male and female individuals crying, with half of both displaying visible tears and the others not displaying tears. Novel is our comparison of willingness to help across all possible gender combinations of tear display and observer. Potential mediation by perceived helplessness, friendliness, and connectedness of the depicted person was tested in male and female participants separately. We replicated the strong effect of tears on willingness to help, and showed this effect to be less potent for male dyads than for female or mixed ones, which is new to the literature. Perceived helplessness mediated the link between crying and helping, whereas perceived connectedness seemed only relevant for female participants, and perceived friendliness was not significant. Possible origins and implications of a differential gender function of crying are discussed.

Cronbach’s alpha is the most frequently used measure to investigate the reliability of measurement instruments. Despite its frequent use, many warn for misinterpretations of alpha. These claims about regular misunderstandings, however, are not based on empirical data. To understand how common such beliefs are, we conducted a survey study to test researchers’ knowledge of and beliefs about alpha. For this survey, we selected authors from recent papers, in which alpha was used. The results provide empirical evidence for the claims that researchers have difficulty interpreting alpha in a proper way. At the same time, we expounded the claims, by showing that whereas some beliefs are fairly typical, others are not so often seen. This non-technical paper, aimed at both statisticians and substantive researchers, is concluded by providing a few suggestions that could be helpful to get us out of the current stalemate regarding the usability of alpha and its alternatives.

Cohen’s kappa coefficient is commonly used for assessing agreement between classifications of two raters on a nominal scale. Three variants of Cohen’s kappa that can handle missing data are presented. Data are considered missing if one or both ratings of a unit are missing. We study how well the variants estimate the kappa value for complete data under two missing data mechanisms—namely, missingness completely at random and a form of missingness not at random. The kappa coefficient considered in Gwet ( Handbook of Inter-rater Reliability, 4th ed.) and the kappa coefficient based on listwise deletion of units with missing ratings were found to have virtually no bias and mean squared error if missingness is completely at random, and small bias and mean squared error if missingness is not at random. Furthermore, the kappa coefficient that treats missing ratings as a regular category appears to be rather heavily biased and has a substantial mean squared error in many of the simulations. Because it performs well and is easy to compute, we recommend to use the kappa coefficient that is based on listwise deletion of missing ratings if it can be assumed that missingness is completely at random or not at random.

Cohen’s kappa is the most widely used coefficient for assessing interobserver agreement on a nominal scale. An alternative coefficient for quantifying agreement between two observers is Bangdiwala’s B. To provide a proper interpretation of an agreement coefficient one must first understand its meaning. Properties of the kappa coefficient have been extensively studied and are well documented. Properties of coefficient B have been studied, but not extensively. In this paper, various new properties of B are presented. Category B-coefficients are defined that are the basic building blocks of B. It is studied how coefficient B, Cohen’s kappa, the observed agreement and associated category coefficients may be related. It turns out that the relationships between the coefficients are quite different for $$2\times 2$$ tables than for agreement tables with three or more categories.

2 × 2 tables are encountered in various scientific disciplines, including biomedical, social and behavioral sciences, economics and ecology. In the literature many different similarity measures have been proposed that can be used to further summarize the information in a 2 × 2 table. Many of these measures are just functions of the four cells. In this paper the important ones are reviewed. Furthermore, it is shown how various similarity measures are related to one another by considering certain general similarity measures that have various important similarity measures as special cases. The presented overview may provide insights that may be helpful to researchers from various scientific disciplines in deciding what similarity measure to use in applications or for studying theoretical properties.

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an ‘absence’ category (for example, no disorder), whereas the second type does include an ‘absence’ category. Cohen’s unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values.

Many external validity indices for comparing different clusterings of the same set of objects are overall measures: they quantify similarity between clusterings for all clusters simultaneously. Because a single number only provides a general notion of what is going on, the values of such overall indices (usually between 0 and 1) are often difficult to interpret. In this paper, we show that a class of normalizations of the mutual information can be decomposed into indices that contain information on the level of individual clusters. The decompositions (1) reveal that overall measures can be interpreted as summary statistics of information reflected in the individual clusters, (2) specify how these overall indices are related to individual clusters, and (3) show that the overall indices are affected by cluster size imbalance. We recom-mend to use measures for individual clusters since they provide much more detailed information than a single overall number

The Gini coefficient is a measure of statistical dispersion that is commonly used as a measure of inequality of income, wealth or opportunity. Empirical research has shown that the coefficient may have a nonnegligible downward bias when data are grouped. It is unknown under which grouping conditions the downward bias occurs. In this note it is shown that the Gini coefficient strictly decreases if the data are partitioned into equal sized groups.

If a function d is metric, a well-known result is that d/(1 + d) is also metric. We consider m-ary analogs of the binary notion of semimetric, called hemi-metrics and super-metrics. The metrics are totally symmetric maps from Xm+1 into R≥0. It is shown that, if d is supermetric, then d/(1 + d) is also super-metric

Circular classifications are classification scales with categories that exhibit a certain periodicity. Since linear scales have endpoints, the standard weighted kappas used for linear scales are not appropriate for analyzing agreement between two circular classifications. A family of kappa coefficients for circular classifications is defined. The kappas differ only in one parameter. It is studied how the circular kappas are related and if the values of the circular kappas depend on the number of categories. It turns out that the values of the circular kappas can be strictly ordered in precisely two ways. The orderings suggest that the circular kappas are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for circular kappas that tend to produce higher values.

The kappa statistic is a widely used as a measure for quantifying agreement between two nominal classifications. The statistic has been extended to the case of two normalized fuzzy classifications. In this paper we define category kappas for quantifying agreement on a particular category of two normalized fuzzy classifications. The overall fuzzy kappa is a weighted average of the proposed category kappas. Since the value of the overall kappa lies between the minimum and maximum values of the category kappas, the overall kappa, in a way, summarizes the agreement reflected in the category kappas. The overall kappa meaningfully reflects the degree of agreement between the fuzzy classifications if the category kappas are approximately equal. If this is not the case, it is more informative to report the category kappas. HighlightsFuzzy kappas for quantifying agreement on a particular category are defined.The fuzzy kappa of Dou et al. is a weighted average of the proposed category kappas.With two categories the category kappas and overall kappa coincide.Adding all other membership functions presents an alternative definition.

Objectives: De Vet, Mokkink, Mosmuller, and Terwee (2017) discovered that the Spearman-Brown formula (SB formula) can be used to transform certain intraclass correlation coefficients (ICCs) for single measurements into the corresponding ICCs for average measurements, without knowledge of the variance components.Results: A formal proof of the discovery by De Vet et al. is presented. It is also proved that, vice versa, the SB formula can be used to transform certain ICCs for average measurements into the corresponding ICCs for single measurements. Furthermore, it is specified to which type of ICC the transformations can be applied. Finally, several illustrations of the transformations are presented.Conclusion: The discovery by De Vet et al. is quite remarkable because ICCs and the SB formula come from quite different scientific disciplines. (C) 2017 Elsevier Inc. All rights reserved.

Variable centering and scaling are functions that are typically used in data normalization. Various properties of centering and scaling functions are presented. It is shown that if we use two centering functions (or scaling functions) successively, the result depends on the order in which the functions are applied: the second function always cancels the centering or scaling of the first function. Furthermore, it is shown that if we use a centering and a scaling function successively, the result does not depend on the order in which the functions are applied. Moreover, certain sets of normalization functions turn out to be semigroups.

Accuracy assessment of classified imagery is an important task in remote sensing. A commonly used approach for assessing classification accuracy is based on the error matrix. The error matrix can be summarized by overall and category accuracy measures, e.g. the overall agreement and the user's and producer's accuracies. Two relatively new measures are the so-called quantity disagreement and allocation disagreement. In this article, relative versions of the category-level variants of these measures are defined. It is also shown how the new category measures are related to the overall quantity and allocation disagreement measures.

If a test consists of two parts the Spearman-Brown formula and Flanagan's coefficient (Cronbach's alpha) are standard tools for estimating the reliability. However, the coefficients may be inappropriate if their associated measurement models fail to hold. We study the robustness of reliability estimation in the two-part case to coefficient misspecification. We compare five reliability coefficients and study various conditions on the standard deviations and lengths of the parts. Various conditional upper bounds of the differences between the coefficients are derived. It is shown that the difference between the Spearman-Brown formula and Horst's formula is negligible in many cases. We conclude that all five reliability coefficients can be used if there are only small or moderate differences between the standard deviations and the lengths of the parts.

Students in secondary education inevitably favour some subjects more than other subjects. This appraisal may affect how motivation relates to performance in these subjects. Whereas autonomous motivation is generally linked to positive school outcomes, the effect of controlled motivation is less clear. This study specifically focused on the associations of controlled motivation with performance in the context of favoured and disfavoured subjects. In the present study, secondary school students (N = 918) identified 2 favoured and 2 disfavoured subjects. Hierarchical linear modelling was performed to investigate the relationship of autonomous and controlled motivation with performance in these subjects. Results showed that autonomous motivation positively related to performance in both types of subjects. The association of controlled motivation with performance was negative in both contexts, and more negative in disfavoured subjects. For teaching practice, this means that teachers should always stimulate autonomous motivation, even for negatively appraised subjects.

Cohen’s kappa is a standard tool for the analysis of agreement in a 2 × 2 reliability study. Researchers are frequently only interested in the kappa-value of a sample. Various authors have observed that if two pairs of raters have the same amount of observed agreement, the pair whose marginal distributions are more similar to each other may have a lower kappa-value than the pair with more divergent marginal distributions. Here we present exact formulations of some of these properties. The results provide a better understanding of the 2 × 2 kappa for situations where it is used as a sample statistic.

Coefficient alpha is the most commonly used internal consistency reliability coefficient. Alpha is the mean of all possible k-split alphas if the items are divided into k parts of equal size. This result gives proper interpretations of alpha: interpretations that also hold if (some of) its assumptions are not valid. Here we consider the cases where the items cannot be split into parts of equal size. It is shown that if a k-split is made such that the items are divided as evenly as possible, the difference between alpha and the mean of all possible k-split alphas can be made arbitrarily small by increasing the number of items.

Kappa coefficients are standard tools for summarizing the information in cross-classifications of two categorical variables with identical categories, here called agreement tables. When two categories are combined the kappa value usually either increases or decreases. There is a class of agreement tables for which the value of Cohen’s kappa remains constant when two categories are combined. It is shown that for this class of tables all special cases of symmetric kappa coincide and that the value of symmetric kappa is not affected by any partitioning of the categories.

It is shown that a symmetric kappa corresponding to a c × c table with c ⩾ 3 categories can be written as a function of the unweighted kappa corresponding to the same table and the c(c − 1)/2 distinct unweighted kappas associated with the (c − 1) × (c − 1) tables that are obtained by combining two categories. The result is a new MGB-type result.

Cohen’s kappa is a popular descriptive statistic for measuring agreement between two raters on a nominal scale. Various authors have generalized Cohen’s kappa to the case of m ≥ 2 raters. We consider a family of multi-rater kappas that are based on the concept of g -agreement ( g = 2 , 3 , … , m ), which refers to the situation in which it is decided that there is agreement if g out of m raters assign an object to the same category. For the family of multi-rater kappas we prove the following existence theorem: In the case of three or more categories there exists for each multi-rater kappa κ ( m , g ) two categories such that, when combined, the κ ( m , g ) value increases. In addition, there exist two categories such that, when combined, the κ ( m , g ) value decreases.

The κ coefficient is a popular descriptive statistic for summarizing an agreement table. It is sometimes desirable to combine some of the categories, for example, when categories are easily confused, and then calculate κ for the collapsed table. Since the categories of an agreement table are nominal and the order in which the categories of a table are listed is irrelevant, combining categories of an agreement table is identical to partitioning the categories in subsets. In this paper we prove that given a partition type of the categories, the overall κ -value of the original table is a weighted average of the κ -values of the collapsed tables corresponding to all partitions of that type. The weights are the denominators of the kappas of the subtables. An immediate consequence is that Cohen’s κ can be interpreted as a weighted average of the κ -values of the agreement tables corresponding to all non-trivial partitions. The κ -value of the 2 × 2 table that is obtained by combining all categories other than the one of current interest into a single “all others” category, reflects the reliability of the individual category. Since the overall κ -value is a weighted average of these 2 × 2 κ -values the category reliability indicates how a category contributes to the overall κ -value. It would be good practice to report both the overall κ -value and the category reliabilities of an agreement table.

Cohen's kappa is presently a standard tool for the analysis of agreement in a 2 × 2 reliability study, and weighted kappa is a standard statistic for summarizing a 2 × 2 validity study. The special cases of weighted kappa, for example Cohen's kappa, are chance-corrected measures of association. For various measures of 2 × 2 association it has been observed in the literature that, after correction for chance, they coincide with a special case of weighted kappa. This paper presents the general function, linear in both numerator and denominator, that becomes weighted kappa after correction for chance.

The kappa coefficient is a popular descriptive statistic for summarizing the cross classification of two nominal variables with identical categories. It has been frequently observed in the literature that combining two categories increases the value of kappa. In this note we prove the following existence theorem for kappa: For any nontrivial k × k agreement table with k ∈ N ≥ 3 categories, there exist two categories such that, when combined, the kappa value of the collapsed ( k − 1 ) × ( k − 1 ) agreement table is higher than the original kappa value. In addition, there exist two categories such that, when combined, the kappa value of the collapsed table is smaller than the original kappa value.

Suppose two judges each classify a group of objects into one of several nominal categories. It has been observed in the literature that, for fixed observed agreement between the judges, Cohen’s kappa penalizes judges with similar marginals compared to judges who produce different marginals. This paper presents a formal proof of this phenomenon.

The paper presents inequalities between four descriptive statistics that can be expressed in the form [P−E(P)]/[1−E(P)], where P is the observed proportion of agreement of a k×k table with identical categories, and E(P) is a function of the marginal probabilities. Scott’s π is an upper bound of Goodman and Kruskal’s λ and a lower bound of both Bennett et al. S and Cohen’s κ. We introduce a concept for the marginal probabilities of the k×k table called weak marginal symmetry. Using the rearrangement inequality, it is shown that Bennett et al. S is an upper bound of Cohen’s κ if the k×k table is weakly marginal symmetric.

k-Adic formulations (for groups of objects of size k) of a variety of 2-adic similarity coefficients (for pairs of objects) for binary (presence/absence) data are presented. The formulations are not functions of 2-adic similarity coefficients. Instead, the main objective of the the paper is to present k-adic formulations that reflect certain basic characteristics of, and have a similar interpretation as, their 2-adic versions. Two major classes are distinguished. The first class is referred to as Bennani-Heiser similarity coefficients, which contains all coefficients that can be defined using just the matches, the number of attributes that are present and that are absent in k objects, and the total number of attributes. The coefficients in the second class can be formulated as functions of Dice’s association indices.

Correspondence analysis is an exploratory technique for analyzing the interaction in a contingency table. Tables with meaningful orders of the rows and columns may be analyzed using a model-based correspondence analysis that incorporates order constraints. However, if there exists a permutation of the rows and columns of the contingency table so that the rows are regression dependent on the columns and, vice versa, the columns are regression dependent on the rows, then both implied orders are reflected in the first dimension of the unconstrained correspondence analysis [Schriever, B.F., 1983. Scaling of order dependent categorical variables with correspondence analysis. International Statistical Review 51, 225-238]. Thus, using unconstrained correspondence analysis, we may still find that the data fit an ordinal stochastic model. Fit measures are formulated that may be used to verify whether the re-ordered contingency table is regression dependent in either the rows or columns. Using several data examples, it is shown that the fit indices may complement the usual geometric interpretation of the unconstrained correspondence analysis solution in low-dimensional space.

It is shown that one can calculate the Hubert-Arabie adjusted Rand index by first forming the fourfold contingency table counting the number of pairs of objects that were placed in the same cluster in both partitions, in the same cluster in one partition but in different clusters in the other partition, and in different clusters in both, and then computing Cohen's ? on this fourfold table.

Student performance is related to motivation to learn. As motivation generally declines during lower secondary education, one might expect performance to decline as well during this period. Though, until now, it has been unclear whether this pattern exists. In the present study, we examined student performance during the early years of secondary education from a developmental perspective. Participants were 1544 Dutch secondary school students across three grades (grades 7 to 9). To investigate student performance trends, we analysed report card grades by using hierarchical linear modelling with two levels (level 1, time point; level 2, student). Potential moderators to be examined were (1) gender, (2) school type and (3) initial level. A linear decline in report card grades from grade 7 to 9 was found for boys and girls, in all school types, and regardless of initial level. Two variables moderated the steepness of the decline: school type and initial level. Gender and school type had a main effect on performance level. The same pattern was observed for the subset of 'core subjects'-Dutch, English and mathematics. Motivational and cognitive factors that may explain the performance decline are discussed.

We signal and discuss common methodological errors in agreement studies and the use of kappa indices, as found in publications in the medical and behavioural sciences. Our analysis is based on a proposed statistical model that is in line with the typical models employed in metrology and measurement theory. A first cluster of errors is related to nonrandom sampling, which results in a potentially substantial bias in the estimated agreement. Second, when class prevalences are strongly nonuniform, the use of the kappa index becomes precarious, as its large partial derivatives result in typically large standard errors of the estimates. In addition, the index reflects rather one-sidedly in such cases the consistency of the most prevalent class, or the class prevalences themselves. A final cluster of errors concerns interpretation pitfalls, which may lead to incorrect conclusions based on agreement studies. These interpretation issues are clarified on the basis of the proposed statistical modelling. The signalled errors are illustrated from actual studies published in prestigious journals. The analysis results in a number of guidelines and recommendations for agreement studies, including the recommendation to use alternatives to the kappa index in certain situations.

The kappa statistic is commonly used for quantifying inter-rater agreement on a nominal scale. In this review article we discuss five interpretations of this popular coefficient. Kappa is a function of the proportion of observed and expected agreement, and it may be interpreted as the proportion of agreement corrected for chance. Furthermore, kappa may be interpreted as the average category reliability as well as an intraclass correlation.

Two test theoretical approaches to item analysis are compared, an approach based on homogeneity analysis and one based on item response theory. The literature on the relationship between the two approaches is briefly reviewed. The paper contains a contribution to the relationship between the two approaches for the case that the scores are dichotomous and a single latent variable is assumed to underlie the data. A loss function is proposed for modelling item response functions with two parameters, one for discrimination and one for difficulty. It turns out that the loss of the proposed loss function is related to loss of homogeneity. Demonstrations with simulated data are used to evaluate the proposed method.

Application of Cohen's weighted kappa for inter-rater agreement requires the specification of a weighting matrix. An explicit formula for the determinants of the principal minors of the weighting matrix with Cicchetti-Allison weights is derived. Since all determinants are strictly positive, it follows that the Cicchetti-Allison weighting matrix is positive definite.

Weighted kappa is a measure that is commonly used for quantifying similarity between two ordinal variables with identical categories. Additive kappa is a special case of weighted kappa that allows the researcher to specify distances between adjacent categories. It is shown that additive kappa is a weighted average of the additive kappas of all collapsed tables of a specific size. It follows that, if the reliability of a categorical rating instrument is assessed with additive kappa, the reliability can be increased by combining categories.

This paper presents a simple rescaling of the odds ratio that transforms the association measure into the weighted kappa statistic for a 2×2 table.

It is shown that if cell weights may be calculated from the data the chance-corrected Zegers-ten Berge coefficients for metric scales are special cases of Cohen's weighted kappa. The corrected coefficients include Pearson's product-moment correlation, Spearman's rank correlation and the intraclass correlation ICC(3, 1).