Search

Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F1 score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications.

The aim of this study was to identify school motivation profiles of Dutch 9th grade students in a four-dimensional motivation space, including mastery, performance, social and extrinsic motivation. Multiple clustering methods (K-means, K-medoids, restricted latent profile analysis) and multiple indices for selecting the optimal number of clusters were applied. The statistical selection methods did not completely concur on the optimal number of clusters, but a clear common denominator was provided by the Calinski-Harabasz index and the minimum and mean Silhouette values. All three indices indicated two clusters as the optimal number, regardless of the clustering method used: one cluster of 9th graders with high average scores on all dimensions and one cluster with low mean scores on all dimensions. In addition, we explored the substantive interpretation of multiple cluster solutions. It was discovered that most students are in clusters that can be classified into one of three profile types that may differ in level: (1) approximately equal mean scores on all dimensions, (2) relative high mean scores on mastery and social motivation, and (3) a relatively low mean score on performance motivation. The latter profile type is believed to be a new discovery.

This article reports outcomes of a quasi-experimental evaluation of Success for All (SfA), a comprehensive school reform program that has recently been introduced in the Netherlands. The Response to Intervention framework is used to describe how SfA supports students at different Tiers. The effects of SfA on five reading subskills were investigated for first-grade students at risk of reading problems. 299 students from two different cohorts were involved. Multilevel analyses demonstrated a significant effect of SfA on reading comprehension (ES = +0.26) in the first cohort. For the second cohort and the other reading subskills, mostly small positive effects of SfA were found, though these effects were not statistically significant. Furthermore, the relationship between tutoring intensity and reading achievement was examined. In the second cohort, a significant negative association of tutoring intensity with word and text reading skills was found. Implementation issues that may have impacted the outcomes are discussed.

The purpose of crying has recently become a topic of interest, with evidence supporting its interpersonal functions. The assumption that tears not only express a need for help, but in reaction also foster willingness to help in an observer, has received preliminary empirical support. The current study replicated previous work using a within-subject design with 140 subjects (50% female) who were exposed to images depicting male and female individuals crying, with half of both displaying visible tears and the others not displaying tears. Novel is our comparison of willingness to help across all possible gender combinations of tear display and observer. Potential mediation by perceived helplessness, friendliness, and connectedness of the depicted person was tested in male and female participants separately. We replicated the strong effect of tears on willingness to help, and showed this effect to be less potent for male dyads than for female or mixed ones, which is new to the literature. Perceived helplessness mediated the link between crying and helping, whereas perceived connectedness seemed only relevant for female participants, and perceived friendliness was not significant. Possible origins and implications of a differential gender function of crying are discussed.

Cronbach’s alpha is the most frequently used measure to investigate the reliability of measurement instruments. Despite its frequent use, many warn for misinterpretations of alpha. These claims about regular misunderstandings, however, are not based on empirical data. To understand how common such beliefs are, we conducted a survey study to test researchers’ knowledge of and beliefs about alpha. For this survey, we selected authors from recent papers, in which alpha was used. The results provide empirical evidence for the claims that researchers have difficulty interpreting alpha in a proper way. At the same time, we expounded the claims, by showing that whereas some beliefs are fairly typical, others are not so often seen. This non-technical paper, aimed at both statisticians and substantive researchers, is concluded by providing a few suggestions that could be helpful to get us out of the current stalemate regarding the usability of alpha and its alternatives.

Cohen’s kappa coefficient is commonly used for assessing agreement between classifications of two raters on a nominal scale. Three variants of Cohen’s kappa that can handle missing data are presented. Data are considered missing if one or both ratings of a unit are missing. We study how well the variants estimate the kappa value for complete data under two missing data mechanisms—namely, missingness completely at random and a form of missingness not at random. The kappa coefficient considered in Gwet ( Handbook of Inter-rater Reliability, 4th ed.) and the kappa coefficient based on listwise deletion of units with missing ratings were found to have virtually no bias and mean squared error if missingness is completely at random, and small bias and mean squared error if missingness is not at random. Furthermore, the kappa coefficient that treats missing ratings as a regular category appears to be rather heavily biased and has a substantial mean squared error in many of the simulations. Because it performs well and is easy to compute, we recommend to use the kappa coefficient that is based on listwise deletion of missing ratings if it can be assumed that missingness is completely at random or not at random.

Cohen’s kappa is the most widely used coefficient for assessing interobserver agreement on a nominal scale. An alternative coefficient for quantifying agreement between two observers is Bangdiwala’s B. To provide a proper interpretation of an agreement coefficient one must first understand its meaning. Properties of the kappa coefficient have been extensively studied and are well documented. Properties of coefficient B have been studied, but not extensively. In this paper, various new properties of B are presented. Category B-coefficients are defined that are the basic building blocks of B. It is studied how coefficient B, Cohen’s kappa, the observed agreement and associated category coefficients may be related. It turns out that the relationships between the coefficients are quite different for $$2\times 2$$ tables than for agreement tables with three or more categories.

Many external validity indices for comparing different clusterings of the same set of objects are overall measures: they quantify similarity between clusterings for all clusters simultaneously. Because a single number only provides a general notion of what is going on, the values of such overall indices (usually between 0 and 1) are often difficult to interpret. In this paper, we show that a class of normalizations of the mutual information can be decomposed into indices that contain information on the level of individual clusters. The decompositions (1) reveal that overall measures can be interpreted as summary statistics of information reflected in the individual clusters, (2) specify how these overall indices are related to individual clusters, and (3) show that the overall indices are affected by cluster size imbalance. We recom-mend to use measures for individual clusters since they provide much more detailed information than a single overall number

If a function d is metric, a well-known result is that d/(1 + d) is also metric. We consider m-ary analogs of the binary notion of semimetric, called hemi-metrics and super-metrics. The metrics are totally symmetric maps from Xm+1 into R≥0. It is shown that, if d is supermetric, then d/(1 + d) is also super-metric

Objectives: De Vet, Mokkink, Mosmuller, and Terwee (2017) discovered that the Spearman-Brown formula (SB formula) can be used to transform certain intraclass correlation coefficients (ICCs) for single measurements into the corresponding ICCs for average measurements, without knowledge of the variance components.Results: A formal proof of the discovery by De Vet et al. is presented. It is also proved that, vice versa, the SB formula can be used to transform certain ICCs for average measurements into the corresponding ICCs for single measurements. Furthermore, it is specified to which type of ICC the transformations can be applied. Finally, several illustrations of the transformations are presented.Conclusion: The discovery by De Vet et al. is quite remarkable because ICCs and the SB formula come from quite different scientific disciplines. (C) 2017 Elsevier Inc. All rights reserved.

Variable centering and scaling are functions that are typically used in data normalization. Various properties of centering and scaling functions are presented. It is shown that if we use two centering functions (or scaling functions) successively, the result depends on the order in which the functions are applied: the second function always cancels the centering or scaling of the first function. Furthermore, it is shown that if we use a centering and a scaling function successively, the result does not depend on the order in which the functions are applied. Moreover, certain sets of normalization functions turn out to be semigroups.

Accuracy assessment of classified imagery is an important task in remote sensing. A commonly used approach for assessing classification accuracy is based on the error matrix. The error matrix can be summarized by overall and category accuracy measures, e.g. the overall agreement and the user's and producer's accuracies. Two relatively new measures are the so-called quantity disagreement and allocation disagreement. In this article, relative versions of the category-level variants of these measures are defined. It is also shown how the new category measures are related to the overall quantity and allocation disagreement measures.

If a test consists of two parts the Spearman-Brown formula and Flanagan's coefficient (Cronbach's alpha) are standard tools for estimating the reliability. However, the coefficients may be inappropriate if their associated measurement models fail to hold. We study the robustness of reliability estimation in the two-part case to coefficient misspecification. We compare five reliability coefficients and study various conditions on the standard deviations and lengths of the parts. Various conditional upper bounds of the differences between the coefficients are derived. It is shown that the difference between the Spearman-Brown formula and Horst's formula is negligible in many cases. We conclude that all five reliability coefficients can be used if there are only small or moderate differences between the standard deviations and the lengths of the parts.