Search

The degree of inter-rater agreement is usually assessed through (Formula presented.) -type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some (Formula presented.) -type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four (Formula presented.) -type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies.

Evaluating the ability of a classifier to make predictions on unseen data and increasing it by tweaking the learning algorithm are two of the main reasons motivating the evaluation of classifier predictive performance. In this study the behavior of Balanced $$AC_1$$ A C 1 — a novel classifier accuracy measure — is investigated under different class imbalance conditions via a Monte Carlo simulation. The behavior of Balanced $$AC_1$$ A C 1 is compared against that of several well-known performance measures based on binary confusion matrix. Study results reveal the suitability of Balanced $$AC_1$$ A C 1 with both balanced and imbalanced data sets. A real example of the effects of class imbalance on the behavior of the investigated classifier performance measures is provided by comparing the performance of several machine learning algorithms in a churn prediction problem.

Decision making processes often rely on subjective evaluations provided by human raters. In the absence of a gold standard against which check evaluation trueness, rater's evaluative performance is generally measured through rater agreement coefficients. In this study some parametric and non-parametric inferential benchmarking procedures for characterizing the extent of rater agreement—assessed via kappa-type agreement coefficients—are illustrated. A Monte Carlo simulation study has been conducted to compare the performance of each procedure in terms of weighted misclassification rate computed for all agreement categories. Moreover, in order to investigate whether the procedures overestimate or underestimate the level of agreement, misclassifications have been computed also for each specific category alone. The practical application of coefficients and inferential benchmarking procedures has been illustrated via two real data sets exemplifying different experimental conditions so as to highlight performance differences due to sample size.

The degree of inter-rater agreement is usually assessed through

In this paper, a weighted variant of the normalized pairwise angular distance metric is proposed. The inclusion of position weights aims at penalizing inversions in the top of the ranking more than inversions in the tail of the ranking. The performance of the proposed weighted distance metric for assessing ranking dissimilarity and its impact on a procedure for testing inter-group heterogeneity have been investigated via a Monte Carlo simulation study under several scenarios—differing for group size, number of ranked alternatives and system of hypotheses—and compared against those obtained for the unweighted variant.

In this paper, a novel approach is proposed to analyze and interpret consumer preference data expressed as weak orderings (ie, preference chains) over a set of alternatives. A simple distance metric based on cosine similarity is used to assess the repeatability and reproducibility of consumer preferences. These concepts have been borrowed from metrology and can be usefully adopted to evaluate the meaningfulness of aggregated measures (eg, median preference chain) defined over the preference data set. The proposed approach is fully exploited through a real case study involving a group of consumers expressing their preferences over alternative types of pizza toppings.

In many research contexts, such as social science, marketing, education, psychology and medicine, it is frequently of interest to compare two or more groups of subjects (e.g. people of different gender, age or nationality), who are asked to rank a set of alternatives according to their personal liking or opinion, for investigating the presence of group effect. The common investigation aim is to detect customers with homogeneous preferences (or priorities) in order to serve each group as properly as possible. Several approaches have been proposed in the literature for testing ranking heterogeneity among groups of subjects. This paper focuses on an approach considering diversity as a generalization of the notion of variation and investigates the performance of a testing procedure for ranking heterogeneity based on the index of segregation power. The performance of the testing procedure has been investigated via a Monte Carlo simulation study under several scenarios, differing for group size, number of ranked alternatives and system of hypothesis. Furthermore, using a real data set, the testing procedure is exploited for investigating whether patient age and gender matter for patient prioritization of quality improvement in healthcare service.

Sensory evaluations are adopted in many fields for measuring and comparing sensory properties of products and improving their quality. The selection of panelists able to provide precise evaluations is a crucial issue to perform reliable sensory analysis. An agreement-based approach is here suggested in order to assess the quality of sensory data in terms of both panelist repeatability and panel reproducibility. The approach has been applied to two case studies involving untrained sensory panelists and trained teaching quality assessors, respectively. The results of the case studies show that although reproducibility can be assumed moderate for both groups of raters, repeatability is generally higher for the group of trained raters.

This paper proposes a new approach to assess consistency of preferences expressed as priorities over a set of alternatives. A procedure based on the analysis of variation has been adopted for testing time-homogeneity of preferences; the statistical behaviour of the testing procedure has been investigated via Monte Carlo simulation.

Students’ Evaluations of Teaching (SETs) are widely used as measures of teaching quality in Higher Education. A review of specialized literature evidences that researchers widely discuss whether SETs can be considered reliable measures of teaching quality evaluation. Though the controversy mainly refers to the role of students as assessors of teaching qual- ity, most of research studies on SETs focus on the design and validation of the evaluation procedure and even when the need of measuring SETs reliability is recognized, it is gener- ally indirectly assessed for the whole group of students by measuring inter-student agree- ment. In this paper the focus is on the direct assessment of the reliability of each student as a measurement instrument of teaching quality. An agreement-based approach is here adopted in order to assess student’s ability to provide consistent and stable evaluations; the sampling uncertainty is accounted for by building non-parametric bootstrap confidence intervals for the adopted agreement coefficients.

The quality of subjective evaluations provided by field experts (e.g. physicians or risk assessors) as well as by trained operators (e.g. visual inspectors) is often evaluated in terms of inter/intra rater agreement via kappa-type coefficients. Despite their popularity, these indices have long been criticized for being affected in complex ways by the presence of bias between raters and by the distributions of data across the rating categories (“prevalence”). This paper presents the results of a Monte Carlo simulation study aimed at investigating the robustness of four kappa-type indices (viz. Gwet’s AC2 and the weighted variants of Scott’s Pi, Cohen’s Kappa and Brennan-Prediger coefficients) taking into consideration the case of two series of ratings provided by the same rater (intra-rater agreement) or by two raters (inter-rater agreement). The robustness of the reviewed indices to changes in the frequency distribution of ratings across categories and in the agreement distribution between the two series of ratings has been analyzed across several simulation scenarios built by varying the sample size (i.e. number of rated items), the dimension of the rating scale, the frequency and agreement distributions between the series of ratings. Simulation results suggest that the level of agreement is sensitive to the distribution of items across the rating categories and to the dimension of rating scale but it is not influenced by the sample size. Among the reviewed indices, the Brennan–Prediger coefficient and Gwet’s AC2 are less sensitive to variation in the distribution of items across the categories for a fixed agreement distribution.

In several industries strategic and operational decisions rely on subjective evaluations provided by raters who are asked to score and/or classify group of items in terms of some technical properties (e.g. classification of faulty material by defect type) and/or perception aspects (e.g. comfort, quality, pain, pleasure, aesthetics). Because of the lack of a gold standard for classifying subjective evaluations as “true” or “false”, rater reliability is generally measured by assessing her/his precision via inter/intra-rater agreement coefficients. Agreement coefficients are useful only if their magnitude can be easily interpreted. A common practice is to apply a straightforward procedure to translate the magnitude of the adopted agreement coefficient into an extent of agreement via a benchmark scale. Many criticisms have been attached to this practice and in order to solve some of them, the adoption of a probabilistic approach to characterize the extent of agreement is recommended. In this study some probabilistic benchmarking procedures are discussed and compared via a wide Monte Carlo simulation study.

Students’ Evaluations of Teaching (SETs) are the most common way to measure teaching quality in Higher Education: they are assuming a strategic role in monitoring teaching quality, becoming helpful in taking the major formative and summative academic decisions. The majority of studies investigating SETs reliability focus on the instruments and the procedures adopted to collect students' evaluations rather than on the capability of the students as teaching quality assessors. In order to overcome this lack, a study has been carried out with the aim of measuring SETs reliability in terms of inter-student agreement and intra-student agreement. The results of our study show that the majority of students provided substantially repeatable evaluations whereas only a few students provided almost perfectly repeatable evaluations; the evaluations provided by different students generally slightly agreed, which means that the students did not share the same opinions and beliefs on teaching quality.

Non-Performing Loans (NPLs) are exposures in state of insolvency, i.e. loans whose collection by banks is uncertain in terms of the amount that will be returned and the time of return. In order to measure the recovery performance of a portfolio of NPLs, our idea is to draw a curve representing the recovery rates during time, here assumed discretized, for example, in years. In this way, the user can get simultaneously information about recovery rate and time to liquidate of the portfolio. However, the computation of this curve becomes infeasible when some elements of the portfolio are right censored, i.e. the recovery rate trajectory is known only until a particular year. In this talk, a new method of estimation of the recovery rate curve in case of right censored data is presented. It is studied from a theoretical point of view in a simulation study, where it is tested and compared with other methods. It is studied from a practical point of view, where it is applied to a real financial data set about some portfolios of Italian unsecured NPLs taken in charge by a specialized operator.

The literature on university-industry relations has gone beyond direct surveys by academic and industrial researchers, and now also focuses on co-patenting, co-working and research in coop eration. These activities represent specific channels of interaction, conveying knowledge flows of different “value”. Each forms of those interactions basically produce a different network and usually each network is analyzed and interpreted individually. In this study, we combine different collaboration relations among firms and among inventors (hired by individual firms/assignees). In particular, we use data on both patenting activities and research collaboration in projects among firms located in an Italian region, namely Friuli Venezia Giulia, which is characterized by a high concentration of research organizations and a lively sector of R&D small firms. Regional patenting activities give rise to a co-patenting networks (inventors- by-inventors), whereas collaboration in research projects leads to a firm-by-firm network. The combination of these two peculiar university-industry interactions when combined with affiliation network of investors and firms naturally leads to a multilevel network. The paper aims at exploring the underlying structure of the firms’ relationships in the region by means of a block- modeling approach suitable for linked/multilevel networks. Finally, we project all networks to the level of firms in order to verify the hypothesis that in- ventors’ cooperation on patents increases the probability of a tie between firms based on projects. The blockmodeling approach used allow us to show how cluster of inventors and firms at different levels are connected.