Search

The degree of inter-rater agreement is usually assessed through (Formula presented.) -type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some (Formula presented.) -type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four (Formula presented.) -type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies.

Evaluating the ability of a classifier to make predictions on unseen data and increasing it by tweaking the learning algorithm are two of the main reasons motivating the evaluation of classifier predictive performance. In this study the behavior of Balanced $$AC_1$$ A C 1 — a novel classifier accuracy measure — is investigated under different class imbalance conditions via a Monte Carlo simulation. The behavior of Balanced $$AC_1$$ A C 1 is compared against that of several well-known performance measures based on binary confusion matrix. Study results reveal the suitability of Balanced $$AC_1$$ A C 1 with both balanced and imbalanced data sets. A real example of the effects of class imbalance on the behavior of the investigated classifier performance measures is provided by comparing the performance of several machine learning algorithms in a churn prediction problem.

Decision making processes often rely on subjective evaluations provided by human raters. In the absence of a gold standard against which check evaluation trueness, rater's evaluative performance is generally measured through rater agreement coefficients. In this study some parametric and non-parametric inferential benchmarking procedures for characterizing the extent of rater agreement—assessed via kappa-type agreement coefficients—are illustrated. A Monte Carlo simulation study has been conducted to compare the performance of each procedure in terms of weighted misclassification rate computed for all agreement categories. Moreover, in order to investigate whether the procedures overestimate or underestimate the level of agreement, misclassifications have been computed also for each specific category alone. The practical application of coefficients and inferential benchmarking procedures has been illustrated via two real data sets exemplifying different experimental conditions so as to highlight performance differences due to sample size.

The degree of inter-rater agreement is usually assessed through

In this paper, a weighted variant of the normalized pairwise angular distance metric is proposed. The inclusion of position weights aims at penalizing inversions in the top of the ranking more than inversions in the tail of the ranking. The performance of the proposed weighted distance metric for assessing ranking dissimilarity and its impact on a procedure for testing inter-group heterogeneity have been investigated via a Monte Carlo simulation study under several scenarios—differing for group size, number of ranked alternatives and system of hypotheses—and compared against those obtained for the unweighted variant.

This paper presents an innovative safety training method based on digital ergonomics simulations and serious games, which are games that focus on education. Digital ergonomics is intended to disseminate the culture of safety among workers, while serious games are used to train the operators on specific safety procedures and verify their skills. The results of the experimentation in a real industrial environment showed that, compared to the traditional training methodology, multimedia contents and quantitative ergonomic analyses improve the level of attention and the awareness of the workers about their own safety. However, serious games turned out to be promising training tools with regard to standard operating procedures that are usually difficult or dangerous to simulate in a real working scenario without stopping production. Practitioner summary: Digital ergonomics and serious games are used to disseminate the culture of safety among the workers and for safety training. Our results show that the proposed methodology improves the level of attention and provides a better feedback about the actual skills of the workers than the standard educational strategies. Abbreviations.

In this paper, a novel approach is proposed to analyze and interpret consumer preference data expressed as weak orderings (ie, preference chains) over a set of alternatives. A simple distance metric based on cosine similarity is used to assess the repeatability and reproducibility of consumer preferences. These concepts have been borrowed from metrology and can be usefully adopted to evaluate the meaningfulness of aggregated measures (eg, median preference chain) defined over the preference data set. The proposed approach is fully exploited through a real case study involving a group of consumers expressing their preferences over alternative types of pizza toppings.

With the growth of industrial process sophistication, new horizons have opened up for modern metrology. Here, we focus on metrological aspects of multi-agent assessment, expressed by prioritization. Specifically, we examine the case of a group of agents expressing their preferences over a fixed set of alternatives by means of weak orders. Preference chains are compared via simple distance metrics based on cosine similarity. Following a metrological approach, the agents’ precision in performing prioritization tasks is evaluated in terms of repeatability and reproducibility. This is done by analyzing the preference variations in different trials of the same prioritization task and among different agents involved in the prioritization process. The paper introduces a median concept for aggregating multi-agent preference orderings to obtain a collective solution. The proposed strategy is illustrated through a case study of consumers expressing their preferences regarding a set of alternatives. Some further research opportunities are also discussed.

In many research contexts, such as social science, marketing, education, psychology and medicine, it is frequently of interest to compare two or more groups of subjects (e.g. people of different gender, age or nationality), who are asked to rank a set of alternatives according to their personal liking or opinion, for investigating the presence of group effect. The common investigation aim is to detect customers with homogeneous preferences (or priorities) in order to serve each group as properly as possible. Several approaches have been proposed in the literature for testing ranking heterogeneity among groups of subjects. This paper focuses on an approach considering diversity as a generalization of the notion of variation and investigates the performance of a testing procedure for ranking heterogeneity based on the index of segregation power. The performance of the testing procedure has been investigated via a Monte Carlo simulation study under several scenarios, differing for group size, number of ranked alternatives and system of hypothesis. Furthermore, using a real data set, the testing procedure is exploited for investigating whether patient age and gender matter for patient prioritization of quality improvement in healthcare service.

Aircraft seat is rated as the most unsatisfying aspect of flying; under-standing the main factors impacting on passenger’s evaluations can provide a concrete opportunity for airlines to improve seat comfort and thus enhance pas-senger satisfaction and loyalty. Although there is a great deal of interest, the re-search on effective assessment strategies for subjective comfort is still underde-veloped. In this study a model-based approach for the analysis of subjective comfort data is suggested. The model adopted can be interpreted as a parametric version of the psychological process generating comfort ratings. The proposed approach is exploited through a case study concerning comfort assessment of aircraft seats designed for regional flights.

In order to investigate differences in comfort and discomfort experiences amongst different regions of the world (America, Asia and Europe), a cross cultural study was performed. A questionnaire was sent to participants out in nine countries (Brazil, Canada, the USA, China, Indonesia, Thailand, Germany, Italy and the Netherlands). In total 795 participants completed the questionnaires. All countries score the comfort of a luxurious bed higher than a simple bed, first-class seats higher than economy class and all countries rate the comfort lower when the duration of sitting increases. The study suggests that in the USA and Canada softer beds, hammocks, more luxurious seats and softer pillows are scored as more comfortable compared with the other countries. There are indications that China and Germany prefer a harder mattress than in the other countries. For pillows, the differences between countries are large, which might show that much is influenced by habitude or hesitation to use something new. The Asian countries score the comfort of a brace neck pillow higher, which might be because these participants better realise the benefits better or feel less concerned to wear something that might give the appearance of an orthotic device. Further studies are needed to confirm these suggestions. The study shows that obvious differences are seen in all countries, which makes the construct of comfort internationally comparable. Practitioner summary: In designing and manufacturing globally, it is important to know how different parts of the world experience (dis)comfort. This study did not show large cultural differences amongst nine countries. Some differences emerge regarding pillows, perhaps as differences in sleeping habits play a role. Abbreviations: MANOVA: multivariate analysis of variance; VDA: Vargha and Delaney’s A statistic; USA: United States of America.

BACKGROUND: Aircraft seating comfort has a significant impact on passenger on-board experience. Its assessment requires the adoption of well-designed strategies for data collection as well as appropriate data analysis methods in order to obtain accurate and reliable results. OBJECTIVES: This paper focuses on the assessment of aircraft seating comfort based on subjective comfort responses collected during laboratory experiments and taking into account seat features and passenger characteristics. METHODS: The subjective comfort evaluations have been analyzed using a model-based approach to investigate the relationship between overall seating comfort and specific seat/user characteristics. RESULTS: The results show that the overall seating comfort perception is significantly influenced by the thickness of the seat pan, the backrest position (upright or reclined), the age of the passenger and the passenger perception of being comfortably supported at the lumbar region. CONCLUSIONS: The adopted model-based approach allows the analysis of subjective seating comfort data taking into account their ordinal nature as well as the dependency between evaluations provided by the same subject.

Sensory evaluations are adopted in many fields for measuring and comparing sensory properties of products and improving their quality. The selection of panelists able to provide precise evaluations is a crucial issue to perform reliable sensory analysis. An agreement-based approach is here suggested in order to assess the quality of sensory data in terms of both panelist repeatability and panel reproducibility. The approach has been applied to two case studies involving untrained sensory panelists and trained teaching quality assessors, respectively. The results of the case studies show that although reproducibility can be assumed moderate for both groups of raters, repeatability is generally higher for the group of trained raters.

This paper introduces a new Bayesian control chart to compare two processes by monitoring the ratio of percentiles of two quality characteristics that are assumed to be independent Weibull distributed random variables with the same and stable shape parameter larger than one. The chart analyses the sampling data directly, instead of transforming them in order to comply with the usual normality assumption, as many charts do. A real application in the wood industry and a wide simulation illustrate the features of the chart and its performance, depending on the number of training data, the quality of prior information, and the magnitude of the shift.

This paper proposes a new approach to assess consistency of preferences expressed as priorities over a set of alternatives. A procedure based on the analysis of variation has been adopted for testing time-homogeneity of preferences; the statistical behaviour of the testing procedure has been investigated via Monte Carlo simulation.

In the scientific literature, the debate about how define and evaluate seat comfort is still open, but three points are not in question [1]: 1. comfort is a construct of a subjective nature; 2. comfort is affected by factors of various nature (physical, physiological, psychological); 3.comfort is a reaction to the environment. The subjective nature of the comfort experience is universally recognized; any comfort analysis cannot disregard subjective methods (‘directly asking people about how comfortable they are’), which can be regarded as the most direct way to detect subjective feelings of comfort and/or discomfort. This paper focuses on the assessment of aircraft seating comfort based on subjective comfort responses collected during laboratory experiments. During each experimental session, participants were asked to express their overall seat comfort perception and to evaluate specific seat design features. Comfort responses were analyzed with the aim to relate the perceived overall seat comfort to some design features, as well as to the user anthropometrical characteristics and feelings. The adopted statistical modeling approach is based on generalized linear mixed models. Differently from the traditional strategies used for the analysis of subjective sitting comfort data (e.g. correlation analysis, non-parametric hypothesis tests), the model-based approach allows to investigate and quantify the relationship between overall seat comfort and specific seat/user characteristics. The results show that the overall comfort perception is significantly influenced by age, lumbar support, height of seat pan and reclining.

This paper focuses on the comparative assessment of comfort and discomfort (hereafter, (dis-)comfort) for air- craft seating. Subjective and objective data of seating (dis-)comfort were collected during an experiment in- volving 20 volunteers who tested 3 aircraft double-seats in upright and reclined position. In order to minimize experimental uncertainty due to well-known noise factors (i.e. patterns of discomfort during the work week and during the work day, order of evaluation, inter-individual differences), experimental trials were performed according to a crossover design. Statistical data analysis aimed mainly at investigating (dis-)comfort differences across seat conditions; gender-based differences in perceived discomfort on different body parts; effect of sitting duration on perceived discomfort on different body parts. The experimental results show that differences across seat conditions impacted differently on perceived discomfort depending on gender, body parts and sitting duration. No significant differences in perceived discomfort across gender were evident for the lightweight seat in both upright and reclined positions. On the contrary, for both baseline configurations, perceived discomfort at head and neck areas was higher for males than for females. For all seat conditions, participants experienced a significant worsening of perceived comfort over time at shoulders, back, sacrum and thighs and, in addition, at upper body area (i.e. neck, arm and forearm) and knees only for seats in reclined position.

This paper presents an interactive design method aimed at improving workplace health and safety. Human performances and anthropometric variability are carefully considered to make the workplace “robust” from a safety point of view. This topic is of increasing interest to industries that plan to make safer workplaces without renouncing to their productivity targets. A challenging issue concerns the evaluation of the effects of sources of anthropometric variability in the process by using just a small sample of real or digital humans. The adoption of a discretization technique helps to solve this problem and saving time and resources. Through real industrial case studies, the authors investigate the main ergonomic and safety issues faced during the development of both manual and human–robot hybrid workcells.

Students’ Evaluations of Teaching (SETs) are widely used as measures of teaching quality in Higher Education. A review of specialized literature evidences that researchers widely discuss whether SETs can be considered reliable measures of teaching quality evaluation. Though the controversy mainly refers to the role of students as assessors of teaching qual- ity, most of research studies on SETs focus on the design and validation of the evaluation procedure and even when the need of measuring SETs reliability is recognized, it is gener- ally indirectly assessed for the whole group of students by measuring inter-student agree- ment. In this paper the focus is on the direct assessment of the reliability of each student as a measurement instrument of teaching quality. An agreement-based approach is here adopted in order to assess student’s ability to provide consistent and stable evaluations; the sampling uncertainty is accounted for by building non-parametric bootstrap confidence intervals for the adopted agreement coefficients.

The best known classification of scales for measuring data, according to their nature, defines four types of data: nominal, ordinal, interval, and ratio. Only the last three have received official legitimization in metrology. Although a number of researchers from the academia, who proposed an alternative typology, criticized the above classification, this did not significantly affect the industrial and business statistics preferring to be in harmony with the current legal metrology. On the eve of the fourth industrial revolution, we are increasingly confronted with new types of data, which does not fall under any of the above-mentioned classifications. Hence, we often find it difficult to interpret the new data. In our lecture, we present only two kinds of them: tree structured data and preference chains. An example of the first type is the international classification of diseases (ICD), proposed by the World Health Organization (WHO), where diagnosis of a disease is associated to a specific node in the hierarchical tree. Preference/priority chains are needed when studying consumer preferences, customer requirements, in a risk management, decision-making, etc. Recently, a certain progress was achieved in determining distance metrics and analysis of variation allowing to extract useful information from such kinds of data. The lecture will report and demonstrate these achievements.

The quality of subjective evaluations provided by field experts (e.g. physicians or risk assessors) as well as by trained operators (e.g. visual inspectors) is often evaluated in terms of inter/intra rater agreement via kappa-type coefficients. Despite their popularity, these indices have long been criticized for being affected in complex ways by the presence of bias between raters and by the distributions of data across the rating categories (“prevalence”). This paper presents the results of a Monte Carlo simulation study aimed at investigating the robustness of four kappa-type indices (viz. Gwet’s AC2 and the weighted variants of Scott’s Pi, Cohen’s Kappa and Brennan-Prediger coefficients) taking into consideration the case of two series of ratings provided by the same rater (intra-rater agreement) or by two raters (inter-rater agreement). The robustness of the reviewed indices to changes in the frequency distribution of ratings across categories and in the agreement distribution between the two series of ratings has been analyzed across several simulation scenarios built by varying the sample size (i.e. number of rated items), the dimension of the rating scale, the frequency and agreement distributions between the series of ratings. Simulation results suggest that the level of agreement is sensitive to the distribution of items across the rating categories and to the dimension of rating scale but it is not influenced by the sample size. Among the reviewed indices, the Brennan–Prediger coefficient and Gwet’s AC2 are less sensitive to variation in the distribution of items across the categories for a fixed agreement distribution.

In several industries strategic and operational decisions rely on subjective evaluations provided by raters who are asked to score and/or classify group of items in terms of some technical properties (e.g. classification of faulty material by defect type) and/or perception aspects (e.g. comfort, quality, pain, pleasure, aesthetics). Because of the lack of a gold standard for classifying subjective evaluations as “true” or “false”, rater reliability is generally measured by assessing her/his precision via inter/intra-rater agreement coefficients. Agreement coefficients are useful only if their magnitude can be easily interpreted. A common practice is to apply a straightforward procedure to translate the magnitude of the adopted agreement coefficient into an extent of agreement via a benchmark scale. Many criticisms have been attached to this practice and in order to solve some of them, the adoption of a probabilistic approach to characterize the extent of agreement is recommended. In this study some probabilistic benchmarking procedures are discussed and compared via a wide Monte Carlo simulation study.

Students’ Evaluations of Teaching (SETs) are the most common way to measure teaching quality in Higher Education: they are assuming a strategic role in monitoring teaching quality, becoming helpful in taking the major formative and summative academic decisions. The majority of studies investigating SETs reliability focus on the instruments and the procedures adopted to collect students' evaluations rather than on the capability of the students as teaching quality assessors. In order to overcome this lack, a study has been carried out with the aim of measuring SETs reliability in terms of inter-student agreement and intra-student agreement. The results of our study show that the majority of students provided substantially repeatable evaluations whereas only a few students provided almost perfectly repeatable evaluations; the evaluations provided by different students generally slightly agreed, which means that the students did not share the same opinions and beliefs on teaching quality.

Ergonomic design of automotive seat is a very challenging task whose results may directly influence driver’s comfort and safety. Seat comfort can be improved by identifying car-packaging solutions that allow an optimal driver’s posture. In order to reduce the time and cost for testing, ergonomic analysis is car- ried out in virtual reality (VR) environment with digital human models (DHM) that can be used to simulate the anthropometric variability of a target population of users and thus verify the robustness of design solutions with respect to the an- thropometrical noise factor. In this paper we illustrate a case study concerning the comfort improvement of a minicar packaging set up via robust ergonomic design (RED) with digital human models. The aim is the identification of the optimum levels for the seat control parameters that minimize the driver’s comfort loss with respect to a preferred posture. The approach adopted for the analysis of data ob- tained from the virtual experiments is based on the joint generalized linear model- ing of mean and dispersion of the driver’s postural comfort loss (i.e. the ergonom- ic response of interest).

Unfortunately, the affiliation of one of the authors “Salvatore Gerbino” was incorrect in the original publication, and the correct version is updated here.

In this work, an efficient and easy statistical method to find an equivalent discrete distribution for a continuous random variable (r.v.) is proposed. The proposed method is illustrated by applying it to the treatment of the anthropometrical noise factors in the context of Robust Ergonomic Design (RED; Lanzotti 2006; Barone S. and Lanzotti A., 2007).

In this work we apply an innovative participative design approach for the quality evaluation of virtual prototypes of new industrial products (i.e. concept designs), by adopting statistical procedures and carrying out tests in an immersive VR environment. This methodology has been fully exploited through a case study concerning the choice of the optimal design for the interiors of a new regional train. Following this approach, the optimal concept design is defined at the end of a process consisting of five phases: identification of the quality elements of the concept design, classification of the quality elements, generation and quality evaluation of product concepts and, finally, definition of the optimal concept. According to the applied methodology after the identification of the customer’s needs, a structured set of quality elements has been defined and, successively, classified according to Kano’s theory. Following the approach of conjoint analysis, the design factors have been combined according to an experimental plan to form product virtual concepts. During the concepts generation phase we have explored those product architectures that integrate design characteristics correlated to the set of quality elements. The concepts have been created according to comfort, ergonomic and safety criteria. In particular we have considered the ergonomics of places and furniture dimensions, through the use of virtual manikins. The evaluation of the quality of the different concepts has been carried out in the VR laboratory (named “VRTest”) of the Competence Centre for the Qualification of Transportation Systems founded by Regione Campania according to an original statistical procedure and has involved a group of experts in train’s interiors design and a group of common users of regional trains.Copyright © 2012 by ASME

PurposeThis paper aims to provide a quantitative decision approach to the service quality management, developed on the basis of Kano's theory of attractive quality. The proposed approach aims at exploiting contacts with service made by “mystery guests” rather than traditional surveys on customer opinions.Design/methodology/approachA specific probabilistic model of the process of serving quality is the adopted basic tool. Multiple comparison tests aimed at controlling the service quality are the core of the proposed decision approach. In order to collect the needed sampling data, a few mystery guests who experience many customer‐service contacts are employed.FindingsA quantitative decision methodology which both allows one to evaluate the actual service quality level and provides, via comparison tests, a tool to highlight the weak and strong points of the service delivery process.Originality/valueThe proposed quality map is an original graphical tool, which enables one to pin‐point strengths and failings in service quality, prioritize corrective actions and recognize improvements, if any. The operative value of the whole methodology is tested through a real application to the hotel service industry.

In this work an original design strategy for product innovation is presented. This strategy is based on a continuous innovation process and takes advantages of both emotional design methodologies and participative design tools in Virtual Reality (VR). It combines techniques for user need identification and virtual reality ex-periments to simulate user-product interaction. This original combination of tech-niques allows the early evaluation of the new product concepts’ quality which is essential for the success of the innovation. To show the main phases of this strategy, three case studies are briefly introduced. In the first case study, concerning the railway coach interiors design, the impor-tance of both the identification of user needs and the evaluation session in VR is highlighted. In the second case study, concerning the railway seat design, the techniques used to generate new concepts and to choose the optimal one among them are briefly illustrated. Finally, in the third case study, concerning the innova-tion of a Postural Seat System, the continuous innovation iterative cycle, which is essential in the statistical design for innovation strategy, is described starting from the designer sketches.

In recent decades, the literature on technology management has proposed S-curves as promising tools for analyzing the life-cycle of technological innovation in order to support company strategies and policies. Nevertheless, the scant attention devoted to the analytical foundations of the S-curve model has limited its capacity to actually model the performance of technological innovation.

In this paper a non-parametric multivariate process control chart has been proposed for monitoring a manifacturing process. By simultaneously monitoring process and product variables, MAPLSS-T2 chart quickly detects and diagnoses unusual events that may occur during the process. The proposed non-parametric control chart allows to handle collinear variables,missing values, outliers and interactions between variables, without imposing any distributional assumption. Further developments of this work could be related to the construction of a chart of the Squared Prediction Error (SPE; Kourti and MacGregor, 1996) on MAPLSS model, in order to monitor any change in the covariance structure and verify that the process conditions during the monitoring stage are not dierent with respect to the time the in control MAPLSS model was developed.

The early identification of the optimal concept is a critical task of the design process in order to increase the chances of satisfying customers. The challenging aspect of the approach proposed in this work relies in the quality evaluation of virtual prototypes of new industrial products (i.e. concept designs) by adopting a statistical procedure previously applied to service industries. Following this approach, the optimal concept design is defined at the end of a process consisting of five phases: identification of the quality elements of the concept design, classification of the quality elements, generation and quality evaluation of product concepts and, finally, definition of the optimal concept. Currently, virtual reality (VR) environment offers the opportunity to evaluate the characteristics of different virtual prototypes by involving experts and/or customers, overcoming the need for several physical prototypes. On the other side, the dynamics of simulation and the stereoscopic visualization in VR environment provides a more realistic and impressive interaction with virtual prototypes than in CAD environment. The proposed methodology is fully exploited through two case studies: the choice of the optimal design for a traditional Neapolitan coffee maker, addressed by the Italian designer Riccardo Dalisi, and for a subassembly of a new minicar.

In this paper we present a probabilistic approach to the evaluation of hotel service quality. The proposed model stems from the need to provide hotel management with a diagnostic tool to evaluate and control the service quality from the customer satisfaction perspective. In order to manage a service effectively, its main quality 'elements' need to be evaluated, identifying suitable standards of comparison and verifying that the quality requirements are fulfilled. The quality elements of the hotel service can be classified into different categories. So the proposed model evaluates the contribution of each category to the hotel service quality level by identifying the logical structures which represent the relationships between them (the categories) and customer satisfaction. The practical implementation of the model does not involve the customers but only some quality experts who are specifically instructed in simulating, in a short time, several contacts between customers and service. A practical applicat...

Non-Performing Loans (NPLs) are exposures in state of insolvency, i.e. loans whose collection by banks is uncertain in terms of the amount that will be returned and the time of return. In order to measure the recovery performance of a portfolio of NPLs, our idea is to draw a curve representing the recovery rates during time, here assumed discretized, for example, in years. In this way, the user can get simultaneously information about recovery rate and time to liquidate of the portfolio. However, the computation of this curve becomes infeasible when some elements of the portfolio are right censored, i.e. the recovery rate trajectory is known only until a particular year. In this talk, a new method of estimation of the recovery rate curve in case of right censored data is presented. It is studied from a theoretical point of view in a simulation study, where it is tested and compared with other methods. It is studied from a practical point of view, where it is applied to a real financial data set about some portfolios of Italian unsecured NPLs taken in charge by a specialized operator.