Intraclass Correlation Coefficient ( ICC): A Framework for Monitoring and Assessing Performance of Trained Sensory Panels and Panelists.

Sensory measurement underlies sensory science. Sensory analysis and decision-making heavily depend on the quality of sensory data, which is determined by the performance of trained sensory panels and panelists. Various methods have been proposed for monitoring and assessing the performance. A weakness of the currently used methods is lack of a unified framework for various criteria and a variety of experiments with different types of data. This paper proposes to use accuracy, validity and reliability as general terminologies to describe sensory measurement and to apply the intraclass correlation coefficient ( ICC) as a framework for monitoring and assessing performance. ICC can measure both similarity among panelists and sensitivity of panels and panelists. Hence, ICC can handle the problems of both reliability and validity. ICC can be obtained from different types of data for diverse experiments. This paper provides the equations and R and S- Plus functions for estimations of ICCs from continuous data (ratings), multivariate continuous data, ordinal data, ranking data, binary-choice data, multiple-choice data and forced-choice data. Confidence intervals, variances of the estimators, comparison with a fixed value and difference and similarity tests for multiple ICCs are also provided. The relationship between Cronbach's coefficient alpha and ICC is discussed. Practical Applications Intraclass correlation coefficient ( ICC) may play a framework role in monitoring and assessing performance of trained sensory panels and panelists. It can be used as an index of the quality of sensory data. The larger the ICC or Cronbach's coefficient alpha value, the better the performance of panels and panelists. If and only if an ICC is significantly larger than a specified value, can the quality of the data for that attribute be regarded as acceptable. A statistical test that fails to show that an ICC or Cronbach's coefficient alpha is significantly larger than a specified lowest limit, e.g., 0.1, suggests that the products are undiscriminating, or the sensory data, at least for that attribute, might not be valid and reliable. We should use that data with caution. [ABSTRACT FROM AUTHOR]