Search

Objectives: In the first of a series of three articles by the present authors, diagnostic agreement between Traditional Chinese Medicine (TCM) practitioners was found to be low. This was the first time that TCM diagnoses had been evaluated with an open population of patients and this result is a cause of concern. In the second article, incorrect statistics were shown to have often been used to calculate chance-removed inter-rater agreement, and appropriate statistics such as Gwet's Agreement Coefficient 2 (AC2) was recommended for future studies. In this, the third article, a novel approach to recording TCM diagnostic patterns, the Traditional Chinese Medical Diagnostic Descriptor (TCMDD), is presented that allows chance-removed agreement calculation. An example of mapping TCM diagnostic patterns to the TCMDD format is given and diagnostic agreement is evaluated. Design, Settings, Subjects: The same 35 subjects used to report agreement in our first article were also diagnosed by additional practitioners using the TCMDD format during the same experimental sessions at the University of Technology, Sydney Clinic. TCM diagnoses from the first article were also mapped to the TCMDD format. Outcome measures: Linearly weighted simple agreement and the AC2 statistic were utilized and all results compared. Results: Linearly weighted simple agreement using the TCMDD and TCM mapped to TCMDD format averaged 0.80 ± 0.02 compared with 0.19 for TCM. TCMDD and TCM mapped to TCMDD chance-removed agreement, as calculated with AC2, ranged between 0.67 and 0.73 ± 0.03. Conclusions: The TCMDD allows the essence of diagnoses expressed by TCM practitioners to be appropriately compared. This was confirmed by the TCM mapped to TCMDD results. In both cases, simple agreement was significantly greater than that obtained with the TCM format. Chance-removed statistics and error estimates can be reliably calculated with the AC2 and the TCMDD in open populations.

Objectives: Fleiss' Kappa (FK) has been commonly, but incorrectly, employed as the "standard" for evaluating chance-removed inter-rater agreement with ordinal data. This practice may lead to misleading conclusions in inter-rater agreement research. An example is presented that demonstrates the conditions where FK produces inappropriate results, compared with Gwet's AC2, which is proposed as a more appropriate statistic. A novel format for recording a Chinese Medical (CM) diagnoses, called the Diagnostic System of Oriental Medicine (DSOM), was used to record and compare patient diagnostic data, which, unlike the contemporary CM diagnostic format, allows agreement by chance to be considered when evaluating patient data obtained with unrestricted diagnostic options available to diagnosticians. Design: Five CM practitioners diagnosed 42 subjects drawn from an open population. Subjects' diagnoses were recorded using the DSOM format. All the available data were initially used to evaluate agreement. Then, the subjects were sorted into three groups to demonstrate the effects of differing data marginality on the calculated chance-removed agreement. Outcome measures: Agreement between the practitioners for each subject was evaluated with linearly weighted simple agreement, FK and Gwet's AC2. Results and Conclusions: In all cases, overall agreement was much lower with FK than Gwet's AC2. Larger differences occurred when the data were more free marginal. Inter-rater agreement determined with FK statistics is unlikely to be correct unless it can be shown that the data from which agreement is determined are, in fact, fixed marginal. It follows that results obtained on agreement between practitioners with FK are probably incorrect. It is shown that inter-rater agreement evaluated with AC2 statistic is an appropriate measure when fixed marginal data are neither expected nor guaranteed. The AC2 statistic should be used as the standard statistical approach for determining agreement between practitioners.

Objectives: An acceptable level of diagnostic agreement is a prerequisite for consistent administration of treatment. It is critical for investigating effectiveness of different treatment approaches using multiple practitioners. To the best of our knowledge, no previous investigation of diagnostic consensus using open populations in Chinese medicine (CM) has been reported. Investigations restricted to individual medical conditions, such as have been usually studied, do not reveal any information as to what occurs in real world clinical settings. This knowledge gap led to the current study being conducted. Design/Location/Subjects/Interventions: Investigating diagnostic agreement specifically in Traditional Chinese Medicine (TCM) in an open population, two or three practitioners diagnosed 35 subjects at the University of Technology, Sydney (UTS), TCM clinic. The practitioners were restricted to a list of the 56 most frequently used TCM diagnoses at the UTS clinic. Up to three diagnostic patterns per subject could be selected, with nominated patterns scored between 1 and 5. Outcome measures: Agreement was determined with two criteria, both expressed as simple percentages: pattern and linearly weighted agreements. Results: The results showed that 23% of practitioners obtained pattern agreement, while 19% demonstrated weighted agreement. Conclusion: There appears to be very low diagnostic agreement between practitioners. This is an important finding. If unchallenged by further investigation, the recognition of such poor diagnostic consensus may lead to rejection of TCM theory before it has been adequately assessed. Diagnostic agreement must be improved so that future investigations into treatment effectiveness or mechanisms of action are made on a valid basis. Additionally, the current TCM diagnostic format must be altered to allow the application of chance-removed statistics or the calculation of a standard error with open populations. This article is the first of a series of three that report problems in TCM diagnostic reliability and proposes solutions to the issues outlined.