Search

In this paper, we focus on measures to evaluate discrimination of prediction models for ordinal outcomes. We review existing extensions of the dichotomous c-indexwhich is equivalent to the area under the receiver operating characteristic (ROC) curvesuggest a new measure, and study their relationships. The volume under the ROC surface (VUS) scores sets of cases including one case from each outcome category. VUS considers sets as either correctly or incorrectly ordered by the model. All other existing measures assess pairs of cases. We propose an ordinal c-index (ORC) that is set-based but, contrary to VUS, scores sets more gradually by indicating the closeness of the model-based ordering to the perfect ordering. As a result, the ORC does not decrease rapidly as the number of outcome categories increases. It turns out that the ORC can be rewritten as the average of pairwise c-indexes. Hence, the ORC has both a set- and pair-based interpretation. There are several relationships between the existing measures, leading to only two types of existing measures: a prevalence-weighted average of pairwise c-indexes and the VUS. Our suggested measure ORC positions itself in between as it is set-based but turns out to equal an unweighted average of pairwise c-indexes. The measures are demonstrated through a case study on the prediction of six-month outcome after traumatic brain injury. In conclusion, the set-based nature and graded scoring system make the ORC an attractive measure with a simple interpretation, together with its prevalence-independence that is a natural property of a discrimination measure.

Current clinical practice heavily relies on technology to support decisions. In particular, machine learning is ever more employed in decision support systems. This can be attributed to information overload, the fact that it becomes impossible for a clinician to take into account all available information. The drawback of this approach is that such decision support systems are often black boxes, yielding no insight into the reason of their decisions. In medical settings however, where trust and accountability are important issues, such systems should preferably be interpretable. In contrast, other domains almost fully rely on observation or subjective patient-reported questionnaires to quantify a medical situation. This is the case with assessment of physical capacity in patients suffering from chronic musculoskeletal conditions. With the advent of wearable technology this quantification can also be performed in an objective way to yield complementary information. This requires automatic activity recognition and assessment of the more challenging, but informative transitory activities, preferably in the home environment. With that situation in mind, the research in this PhD focuses on two themes: interpretable decision support and patient monitoring using wearables. It also connects them by developing interpretable models for activity assessment. A first set of objectives is connected to interpretable classification systems. A second set is linked to the development of activity recognition and assessment algorithms within the scope of the SPARKLE project. The first part of this work develops and improves two algorithms to extract interpretable medical scoring systems for binary classification from data. Jointly, they are called Interval Coded Scoring (ICS). The resulting models are piecewise constant over the intervals of selected variables. User interaction is possible at several points to steer the training process with expert knowledge and make a desired trade-off between performance and model simplicity. A first existing approach, lpICS, is extended to support interactions. It consists of a total variation regularized classification problem cast in the framework of Support Vector Machines, solved via Linear Programming. However, its time complexity and its uniqueness properties can be improved. Therefore, preselection is introduced as an interval-informed way to decrease the data dimensionality. Also, a second approach, enICS, is developed. It is based on elastic net cast as a dual Support Vector Machine and can be solved independently of the data dimensionality. However, it does not support variable interactions. Both algorithms are validated on several publicly available biomedical datasets, proving a performance that is similar to several standard machine learning algorithms. ICS is implemented in a Matlab toolbox offering an interactive interface for model setup, training, risk estimation and visualization. The second part designs and implements two pattern-based approaches for activity recognition using data from a single accelerometer mounted on the upper arm. They are both based on the assumption that pattern matching is more suitable for recognition of transitory activities, whereas sliding windows with statistical features are more prevalent in literature for repetitive activities of longer duration. One approach improves pattern matching recognition via the combination of dynamic time warping and common statistical features. It consists of an automatic segmentation approach followed by a multilayer random forest classification framework to reject false positives belonging to the rejection class. The other approach exploits the multilinear structure of the data. Higher Order Discriminant Analysis is applied on pattern-matched data to extract discriminatory features from the automatically segmented data. Both methods have been validated on acquired patient datasets in a protocol setting suitable for a home environment. They are proven to significantly outperform approaches based on either simple pattern matching or statistical features alone. Moreover, the second part also studies assessment of activity recognition. Two informative parameter extraction methods are developed. Furthermore, three different ways to implement an objective complement to the BASFI questionnaire, via classification with ICS or regression, are presented. To end, ICS is applied to create a scoring system for comparison with the current standard in evaluation of disease activity. status: published

Objective To determine the ability of acoustic streaming to discriminate between endometriomas and other adnexal masses. Methods We used data from 1938 patients with an adnexal mass included in Phase 2 of the International Ovarian Tumor Analysis (IOTA) study. All patients had been examined by transvaginal gray-scale and Doppler ultrasound following a standardized research protocol. Assessment of acoustic streaming was voluntary and was carried out only in lesions containing echogenic cyst fluid. Acoustic streaming was defined as movement of particles inside the cyst fluid during gray-scale and/or color Doppler examination provided that the probe had been held still for two seconds to ensure that the movement of the particles was not caused by movement of the probe or the patient. Only centers where acoustic streaming had been evaluated in > 90% of cases were included. Sensitivity, specificity, positive and negative likelihood ratios (LR+, LR-), and positive and negative predictive values (PPV and NPV) of acoustic streaming with regard to endometrioma were calculated. Results 460 (24%) masses were excluded because they were examined in centers where

OBJECTIVES: To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours. DESIGN: Observational diagnostic study using prospectively collected clinical and ultrasound data. SETTING: 24 ultrasound centres in 10 countries. PARTICIPANTS: Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients. MAIN OUTCOME MEASURES: Histological classification and surgical staging of the mass. RESULTS: The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate. CONCLUSIONS: The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology. ispartof: BMJ - British Medical Journal vol:349 issue:oct07 3 ispartof: location:England status: published