Prospective validation study of an epilepsy seizure risk system for outpatient evaluation

ObjectiveWe conducted clinical testing of an automated Bayesian machine learningalgorithm (Epilepsy Seizure Assessment Tool [EpiSAT]) for outpatient seizure risk assessment using seizure counting data, and validated performance against specialized epilepsy clinicianexperts.MethodsWe conducted a prospective longitudinal study of EpiSAT performance against 24 specialized clinician expertsat three tertiary referral epilepsy centers in the United States. Accuracy, interrater reliability, and intra-rater reliability of EpiSATfor correctly identifyingchanges inseizure risk (improvements, worsening, or no change) were evaluated using 120 seizures from four synthetic seizure diaries (seizure risk known) and 120 seizures from four real seizure diaries (seizure risk unknown). The proportion of observed agreement between EpiSAT and clinicians was evaluated to assess compatibility of EpiSAT with clinical decision patterns by epilepsy experts.ResultsEpiSAT exhibited substantial observed agreement (75.4%) with clinicians for assessing seizure risk. The mean accuracy of epilepsy providers for correctly assessing seizure risk was 74.7%. EpiSAT accurately identified seizure risk in 87.5% of seizure diary entries, corresponding to a significant improvement of 17.4% (P=.002). Clinicians exhibited low-to-moderate interrater reliability for seizure risk assessment (Krippendorff's α=0.46) with good intrarater reliability across a 4- to 12-week evaluation period (Scott's π=0.89).SignificanceThese results validate the ability of EpiSAT to yield objective clinical recommendations on seizure risk which follow decision patterns similar to those from specialized epilepsy providers, but with improved accuracy and reproducibility. This algorithm may serve as a useful clinical decision support system for quantitative analysis of clinical seizure frequency in clinical epilepsy practice.