Multimodal Machine Learning Using Visual Fields and Peripapillary Circular OCT Scans in Detection of Glaucomatous Optic Neuropathy

Purpose To develop and validate a multimodal artificial intelligence algorithm, FusionNet, using the pattern deviation probability plots from visual field (VF) reports and circular peripapillary OCT scans to detect glaucomatous optic neuropathy (GON). Design Cross-sectional study. Subjects Two thousand four hundred sixty-three pairs of VF and OCT images from 1083 patients. Methods FusionNet based on bimodal input of VF and OCT paired data was developed to detect GON. Visual field data were collected using the Humphrey Field Analyzer (HFA). OCT images were collected from 3 types of devices (DRI-OCT, Cirrus OCT, and Spectralis). Two thousand four hundred sixty-three pairs of VF and OCT images were divided into 4 datasets: 1567 for training (HFA and DRI-OCT), 441 for primary validation (HFA and DRI-OCT), 255 for the internal test (HFA and Cirrus OCT), and 200 for the external test set (HFA and Spectralis). GON was defined as retinal nerve fiber layer thinning with corresponding VF defects. Main Outcome Measures Diagnostic performance of FusionNet compared with that of VFNet (with VF data as input) and OCTNet (with OCT data as input). Results FusionNet achieved an area under the receiver operating characteristic curve (AUC) of 0.950 (0.931–0.968) and outperformed VFNet (AUC, 0.868 [95% confidence interval (CI), 0.834–0.902]), OCTNet (AUC, 0.809 [95% CI, 0.768–0.850]), and 2 glaucomatologists (glaucomatologist 1: AUC, 0.882 [95% CI, 0.847–0.917]; glaucomatologist 2: AUC, 0.883 [95% CI, 0.849–0.918]) in the primary validation set. In the internal and external test sets, the performances of FusionNet were also superior to VFNet and OCTNet (FusionNet vs VFNet vs OCTNet: internal test set 0.917 vs 0.854 vs 0.811; external test set 0.873 vs 0.772 vs 0.785). No significant difference was found between the 2 glaucomatologists and FusionNet in the internal and external test sets, except for glaucomatologist 2 (AUC, 0.858 [95% CI, 0.805–0.912]) in the internal test set. Conclusions FusionNet, developed using paired VF and OCT data, demonstrated superior performance to both VFNet and OCTNet in detecting GON, suggesting that multimodal machine learning models are valuable in detecting GON.