Search

PURPOSE: To assess changes in retinopathy of prematurity (ROP) diagnosis in single and serial retinal images. DESIGN: Cohort study. PARTICIPANTS: Cases of ROP recruited from the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) consortium evaluated by 7 graders. METHODS: Seven ophthalmologists reviewed both single and 3 consecutive serial retinal images from 15 cases with ROP, and severity was assigned as plus, preplus, or none. Imaging data were acquired during routine ROP screening from 2011 to 2015, and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1 to 9, with 9 being the most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise. MAIN OUTCOME MEASURES: Grading severity of ROP as defined by plus, preplus, or no ROP. RESULTS: Assessment of serial retinal images changed the grading severity for > 50% of the graders, although there was wide variability. Cohen’s kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in the grading of serial retinal images were noted more commonly in cases of preplus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease and overall agreement with the mode class (P = 0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient, 0.89). The more aggressive graders tended to be influenced by serial images to increase the severity of their grading. The VSS also demonstrated agreement with disease progression across serial images, which progressed to preplus and plus disease. CONCLUSIONS: Clinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment.

Purpose To evaluate the clinical usefulness of a quantitative deep learning-derived vascular severity score for retinopathy of prematurity (ROP) by assessing its correlation with clinical ROP diagnosis and by measuring clinician agreement in applying a novel scale. Design Analysis of existing database of posterior pole fundus images and corresponding ophthalmoscopic examinations using 2 methods of assigning a quantitative scale to vascular severity. Participants Images were from clinical examinations of patients in the Imaging and Informatics in ROP Consortium. Four ophthalmologists and 1 study coordinator evaluated vascular severity on a scale from 1 to 9. Methods A quantitative vascular severity score (1–9) was applied to each image using a deep learning algorithm. A database of 499 images was developed for assessment of interobserver agreement. Main Outcome Measures Distribution of deep learning-derived vascular severity scores with the clinical assessment of zone (I, II, or III), stage (0, 1, 2, or 3), and extent (6 clock hours) of stage 3 evaluated using multivariate linear regression and weighted κ values and Pearson correlation coefficients for interobserver agreement on a 1-to-9 vascular severity scale. Results For deep learning analysis, a total of 6344 clinical examinations were analyzed. A higher deep learning-derived vascular severity score was associated with more posterior disease, higher disease stage, and higher extent of stage 3 disease (P < 0.001 for all). For a given ROP stage, the vascular severity score was higher in zone I than zones II or III (P < 0.001). Multivariate regression found zone, stage, and extent all were associated independently with the severity score (P < 0.001 for all). For interobserver agreement, the mean ± standard deviation weighted κ value was 0.67 ± 0.06, and the Pearson correlation coefficient ± standard deviation was 0.88 ± 0.04 on the use of a 1-to-9 vascular severity scale. Conclusions A vascular severity scale for ROP seems feasible for clinical adoption; corresponds with zone, stage, extent of stage 3, and plus disease; and facilitates the use of objective technology such as deep learning to improve the consistency of ROP diagnosis.

Purpose Retinopathy of prematurity is a leading cause of childhood blindness worldwide, but clinical diagnosis is subjective, which leads to treatment differences. Our goal was to determine objective differences in the diagnosis of plus disease between clinicians using an automated retinopathy of prematurity (ROP) vascular severity score. Design This retrospective cohort study used data from the Imaging and Informatics in ROP Consortium, which comprises 8 tertiary care centers in North America. Fundus photographs of all infants undergoing ROP screening examinations between July 1, 2011, and December 31, 2016, were obtained. Participants Infants meeting ROP screening criteria who were diagnosed with plus disease and treatment initiated by an examining physician based on ophthalmoscopic examination results. Methods An ROP severity score (1–9) was generated for each image using a deep learning (DL) algorithm. Main Outcome Measures The mean, median, and range of ROP vascular severity scores overall and for each examiner when the diagnosis of plus disease was made. Results A total of 5255 clinical examinations in 871 babies were analyzed. Of these, 168 eyes were diagnosed with plus disease by 11 different examiners and were included in the study. The mean ± standard deviation vascular severity score for patients diagnosed with plus disease was 7.4 ± 1.9, median was 8.5 (interquartile range, 5.8–8.9), and range was 1.1 to 9.0. Within some examiners, variability in the level of vascular severity diagnosed as plus disease was present, and 1 examiner routinely diagnosed plus disease in patients with less severe disease than the other examiners (P Conclusions We observed variability both between and within examiners in the diagnosis of plus disease using DL. Prospective evaluation of clinical trial data using an objective measurement of vascular severity may help to define better the minimum necessary level of vascular severity for the diagnosis of plus disease or how other clinical features such as zone, stage, and extent of peripheral disease ought to be incorporated in treatment decisions.

PURPOSE: To evaluate a tele-education system developed to improve diagnostic competency in retinopathy of prematurity (ROP) by ophthalmologists-in-training in Mexico. DESIGN: Prospective, randomized cohort study. PARTICIPANTS: Fifty-eight ophthalmology residents and fellows from a training program in Mexico consented to participate. Twenty-nine of 58 trainees (50%) were randomized to the educational intervention (pretest, ROP tutorial, ROP educational chapters, and posttest), and 29 of 58 trainees (50%) were randomized to a control group (pretest and posttest only). METHODS: A secure web-based educational system was created using clinical cases (20 pretest, 20 posttest, and 25 training chapter-based) developed from a repository of over 2500 unique image sets of ROP. For each image set used, a reference standard ROP diagnosis was established by combining the clinical diagnosis by indirect ophthalmoscope examination and image-based diagnosis by multiple experts. Trainees were presented with image-based clinical cases of ROP during a pretest, posttest, and training chapters. MAIN OUTCOME MEASURES: The accuracy of ROP diagnosis (e.g., plus disease, zone, stage, category) was determined using sensitivity and specificity calculations from the pretest and posttest results of the educational intervention group versus control group. The unweighted kappa statistic was used to analyze the intragrader agreement for ROP diagnosis by the ophthalmologists-in-training during the pretest and posttest for both groups. RESULTS: Trainees completing the tele-education system had statistically significant improvements (P < 0.01) in the accuracy of ROP diagnosis for plus disease, zone, stage, category, and aggressive posterior ROP (AP-ROP). Compared with the control group, trainees who completed the ROP tele-education system performed better on the posttest for accurately diagnosing plus disease (67% vs. 48%; P = 0.04) and the presence of ROP (96% vs. 91%; P < 0.01). The specificity for diagnosing AP-ROP (94% vs. 78%; P < 0.01), type 2 ROP or worse (92% vs. 84%; P = 0.04), and ROP requiring treatment (89% vs. 79%; P < 0.01) was better for the trainees completing the tele-education system compared with the control group. Intragrader agreement improved for identification of plus disease, zone, stage, and category of ROP after completion of the educational intervention. CONCLUSIONS: A tele-education system for ROP education was effective in improving the diagnostic accuracy of ROP by ophthalmologists-in-training in Mexico. This system has the potential to increase competency in ROP diagnosis and management for ophthalmologists-in-training from middle-income nations.

Purpose Retinopathy of prematurity (ROP) is a vision-threatening complication occurring in pre-term neonates. The standard of care entails regular monitoring by dilated ophthalmoscopy examinations, which entail stress and potential morbidity. In this pilot study, we used plane-wave ultrasound (PWUS) to image, measure, and assess the association of blood-flow velocities in the retrobulbar vessels with ROP stages ranging from stage 0 (immature vessels without ROP) to stage 3. Methods Both eyes of 14 preterm neonates at risk for ROP were examined by 18 MHz PWUS. All but two subjects had a follow-up examination. PWUS was acquired for 1.5 seconds at 3000 compound B-scans/sec. Data were postprocessed to form color-flow images and spectrograms depicting flow velocity in the central retinal artery (CRA), central retinal vein (CRV), and the short posterior ciliary arteries (SPCA). Flow parameters derived from spectrograms were compared by ROP stage. Results ROP stage was found to correlate with flow velocities. Velocities were significantly elevated with respect to non-ROP eyes in all vessels at stage 3 and in the SPCAs at stage 2. Conclusions PWUS measurement of blood flow may provide a quantitative, clinically important, and easily tolerated means for detecting and assessing the risk of ROP in preterm neonates. We speculate that the observed increase in flow velocity results from elevated vascular endothelial growth factor (VEGF) in ROP eyes. Translational relevance PWUS offers a gentle, nonmydriatic method for monitoring neonates at risk for ROP that would complement ophthalmoscopy.

Purpose To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. Design We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. Participants Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. Methods Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. Main Outcome Measures Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. Results There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06–0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74–0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95–0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). Conclusions Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

Purpose To determine the accuracy of image-based diagnosis for stage 4 or worse retinopathy of prematurity (ROP) disease. Design Prospective cohort study. Participants We prospectively obtained data, from 8 major ROP centers, for 1220 eye examinations from 230 infants. Methods An ophthalmologist at each center provided a clinical diagnosis using indirect ophthalmoscopy. Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were then obtained, and these were independently read by 2 ROP experts using a web-based system for an image-based diagnosis. Main Outcome Measures Sensitivity and specificity of image-based diagnosis from the ROP experts were calculated using the clinical diagnosis as the reference standard. Results Of 1220 examinations, 28 (2%) had a clinical diagnosis of stage 4 or worse. Sensitivity and specificity for stage 4 or worse disease were 75% and 99% for expert 1, and 86% and 99% for expert 2. Sensitivity and specificity for the detection of stage 5 disease were 69% and 99% for both experts. Conclusions There are inconsistencies in the accuracy of image-based diagnosis of stage 4 and stage 5 ROP when compared with the clinical diagnosis.

To identify any temporal trends in the diagnosis of plus disease in retinopathy of prematurity (ROP) by experts.Reliability analysis.ROP experts were recruited in 2007 and 2016 to classify 34 wide-field fundus images of ROP as plus, pre-plus, or normal, coded as "3," "2," and "1," respectively, in the database. The main outcome was the average calculated score for each image in each cohort. Secondary outcomes included correlation on the relative ordering of the images in 2016 vs 2007, interexpert agreement, and intraexpert agreement.The average score for each image was higher for 30 of 34 (88%) images in 2016 compared with 2007, influenced by fewer images classified as normal (P.01), a similar number of pre-plus (P = .52), and more classified as plus (P.01). The mean weighted kappa values in 2006 were 0.36 (range 0.21-0.60), compared with 0.22 (range 0-0.40) in 2016. There was good correlation between rankings of disease severity between the 2 cohorts (Spearman rank correlation ρ = 0.94), indicating near-perfect agreement on relative disease severity.Despite good agreement between cohorts on relative disease severity ranking, the higher average score and classifications for each image demonstrate that experts are diagnosing pre-plus and plus disease at earlier stages of disease severity in 2016, compared with 2007. This has implications for patient care, research, and teaching, and additional studies are needed to better understand this temporal trend in image-based plus disease diagnosis.

BACKGROUND: Telemedicine is an emerging technology with potential to improve retinopathy of prematurity (ROP) care. This study evaluates parental perceptions about digital imaging and telemedicine for ROP care. METHODS: During a one-year period, one parent of each infant who underwent wide-field retinal imaging for ROP was given a questionnaire designed to evaluate parental perceptions using a 5-point Likert scale. Five items assessed perceptions toward digital retinal imaging, and 10 items assessed attitudes toward telemedicine. Construct validity of the questionnaire was examined using factor analysis. Responses were summarized using descriptive and correlational statistics. RESULTS: Forty-two parents participated. Factor analysis extracted two factors explaining 79% of the total variance in digital retinal imaging items (Cronbach’s alpha=0.843), and three factors explaining 63% of the total variance in telemedicine items (Cronbach’s alpha=0.631). Among digital imaging items, the highest mean (±SD) score was for “digital pictures of my child’s retinopathy should be included in the permanent medical record” (4.4±0.6), and the lowest was for “digital cameras and computers are reliable” (3.8±0.8). Among telemedicine items, the highest mean (±SD) score was for “technology will improve the quality of medical care for my child” (4.3±0.6), and the lowest was for “technology will make it harder for a patient and doctor to establish a good relationship” (2.6±1.1). CONCLUSIONS: Parents reported positive perceptions about telemedical ROP diagnosis, but expressed some preference for face-to-face care. Telemedicine has potential to alter the nature of the patient-physician relationship.

Purpose To compare performance of single-image vs multiple-image telemedicine examinations for retinopathy of prematurity (ROP) diagnosis. Design Prospective comparative study. Methods A total of 248 eyes from 67 consecutive infants underwent wide-angle retinal imaging by a trained neonatal nurse at 31 to 33 weeks and/or 35 to 37 weeks postmenstrual age (PMA) at a single academic institution. Data were uploaded to a web-based telemedicine system and interpreted by three masked retinal specialists. Diagnoses were provided based on single images, and subsequently on multiple images, from both eyes of each infant. Findings were compared to a reference standard of indirect ophthalmoscopy by a pediatric ophthalmologist. Primary outcome measures were recommended follow-up interval, presence of plus disease, presence of type-2 or worse ROP, and presence of visible peripheral ROP. Results Among the three graders, mean sensitivity/specificity for detection of infants requiring follow-up in less than one week were 0.85/0.93 by single-image examination and 0.91/0.88 by multiple-image examination at 35 to 37 weeks PMA. Mean sensitivity/specificity for detection of infants with type-2 or worse ROP were 0.82/0.95 by single-image examination and 1.00/0.91 by multiple-image examination at 35 to 37 weeks PMA. Mean sensitivity/specificity for detection of plus disease were 1.00/0.86 by single-image examination and 1.00/0.87 by multiple-image examination at 35 to 37 weeks PMA. There were no statistically-significant intragrader differences between accuracy of single-image and multiple-image telemedicine examinations for detection of plus disease. Conclusions Single-image and multiple-image telemedicine examinations perform comparably for determination of recommended follow-up interval and detection of plus disease. This may have implications for development of screening protocols, particularly in areas with limited access to ophthalmic care.

To evaluate the intraphysician agreement between ophthalmoscopic examination and image-based telemedical interpretation for retinopathy of prematurity (ROP) diagnosis, when performed by the same expert physician grader.Prospective, nonrandomized, comparative study.Sixty-seven consecutive premature infants who underwent ROP examination at a major university medical center whose parents consented for participation.Infants underwent standard dilated ophthalmoscopy by one of two pediatric ophthalmologists, followed by retinal imaging with a commercially available wide-angle fundus camera by a trained neonatal nurse. Study examinations were performed at 31 to 33 weeks postmenstrual age (PMA) and/or 35 to 37 weeks PMA. Images were uploaded to a Web-based telemedicine system developed by the authors. After a 4- to 12-month period, telemedical interpretations were performed in which each physician graded images from infants upon whom he had initially performed ophthalmoscopic examinations. Diagnoses were classified using an ordinal scale: no ROP, mild ROP, type 2 prethreshold ROP, and treatment-requiring ROP.Absolute intraphysician agreement and kappa statistic between ophthalmoscopic examination and telemedical interpretation were calculated by eye. All intraphysician discrepancies were reviewed, and underlying causes were classified by eye as no ROP identified by ophthalmoscopic examination, no ROP identified by telemedical interpretation, discrepancy about presence of zone 1 ROP, discrepancy about presence of plus disease, or other discrepancy in classification of ROP stage.Absolute intraphysician agreement between ophthalmoscopic examination and telemedical interpretation was 86.3%. The kappa statistic for intraphysician agreement between examinations ranged from 0.657 (substantial agreement) for diagnosis of treatment-requiring ROP to 0.854 (near-perfect agreement) for diagnosis of mild or worse ROP. Among 206 eye examinations (103 infant examinations), there were 28 (13.6%) intraphysician discrepancies in diagnosis, 8 of which resulted from uncertainty about presence of zone 1 disease and 4 from uncertainty about presence of plus disease.Intraphysician agreement between ophthalmoscopic examination and telemedical interpretation for ROP was very high. Neither examination modality appeared to have a systematic tendency to overdiagnose or underdiagnose ROP. Diagnosis of zone 1 disease and plus disease were major sources of clinically significant discrepancies.

Objective To evaluate the intraphysician agreement between ophthalmoscopic examination and image-based telemedical interpretation for retinopathy of prematurity (ROP) diagnosis, when performed by the same expert physician grader. Design Prospective, nonrandomized, comparative study. Participants Sixty-seven consecutive premature infants who underwent ROP examination at a major university medical center whose parents consented for participation. Methods Infants underwent standard dilated ophthalmoscopy by one of two pediatric ophthalmologists, followed by retinal imaging with a commercially available wide-angle fundus camera by a trained neonatal nurse. Study examinations were performed at 31 to 33 weeks postmenstrual age (PMA) and/or 35 to 37 weeks PMA. Images were uploaded to a Web-based telemedicine system developed by the authors. After a 4- to 12-month period, telemedical interpretations were performed in which each physician graded images from infants upon whom he had initially performed ophthalmoscopic examinations. Diagnoses were classified using an ordinal scale: no ROP, mild ROP, type 2 prethreshold ROP, and treatment-requiring ROP. Main Outcome Measures Absolute intraphysician agreement and κ statistic between ophthalmoscopic examination and telemedical interpretation were calculated by eye. All intraphysician discrepancies were reviewed, and underlying causes were classified by eye as no ROP identified by ophthalmoscopic examination, no ROP identified by telemedical interpretation, discrepancy about presence of zone 1 ROP, discrepancy about presence of plus disease, or other discrepancy in classification of ROP stage. Results Absolute intraphysician agreement between ophthalmoscopic examination and telemedical interpretation was 86.3%. The κ statistic for intraphysician agreement between examinations ranged from 0.657 (substantial agreement) for diagnosis of treatment-requiring ROP to 0.854 (near-perfect agreement) for diagnosis of mild or worse ROP. Among 206 eye examinations (103 infant examinations), there were 28 (13.6%) intraphysician discrepancies in diagnosis, 8 of which resulted from uncertainty about presence of zone 1 disease and 4 from uncertainty about presence of plus disease. Conclusions Intraphysician agreement between ophthalmoscopic examination and telemedical interpretation for ROP was very high. Neither examination modality appeared to have a systematic tendency to overdiagnose or underdiagnose ROP. Diagnosis of zone 1 disease and plus disease were major sources of clinically significant discrepancies.

Objective To prospectively measure accuracy, reliability, and image quality of telemedical retinopathy of prematurity (ROP) diagnosis. Methods Two-hundred forty-eight eyes from 67 consecutive infants underwent wide-angle retinal imaging by a trained neonatal nurse at 31 to 33 weeks’ and/or 35 to 37 weeks' postmenstrual age (PMA) using a standard protocol. Data were uploaded to a Web-based telemedicine system and interpreted by 3 expert retinal specialist graders who provided a diagnosis (no ROP, mild ROP, type 2 prethreshold ROP, treatment-requiring ROP) and an evaluation of image quality for each eye. Findings were compared with a reference standard of indirect ophthalmoscopy by an experienced pediatric ophthalmologist. Results At 35 to 37 weeks' PMA, sensitivity and specificity for diagnosis of mild or worse ROP were 0.908 and 1.000 for grader A, 0.971 and 1.000 for grader B, and 0.908 and 0.977 for grader C. Sensitivity and specificity for diagnosis of type 2 prethreshold or worse ROP were 1.000 and 0.943 for grader A, 1.000 and 0.930 for grader B, and 1.000 and 0.851 for grader C. At 35 to 37 weeks' PMA, weighted κ for intergrader reliability was 0.791 to 0.889, and κ for intragrader reliability for detection of type 2 prethreshold or worse ROP was 0.769 to 1.000. Image technical quality was rated as “adequate” or “possibly adequate” for diagnosis in 93.3% to 100% of eyes. Conclusion A telemedicine system using nurse-captured retinal images has the potential to improve existing shortcomings of ROP management, particularly at later PMAs.