Search

Objective To develop and validate a Zika virus disease syndrome definition within the GUARDIAN (Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification) surveillance system. Introduction In 2016, the World Health Organization declared Zika virus a global public health emergency. Zika infection during pregnancy can cause microcephaly and other fetal brain defects. To facilitate clinicians’ ability to detect Zika, various syndrome definitions have been developed. Methods To create and validate a detailed syndrome definition for Zika, we utilized the literature based methodology developed and documented by GUARDIAN researchers. 1,2 The syndrome definition utilized clinical signs and symptoms that were documented in historical Zika cases. A testing sample of 1000 randomly selected emergency department cases (i.e., true negative cases) and 200 synthetically generated cases (i.e., true positive cases) was created. These 1,200 sample cases were evaluated by the GUARDIAN surveillance system to determine the probability of matching the Zika syndrome definition. A probability of ≥ 90% was utilized to designate positive Zika cases. We identified the main signs and symptoms contributing to the identification of Zika cases and conducted statistical performance metrics. Clinical review of the false positive and false negative cases along with a sample of true positive and true negative cases was conducted by a board certified emergency physician. Results The Zika syndrome definition was developed with eleven articles (six used for developing the syndrome definition, and five used for testing the definition). The sample size for these articles was between 1 and 72 positive Zika cases, with a total of 139 cases across the 11 articles. The article with the most number of Zika cases was based on pregnant women with rash. The publication timeframe for the articles was from 1962 to 2016. Some of the main signs and symptoms from the historical cases that contribute to the Zika syndrome definition are presented in Table 1. The initial results for the sample testing data showed accuracy, sensitivity, and specificity were 94.7%, 93%, and 95% respectively. There were a total of 14 false negative and 50 false positive cases. Conclusions The initial Zika syndrome definition utilized by the GUARDIAN surveillance system contains similar signs and symptoms to the current CDC case definition, but also includes additional signs and symptoms such as pruritus/itching, malaise/fatigue/generalized weakness, headache, retro-orbital pain, myalgia/muscle pain, and lymphadenopathy In addition, the GUARDIAN system provides the relative importance of identified signs and symptoms and allows for proactive surveillance of emergency department patients in real-time. Though we did not include epidemiologic risk factors, such as travel to an infected region or contact with an infected person in the syndrome definition, GUARDIAN has above 90% sensitivity and specificity. Thus, inclusion of epidemiologic risk factors would further enhance the early detection of Zika, when used with the appropriate high risk population. Table 1. Main signs and symptoms of Zika syndrome definition *Signs and symptoms included in the Centers for Disease Control and Prevention (CDC)’s Zika clinical case definition

Objective To explain the utility of using an automated syndromic surveillance program with advanced natural language processing (NLP) to improve clinical quality measures reporting for influenza immunization. Introduction Clinical quality measures (CQMs) are tools that help measure and track the quality of health care services. Measuring and reporting CQMs helps to ensure that our health care system is delivering effective, safe, efficient, patient-centered, equitable, and timely care. The CQM for influenza immunization measures the percentage of patients aged 6 months and older seen for a visit between October 1 and March 31 who received (or reports previous receipt of) an influenza immunization. Centers for Disease Control and Prevention recommends that everyone 6 months of age and older receive an influenza immunization every season, which can reduce influenza- related morbidity and mortality and hospitalizations. Methods Patients at a large academic medical center who had a visit to an affiliated outpatient clinic during June 1 - 8, 2016 were initially identified using their electronic medical record (EMR). The 2,543 patients who were selected did not have documentation of influenza immunization in a discrete field of the EMR. All free text notes for these patients between August 1, 2015 and March 31, 2016 were retrieved and analyzed using the sophisticated NLP built within Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN) – a syndromic surveillance program – to identify any mention of influenza immunization. The goal was to identify additional cases that met the CQM measure for influenza immunization and to distinguish documented exceptions. The patients with influenza immunization mentioned were further categorized by GUARDIAN NLP into Received, Recommended, Refused, Allergic, and Unavailable. If more than one category was applicable for a patient, they were independently counted in their respective categories. A descriptive analysis was conducted, along with manual review of a sample of cases per each category. Results For the 2,543 patients who did not have influenza immunization documentation in a discrete field of the EMR, a total of 78,642 free text notes were processed using GUARDIAN. Four hundred fifty three (17.8%) patients had some mention of influenza immunization within the notes, which could potentially be utilized to meet the CQM influenza immunization requirement. Twenty two percent (n=101) of patients mentioned already having received the immunization while 34.7% (n=157) patients refused it during the study time frame. There were 27 patients with the mention of influenza immunization, who could not be differentiated into a specific category. The number of patients placed into a single category of influenza immunization was 351 (77.5%), while 75 (16.6%) were classified into more than one category. See Table 1. Conclusions Using GUARDIAN’s NLP can identify additional patients who may meet the CQM measure for influenza immunization or who may be exempt. This tool can be used to improve CQM reporting and improve overall influenza immunization coverage by using it to alert providers. Next steps involve further refinement of influenza immunization categories, automating the process of using the NLP to identify and report additional cases, as well as using the NLP for other CQMs. Table 1. Categorization of influenza immunization documentation within free text notes of 453 patients using NLP

Objective To examine the baseline influenza-like illness (ILI) rates in the emergency departments (ED) of a large academic medical center (AMC), community hospital (CH), and neighboring adult and pediatric primary care clinics. Introduction The primary goal of syndromic surveillance is early recognition of disease trends, in order to identify and control infectious disease outbreaks, such as influenza. For surveillance of influenza-like illness (ILI), public health departments receive data from multiple sources with varying degrees of patient acuity, including outpatient clinics and emergency departments. However, the lack of standardization of these data sources may lead to varying baseline levels of ILI activity within a local area. Methods Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN) – a syndromic surveillance program – was used to automate ILI detection using free text chief complaint/reason for visit fields and vital signs for a large AMC - ED, CH - ED, and neighboring outpatient clinics during the summer (June 15, 2016 to August 18, 2016) in order to create a baseline. The GUARDIAN system defined ILI as fever (temperature ≥ 100°F) and cough and/or sore throat. Descriptive analysis of the observed ILI rates along with bivariate ANOVA with post hoc Bonferroni and t-test were utilized to examine the difference within the settings. Results The average ILI rate for EDs is higher than the clinics by at least 0.39%. The CH- ED had 4.23% baseline ILI rate as compared to 1.35% for AMC-ED. While the AMC – Clinics have 0.96% baseline ILI rate as compared to 0.25% for CH – Clinics. The CH- ED and AMC – Clinics represented higher variations. Based on bivariate test, CH – ED was significantly different than AMC – ED, AMC - Clinics, and CH – Clinics (F= 10.58, df = 1238, p

Processing free-text clinical information in an electronic medical record may enhance surveillance systems for early identification of influenza-like illness outbreaks. However, processing clinical text using natural language processing (NLP) poses a challenge in preserving the semantics of the original information recorded. In this study, we discuss several NLP and technical issues as well as potential solutions for implementation in syndromic surveillance systems.

This was a retrospective cross-sectional study of 100 emergency department positive influenza-like illness (ILI) patients at an academic medical center to investigate which section(s) of a patient's electronic medical record (EMR) contains the most relevant information for timely detection of ILI. The history of present illness and review of systems, followed by the nursing notes sections of the EMR were information rich and the most relevant sections for ILI surveillance for the study site.

Emergency department (ED) data are key components for syndromic surveillance systems. However, the lack of standardization for the content in chief complaint (CC) free-text fields may make it challenging to use these elements in syndromic surveillance systems. Furthermore, little is known regarding how ED data sources should be structured or combined to increase sensitivity without elevating false positives. In this study, we constructed two different models of ED data sources and evaluated the resulting ILI rates obtained in two different institutions.

Objective: To investigate the impact of excluding cases with alternative diagnoses on the sensitivity and specificity of the Centers for Disease Control and Prevention’s (CDC) influenza-like illness (ILI) case definition in detecting the 2009 H1N1 influenza, using Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification, a disease surveillance system.Design: Retrospective cross-sectional study design.Setting: Emergency department of an urban tertiary care academic medical center.Patients: 1,233 ED cases, which were tested for respiratory viruses from September 5, 2009 to May 5, 2010.Main Outcome Measure: The main outcome measures were positive predictive value, negative predictive value, sensitivity, specificity, and accuracy of the ILI case definition (both including and excluding alternative diagnoses) to detect H1N1.Results: There was a significant decrease in sensitivity (χ2 = 9.09, p 0.001) and significant improvement in specificity (χ2 = 179, p 0.001), after excluding cases with alternative diagnoses.Conclusion: When early detection of an influenza epidemic is of prime importance, pursuing alternative diagnoses as part of CDC’s ILI case definition may not be warranted for public health reporting due to the significant decrease in sensitivity, in addition to the resources required for detecting these alternative diagnoses.

Description of a statistical model to account for weather variation in influenza-like illness surveillance.

Description of the development of an auto-generated color coded map showing an academic medical centers current seven day moving average influenza-like illness rate by zip code.

OBJECTIVETo identify clinical signs and symptoms (ie, “terms”) that accurately predict laboratory-confirmed influenza cases and thereafter generate and evaluate various influenza-like illness (ILI) case definitions for detecting influenza. A secondary objective explored whether surveillance of data beyond the chief complaint improves the accuracy of predicting influenza.DESIGNRetrospective, cross-sectional study.SETTINGLarge urban academic medical center hospital.PARTICIPANTSA total of 1,581 emergency department (ED) patients who received a nasopharyngeal swab followed by rRT-PCR testing between August 30, 2009, and January 2, 2010, and between November 28, 2010, and March 26, 2011.METHODSAn electronic surveillance system (GUARDIAN) scanned the entire electronic medical record (EMR) and identified cases containing 29 clinical terms relevant to influenza. Analyses were conducted using logistic regressions, diagnostic odds ratio (DOR), sensitivity, and specificity.RESULTSThe best predictive model for identifying influenza for all ages consisted of cough (DOR=5.87), fever (DOR=4.49), rhinorrhea (DOR=1.98), and myalgias (DOR=1.44). The 3 best case definitions that included combinations of some or all of these 4 symptoms had comparable performance (ie, sensitivity=89%–92% and specificity=38%–44%). For children CONCLUSIONSA simplified case definition of fever and cough may be suitable for implementation for all ages, while inclusion of rhinorrhea may further improve influenza detection for the 0–4-year-old age group. Finally, ILI surveillance based on the entire EMR is recommended.Infect Control Hosp Epidemiol 2015;00(0): 1–8

Describes the development and validation of an Ebola virus disease syndrome definition within the GUARDIAN (Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification) surveillance system.

This session describes the technical process, challenges, and lessons learned in scaling up from a local to regional syndromic surveillance system using the MetroChicago Health Information Exchange (HIE) and GUARDIAN collaborative initiative.

A highly sensitive real-time syndrome surveillance system is critical to detect, monitor, and control infectious disease outbreaks, such as influenza. Direct comparisons of diagnostic accuracy of various surveillance systems are scarce.To statistically compare sensitivity and specificity of multiple proprietary and open source syndrome surveillance systems to detect influenza-like illness (ILI).A retrospective, cross-sectional study was conducted utilizing data from 1122 patients seen during November 1–7, 2009 in the emergency department of a single urban academic medical center. The study compared the Geographic Utilization of Artificial Intelligence in Real-time for Disease Identification and Alert Notification (GUARDIAN) system to the Complaint Coder (CoCo) of the Real-time Outbreak Detection System (RODS), the Symptom Coder (SyCo) of RODS, and to a standardized report generated via a proprietary electronic medical record (EMR) system. Sensitivity, specificity, and accuracy of each classifier's ability to identify ILI cases were calculated and compared to a manual review by a board-certified emergency physician. Chi-square and McNemar's tests were used to evaluate the statistical difference between the various surveillance systems.ResultsThe performance of GUARDIAN in detecting ILI in terms of sensitivity, specificity, and accuracy, as compared to a physician chart review, was 95.5%, 97.6%, and 97.1%, respectively. The EMR-generated reports were the next best system at identifying disease activity with a sensitivity, specificity, and accuracy of 36.7%, 99.3%, and 83.2%, respectively. RODS (CoCo and SyCo) had similar sensitivity (35.3%) but slightly different specificity (CoCo = 98.9%; SyCo = 99.3%). The GUARDIAN surveillance system with its multiple data sources performed significantly better compared to CoCo (χ2 = 130.6, p0.05), SyCo (χ2 = 125.2, p0.05), and EMR-based reports (χ2 = 121.3, p0.05). In addition, similar significant improvements in the accuracy (12%) and sensitivity (47%) were observed for GUARDIAN with only chief complaint data as compared to RODS (CoCo and SyCo) and EMR-based reports.In our study population, the GUARDIAN surveillance system, with its ability to utilize multiple data sources from patient encounters and real-time automaticity, demonstrated a more robust performance when compared to standard EMR-based reports and the RODS systems in detecting ILI. More large-scale studies are needed to validate the study findings, and to compare the performance of GUARDIAN in detecting other infectious diseases.

Objective To develop and implement a framework for special event surveillance using GUARDIAN, as well as document lessons learned post-event regarding design challenges and usability. Introduction Special event driven syndromic surveillance is often initiated by public health departments with limited time for development of an automated surveillance framework, which can result in heavy reliance on frontline care providers and potentially miss early signs of emerging trends. To address timelines and reliability issues, automated surveillance system are required. Methods The North Atlantic Treaty Organization (NATO) summit was held in Chicago, IL, May 19–21, 2012. During the NATO summit, the Chicago Department of Public Health (CDPH) was charged with collecting and analyzing syndromic surveillance data from emergency department (ED) visits that may indicate a man-made or naturally occurring infectious disease threat. Ten days prior to the NATO summit surveillance period, Rush University Medical Center (RUMC) received a guidance document from CDPH outlining the syndromes for systematic surveillance, specifically febrile rash illness, localized cutaneous lesion, acute febrile respiratory illness, gastrointestinal illness, botulism-like illness, hemorrhagic illness, along with unexplained deaths or severe illness potentially due to infectious disease and cases due to toxins or suspected poisoning. RUMC researchers collected relevant ICD-9 codes for each syndrome category. GUARDIAN (1), an automated surveillance system, was programmed to scan patient charts and match free text using National Library of Medicine free-text term to unique medical concept, which were further mapped to relevant ICD-9 codes. The baselines were developed using ED patient data from 1/1/2010 to 12/31/2011. Statistical references were established for unsmoothed, 24 hour counts (Baseline = Average; Threshold = +2 standard deviations). During the NATO surveillance timeframe (May 13–26, 2012) automated results with prior reporting period’s counts, reference statistics, and charts were electronically sent to CDPH. In addition, ED charge nurses made manual surveillance reports by telephone at least daily. Open lines of communication were maintained between RUMC and CDPH during the event to discuss potential positive cases. In addition, a post-event debriefing was conducted to document lessons learned. Results The automated GUARDIAN surveillance reports not only provided timely counts of potentially positive cases for each syndrome but also provided trend analysis with baseline measures. The GUARDIAN User Interface was used to explain what data points could trigger positive cases. The Epic system was used to review patient charts, if further explanation was necessary. The observed counts never exceeded +2 standard deviations during the NATO surveillance period for any of the syndromes. Based on the debriefing meeting between RUMC and CDPH, the top three achievements and lessons learned were as follows: Quick turnaround time (∼ 10 days) from surveillance concept development to automated implementation using GUARDIANSurveillance data was timely and reliableAdditional statistical information was beneficial to put trends in contextSystem may be too sensitive resulting in false alarms and additional investigative burden on public health departmentsNeed for development of user-interfaces with drill down capabilities to patient level dataClinicians don’t necessarily utilize exact terminology used in ICD-9 codes which could result in undetected cases. Conclusions This exercise successfully highlights rapid development and implementation of special event driven automated surveillance as well as collaborative approach between front-line entities such as emergency departments, surveillance researchers, and the department of public health. In addition, valuable lessons learned with potential solutions are documented for further refinements of such surveillance activities.

Objective To develop and test a novel syndrome definition validation approach for rarely occurring diseases. Introduction Early detection of rarely occurring but potentially harmful diseases such as bio-threat agents (e.g., anthrax), chemical agents (e.g., sarin), and naturally occurring diseases (e.g., meningitis) is critical for rapid initiation of treatment, infection control measures, and emergency response plans. To facilitate clinicians’ ability to detect these diseases, various syndrome definitions have been developed. Due to the rarity of these diseases, standard statistical methodologies for validating syndrome definitions are not applicable. Methods Syndrome definitions were developed by researchers for the Geographic Utilization of Artificial Intelligence in Real-Time for Disease Identification and Alert Notification (GUARDIAN) surveillance system (1). The main steps for validation of the syndrome definitions were: Partition of literature articles: Literature articles that described positive cases were randomly divided to generate detection (75% of articles) and testing (25% of articles) syndrome definitions.Synthetic case generation: Syndrome definitions and associated statistical measures were reverse engineered using probability of occurrence and inverse Gaussian function to generate potentially infinite positive artificial cases.Clinical filter application: To avoid clinically incompatible combinations of newly generated symptoms, rules based on clinically guided knowledge from emergency department (ED) physicians were applied. Steps 2 and 3 were repeated for both detection and testing syndrome definitions.Detection phase validation: ED negative case sample: Detection syndrome definitions were tested using a random sample of negative ED cases. Knowledge gained through false positive cases was utilized to modify the surveillance algorithms and system thresholds.10-fold cross-validation: Standard 10-fold cross-validation on detection articles of positive cases and ED negative cases was utilized to generate performance metrics. Suspected cases were reviewed by ED clinicians for threshold enhancement.Literature articles (n=1): The ability of syndrome definitions to correctly flag literature articles with n=1 case was documented.Testing phase validation: Testing sample: Synthetic positive cases generated from the testing articles along with another set of ED negative cases were evaluated by the respective syndrome definition. Suspected cases were clinically evaluated.Literature articles (n=1): Similar to detection step 4c, articles with n=1 were tested using syndrome definitions.True positive samples: When available, true positive cases from an ED were identified and sent through the GUARDIAN system.Multi-syndrome validation: A combined sample of positive cases of multiple syndromes and ED negative cases were evaluated for detection of individual syndromes among other similar syndromes. Results To demonstrate the validation approach, the anthrax syndrome definition was utilized. This syndrome definition was developed with 25 articles containing positive anthrax cases used for detection, and the remaining 11 articles used for testing. With a 10-fold cross validation of the detection phase, the initial results showed accuracy was 99.4% (false positive rate of 0.65% and false negative rate of 0.00%). The testing phase initial validation revealed 99.2% accuracy for the anthrax syndrome definition. Conclusions Syndrome specific synthetic samples that are validated through clinical filters allowed the generation of an unlimited number of positive cases. Correct identification by GUARDIAN of these cases indicates robust and reliable syndrome definitions. Utilization of these cases, in conjunction with adherence to a methodological process, was the cornerstone of the GUARDIAN syndrome definition validation approach. The validation approach was successfully demonstrated on anthrax and can be applied to other bio-threat agents, chemical agents, and naturally occurring diseases.