Search

Background A high proportion of health care services are persistently utilized by a small subpopulation of patients. To improve clinical outcomes while reducing costs and utilization, population health management programs often provide targeted interventions to patients who may become persistent high users/utilizers (PHUs). Enhanced prediction and management of PHUs can improve health care system efficiencies and improve the overall quality of patient care. Objective The aim of this study was to detect key classes of diseases and medications among the study population and to assess the predictive value of these classes in identifying PHUs. Methods This study was a retrospective analysis of insurance claims data of patients from the Johns Hopkins Health Care system. We defined a PHU as a patient incurring health care costs in the top 20% of all patients’ costs for 4 consecutive 6-month periods. We used 2013 claims data to predict PHU status in 2014-2015. We applied latent class analysis (LCA), an unsupervised clustering approach, to identify patient subgroups with similar diagnostic and medication patterns to differentiate variations in health care utilization across PHUs. Logistic regression models were then built to predict PHUs in the full population and in select subpopulations. Predictors included LCA membership probabilities, demographic covariates, and health utilization covariates. Predictive powers of the regression models were assessed and compared using standard metrics. Results We identified 164,221 patients with continuous enrollment between 2013 and 2015. The mean study population age was 19.7 years, 55.9% were women, 3.3% had ≥1 hospitalization, and 19.1% had 10+ outpatient visits in 2013. A total of 8359 (5.09%) patients were identified as PHUs in both 2014 and 2015. The LCA performed optimally when assigning patients to four probability disease/medication classes. Given the feedback provided by clinical experts, we further divided the population into four diagnostic groups for sensitivity analysis: acute upper respiratory infection (URI) (n=53,232; 4.6% PHUs), mental health (n=34,456; 12.8% PHUs), otitis media (n=24,992; 4.5% PHUs), and musculoskeletal (n=24,799; 15.5% PHUs). For the regression models predicting PHUs in the full population, the F1-score classification metric was lower using a parsimonious model that included LCA categories (F1=38.62%) compared to that of a complex risk stratification model with a full set of predictors (F1=48.20%). However, the LCA-enabled simple models were comparable to the complex model when predicting PHUs in the mental health and musculoskeletal subpopulations (F1-scores of 48.69% and 48.15%, respectively). F1-scores were lower than that of the complex model when the LCA-enabled models were limited to the otitis media and acute URI subpopulations (45.77% and 43.05%, respectively). Conclusions Our study illustrates the value of LCA in identifying subgroups of patients with similar patterns of diagnoses and medications. Our results show that LCA-derived classes can simplify predictive models of PHUs without compromising predictive accuracy. Future studies should investigate the value of LCA-derived classes for predicting PHUs in other health care settings.

Feral swine populations in the United States (US) are capable of carrying diseases that threaten the health of the domestic swine industry. Performing routine, near-real time monitoring for an unusual rise in feral swine slaughter condemnation will increase situational awareness and early detection of potential animal health issues, trends, and emerging diseases. In preparation to add feral swine to APHIS weekly monitoring, a descriptive analysis of feral swine slaughter and condemnations was conducted to understand the extent of commercial feral swine slaughter in the US at federally inspected slaughter establishments and to determine which condemnation reasons should be included. There were 17 establishments that slaughtered 242,198 feral swine across seven states from 2017 to 2019. For all 17 establishments combined, feral swine accounted for 63% of slaughtered animals. A total of 23 types of condemnation reasons were noted: Abscess/Pyemia, Arthritis, Contamination, Deads, Emaciation, General Miscellaneous, Icterus, Injuries, Metritis, Miscellaneous Degenerative & Dropsical Condition, Miscellaneous Inflammatory Diseases, Miscellaneous Parasitic Conditions, Moribund, Nephritis/Pyelitis, Non-ambulatory, Pericarditis, Pneumonia, Residue, Sarcoma, Septicemia, Sexual Odor, Toxemia, and Uremia. Exploratory analysis was conducted to determine which condemnation reasons should be included for weekly monitoring. For most condemn reasons, weeks of unusually high condemnations were noted. For example, a period of high pneumonia condemnations occurred from December 2, 2018 through February 3, 2019 with a spike on January 6, 2019 and a spike in dead swine occurred on November 3, 2019. The seasonal impacts on limited quality food resources, seasonal variation in the pathogen(s) causing pneumonia, and harsher weather are suspected to have an impact on the higher condemnation rates of pneumonia and dead swine during the winter months. Based on condemnation frequencies and the likelihood of enabling situational awareness and early detection of feral swine health emerging diseases, the following were selected for weekly monitoring: abscess/pyemia, contamination/peritonitis, deads, emaciation, injuries, miscellaneous parasitic conditions, moribund, pneumonia and septicemia. Detection of notable increases in condemnation reasons strongly suggestive of foreign animal or emerging diseases should contribute valuable evidence toward the overall disease discovery process when the anomalies are both confirmed with follow up investigation and combined with other types of surveillance.

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) is a secure web-based tool that enables health care practitioners to monitor health indicators of public health importance for detection and tracking of disease outbreaks, consequences of severe weather, and other events of concern. The ESSENCE concept began in an internally funded project at the Johns Hopkins University Applied Physics Laboratory (JHU/APL), advanced with funding from the State of Maryland, and broadened in 1999 as a collaboration with the Walter Reed Army Institute for Research. Versions of the system have been further developed by JHU/APL in multiple military and civilian programs for timely detection and tracking of health threats. Features of ESSENCE include spatial and temporal statistical alerting, custom querying, user-defined alert notifications, geographical mapping, remote data capture, and event communications. These features allow ESSENCE users to gather and organize the resulting wealth of information into a coherent view of population health status and communicate findings among users. The resulting broad utility, applicability and adaptability of this system led to adoption of ESSENCE by the Centers for Disease Control and Prevention (CDC), numerous state and local health departments, and the Department of Defense (DOD) both nationally and globally. With emerging high-consequence communicable diseases and other health conditions, the continued user-requirements-driven enhancements of ESSENCE demonstrate an adaptable disease surveillance capability focused on the everyday needs of public health. The challenge of a live system for widely distributed users with multiple different data sources and high throughput requirements has driven an novel, evolving architecture design.

Display Omitted Used data from12 national notifiable diseases and simulated alerting signals.Compared 4 detection methods by background alert rate, sensitivity, and timeliness.Long-term trend adjustment improved detection performance.The modified Historical Limits Method (HLM) outperformed the traditional HLM.The Farrington-like method provided the best detection performance overall. To compare the performance of the standard Historical Limits Method (HLM), with a modified HLM (MHLM), the Farrington-like Method (FLM), and the Serfling-like Method (SLM) in detecting simulated outbreak signals. We used weekly time series data from 12 infectious diseases from the U.S. Centers for Disease Control and Preventions National Notifiable Diseases Surveillance System (NNDSS). Data from 2006 to 2010 were used as baseline and from 2011 to 2014 were used to test the four detection methods. MHLM outperformed HLM in terms of background alert rate, sensitivity, and alerting delay. On average, SLM and FLM had higher sensitivity than MHLM. Among the four methods, the FLM had the highest sensitivity and lowest background alert rate and alerting delay. Revising or replacing the standard HLM may improve the performance of aberration detection for NNDSS standard weekly reports.

Objective In a partnership between the Public Health Division of the Oregon Health Authority (OHA) and the Johns Hopkins Applied Physics Laboratory (APL), our objective was develop an analytic fusion tool using streaming data and report-based evidence to improve the targeting and timing of evidence-based interventions in the ongoing opioid overdose epidemic. The tool is intended to enable practical situational awareness in the ESSENCE biosurveillance system to target response programs at the county and state levels. Threats to be monitored include emerging events and gradual trends of overdoses in three categories: all prescription and illicit opioids, heroin, and especially high-mortality synthetic drugs such as fentanyl and its analogues. Traditional sources included emergency department (ED) visits and emergency management services (EMS) call records. Novel sources included poison center calls, death records, and report-based information such as bad batch warnings on social media. Using available data and requirements analyses thus far, we applied and compared Bayesian networks, decision trees, and other machine learning approaches to derive robust tools to reveal emerging overdose threats and identify at-risk subpopulations. Introduction Unlike other health threats of recent concern for which widespread mortality was hypothetical, the high fatality burden of opioid overdose crisis is present, steadily growing, and affecting young and old, rural and urban, military and civilian subpopulations. While the background of many public health monitors is mainly infectious disease surveillance, these epidemiologists seek to collaborate with behavioral health and injury prevention programs and with law enforcement and emergency medical services to combat the opioid crisis. Recent efforts have produced key terms and phrases in available data sources and numerous user-friendly dashboards allowing inspection of hundreds of plots. The current effort seeks to distill and present combined fusion alerts of greatest concern from numerous stratified data outputs. Near-term plans are to implement best-performing fusion methods as an ESSENCE module for the benefit of OHA staff and other user groups. Methods By analyzing historical OHA data, we formed features to monitor in each data source to adapt diagnosis codes and text strings suggested by CDC’s injury prevention division, published EMS criteria [Reference 1], and generic product codes from CDC toxicologists, with guidance from OHA Emergency Services Director David Lehrfeld and from Oregon Poison Center Director Sandy Giffen. These features included general and specific opioid abuse indicators such as daily counts of records labelled with the “poisoning” subcategory and containing “fentanyl” or other keywords in the free-text. Matrices of corresponding time series were formed for each of 36 counties and the entire state as inputs to region-specific fusion algorithms. To obtain truth data for detection, the OHA staff provided guidance and design help to generate plausible overdose threat scenarios that were quantified as realistic data distributions of monitored features accounting for time delays and historical distributions of counts in each data source. We sampled these distributions to create 1000 target sets for detection based on the event duration and affected counties for each event scenario. We used these target datasets to compare the detection performance of fusion detection algorithms. Tested algorithms included Bayesian Networks formed with the R package gRain, and also random forest, logistic regression, and support vector machine models implemented with the Python scikit-learn package using default settings. The first 800 days of the data were used for model training, and the last 400 days for testing. Model results were evaluated with the metrics: Sensitivity = (number of target event days signaled) / (all event days) and Positive predictive value (PPV) = (number of target event days signaled) / (all days signaled). These metrics were combined with specificity regarded as the expected fusion alert rate calculated from the historical dataset with no simulated cases injected. Results The left half of Figure 1 illustrates a threat scenario along Oregon’s I5 corridor in which string of fentanyl overdoses with a few fatalities affects the monitored data streams in three counties over a seven-day period. The right half of the figure charts the performance metrics for random forest and Bayesian network machine learning methods applied to both training and test datasets assuming total case counts of 50, 20, and 10 overdoses. Sensitivity values were encouraging, especially for the Bayesian networks and even for the 10-case scenario. Computed PPV levels suggested a manageable public health investigation burden. Conclusions The detection results were promising for a threat scenario of particular concern to OHA based on a data scenario deemed plausible and realistic based on historical data. Trust and acceptance from public health surveillance of outputs from supervised machine learning methods beyond traditional statistical methods will require user experience and similar evaluation with additional threat scenarios and authentic event data. Credible truth data can be generated for testing and evaluation of analytic fusion methods with the advantages of several years of historical data from multiple sources and the expertise of experienced monitors. The collaborative generation process may be standardized and extended to other threat types and data environments. Next steps include the addition to the analytic fusion capability of report-based data that can influence data interpretation, including mainstream and social media reports, events in neighboring regions, and law enforcement data. References 1. Rhode Island Enhanced State Opioid Overdose Surveillance (ESOOS) Case Definition for Emergency Medical Services (EMS), http://www.health.ri.gov/publications/guidelines/ESOOSCaseDefinitionForEMS.pdf, last accessed: Sept. 9, 2018.

Objective This abstract describes an ISDS initiative to bring together public health practitioners and analytics solution developers from both academia and industry to define a roadmap for the development of algorithms, tools, and datasets to improve the capabilities of current text processing algorithms to identify negated terms (i.e. negation detection). Introduction Despite considerable effort since the turn of the century to develop Natural Language Processing (NLP) methods and tools for detecting negated terms in chief complaints, few standardised methods have emerged. Those methods that have emerged (e.g. the NegEx algorithm [1]) are confined to local implementations with customised solutions. Important reasons for this lack of progress include (a) limited shareable datasets for developing and testing methods (b) jurisdictional data silos, and (c) the gap between resource-constrained public health practitioners and technical solution developers, typically university researchers and industry developers. To address these three problems ISDS, funded by a grant from the Defense Threat Reduction Agency, organized a consultancy meeting at the University of Utah designed to bring together (a) representatives from public health departments, (b) university researchers focused on the development of computational methods for public health surveillance, (c) members of public health oriented non-governmental organisations, and (d) industry representatives, with the goal of developing a roadmap for the development of validated, standardised and portable resources (methods and data sets) for negation detection in clinical text used for public health surveillance. Methods Free-text chief complaints remain a vital resource for syndromic surveillance. However, the widespread adoption of Electronic Health Records (and federal Meaningful Use requirements) has brought changes to the syndromic surveillance practice ecosystem. These changes have included the widespread use of EHR-generated chief complaint “pick lists” (i.e. pre-defined chief complaints that are selected by the user, rather than text strings input by the user at a keyboard), triage note templated text, and triage note free-text (typically much more comprehensive than traditional chief complaints). A key requirement for a negation detection algorithm is the ability to successfully and accurately process these new and challenging data streams. Preparations for the consultancy included an email thread and a shared website for published articles and data samples leading to a structured pre-consultancy call designed to inform participants regarding the purpose of the consultancy and to align expectations. Then, health department users were requested to provide data samples exemplifying negation issues in the classification process. Presenting developers were asked to explain their underlying ideas, details of method implementation, size and composition of corpora used for evaluation, and classification performance results. Results The consultancy was held on January 19th & 20th 2017 at the University of Utah’s Department of Biomedical Informatics, and consisted of 25 participants. Participants were drawn from various different sectors, with representation from ISDS (2), the Defense Threat Reduction Agency (1), universities and research institutes (10), public health departments (5), the Department of Veterans Affairs (4), non-profit organisations (2), and technology firms (1). Participants were drawn from a variety of different professional backgrounds, including research scientists, software developers, public health executives, epidemiologists, and analysts. Day 1 of the consultancy was devoted to providing an overview of NLP and current trends in negation detection, including a detailed description of widely used algorithms and tools for the negation detection task. Key questions included: Should our focus be chief complaints only, or should we widen our scope to emergency department triage notes?, How many other NLP tasks (e.g. reliable concept recognition) is it necessary to address on the road to improved negation detection? With this background established, Day 2 centered on presentations from five different United States local and regional health departments (King County WA, Boston MA, North Carolina, Georgia, and Tennessee) on the various approaches to text processing and negation detection utilized across several jurisdictions. Several key areas of focus emerged as a result of the consultancy discussion. First, there is a clear need for a large, easily accessible corpus of free-text chief complaints that can form a standardised testbed for negation detection algorithm development and evaluation. Annotated data, in this context, consists of chief complaints annotated for concepts (e.g. vomiting, pain in chest) and the negation status of those concepts. It is important that the annotation include both annotated clinical concepts and negation status to allow for the uniform evaluation and performance comparison of candidate negation detection algorithms. Further, the annotated corpus should consist of several thousand (as opposed to several hundred) distinct and representative chief complaints in order to compare algorithms against a sufficient variety and volume of negation patterns. Conclusions The consultancy was stimulating and eye-opening for both public health practitioner and technology developer attendees. Developers unfamiliar with the everyday health-monitoring context gained an appreciation of the difficulty of deriving useful indicators from chief complaints. Also highlighted was the challenge of processing triage notes and other free-text fields that are often unused for surveillance purposes. Practitioners were provided with concise explanations and evaluations of recent NLP approaches applicable to negation processing. The event afforded direct dialogue important for communication across professional cultures. Please note that a journal paper describing the consultancy has recently been published in the Online Journal of Public Health Informatics [2]. References [1] Chapman W, Bridewell W, Henbury P, Cooper G, Buchanan B. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001, 34(5):301-310. [2] Conway M, Mowery D, Ising A, Velupillai S, Doan S, Gunn J, Donovan M, Wiedeman C, Ballester L, Soetebier K, Tong C, Burkom H. Cross-disciplinary consultance to breige public health technical needs and analytic developers: negation detection use case. Online Journal of Public Health Informatics. 2018, 10(2)

Introduction The Risk Identification Unit (RIU) of the US Dept. of Agriculture’s Center for Epidemiology and Animal Health (CEAH) conducts weekly surveillance of national livestock health data and routine coordination with agricultural stakeholders. As part of an initiative to increase the number of species, health issues, and data sources monitored, CEAH epidemiologists are building a surveillance system based on weekly syndromic counts of laboratory test orders in consultation with Colorado State University laboratorians and statistical analysts from the Johns Hopkins University Applied Physics Laboratory. Initial efforts focused on 12 years of equine test records from three state labs. Trial syndrome groups were formed based on RIU experience and published literature. Exploratory analysis, stakeholder input, and laboratory workflow details were needed to modify these groups and filter the corresponding data to eliminate alerting bias. Customized statistical detection methods were sought for effective monitoring based on specialized laboratory information characteristics and on the likely presentation and animal health significance of diseases associated with each syndrome. Methods Data transformation and syndrome formation focused on test battery type, test name, submitter source organization, and specimen type. We analyzed time series of weekly counts of tests included in candidate syndrome groups and conducted an iterative process of data analysis and veterinary consultation for syndrome refinement and record filters. This process produced a rule set in which records were directly classified into syndromes using only test name when possible, and otherwise, the specimen type or related body system was used with test name to determine the syndrome. Test orders associated with government regulatory programs, veterinary teaching hospital testing protocols, or research projects, rather than clinical concerns, were excluded. We constructed a testbed for sets of 1000 statistical trials and applied a stochastic injection process assuming lognormally distributed incubation periods to choose an alerting algorithm with the syndrome-required sensitivity and an alert rate within the specified acceptable range for each resulting syndrome. Alerting performance of the EARS C3 algorithm traditionally used by CEAH was compared to modified C2, CuSUM, and EWMA methods, with and without outlier removal and adjustments for the total weekly number of non-mandatory tests. Results The equine syndrome groups adopted for monitoring were abortion/reproductive, diarrhea/GI, necropsy, neurological, respiratory, systemic fungal, and tickborne. Data scales, seasonality, and variance differed widely among the weekly time series. Removal of mandatory and regulatory tests reduced weekly observed counts significantly—by >80% for diarrhea/GI syndrome. The RIU group studied outcomes associated with each syndrome and called for detection of single-week signals for most syndromes with expected false-alert intervals >8 and

Background The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) is a secure web-based tool that enables health care practitioners to monitor health indicators of public health importance for the detection and tracking of disease outbreaks, consequences of severe weather, and other events of concern. The ESSENCE concept began in an internally funded project at the Johns Hopkins University Applied Physics Laboratory, advanced with funding from the State of Maryland, and broadened in 1999 as a collaboration with the Walter Reed Army Institute for Research. Versions of the system have been further developed by Johns Hopkins University Applied Physics Laboratory in multiple military and civilian programs for the timely detection and tracking of health threats. Objective This study aims to describe the components and development of a biosurveillance system increasingly coordinating all-hazards health surveillance and infectious disease monitoring among large and small health departments, to list the key features and lessons learned in the growth of this system, and to describe the range of initiatives and accomplishments of local epidemiologists using it. Methods The features of ESSENCE include spatial and temporal statistical alerting, custom querying, user-defined alert notifications, geographical mapping, remote data capture, and event communications. To expedite visualization, configurable and interactive modes of data stratification and filtering, graphical and tabular customization, user preference management, and sharing features allow users to query data and view geographic representations, time series and data details pages, and reports. These features allow ESSENCE users to gather and organize the resulting wealth of information into a coherent view of population health status and communicate findings among users. Results The resulting broad utility, applicability, and adaptability of this system led to the adoption of ESSENCE by the Centers for Disease Control and Prevention, numerous state and local health departments, and the Department of Defense, both nationally and globally. The open-source version of Suite for Automated Global Electronic bioSurveillance is available for global, resource-limited settings. Resourceful users of the US National Syndromic Surveillance Program ESSENCE have applied it to the surveillance of infectious diseases, severe weather and natural disaster events, mass gatherings, chronic diseases and mental health, and injury and substance abuse. Conclusions With emerging high-consequence communicable diseases and other health conditions, the continued user requirement–driven enhancements of ESSENCE demonstrate an adaptable disease surveillance capability focused on the everyday needs of public health. The challenge of a live system for widely distributed users with multiple different data sources and high throughput requirements has driven a novel, evolving architecture design.

Driven by the growing importance of situational awareness of bioterrorism threats, the Republic of Korea (ROK) and the United States have constructed a joint military capability, called the Biosurveillance Portal (BSP), to enhance biosecurity. As one component of the BSP, we developed the Military Active Real-time Syndromic Surveillance (MARSS) system to detect and track natural and deliberate disease outbreaks. This article describes the ROK military health data infrastructure and explains how syndromic data are derived and made available to epidemiologists. Queries corresponding to 8 syndromes, based on published clinical effects of weaponized pathogens, were used to classify military hospital patient records to form aggregated daily syndromic counts. A set of ICD-10 codes for each syndrome was defined through literature review and expert panel discussion. A study set of time series of national daily counts for each syndrome was extracted from the Defense Medical Statistical Information System between January 1, 2011, and May 31, 2014. A stratified, adjusted cumulative summation algorithm was implemented for each syndrome group to signal alerts prompting investigation. The algorithm was developed by calculating sensitivity to sets of 1,000 artificial outbreak signals randomly injected in the dataset, with each signal injected in a separate trial. Queries and visualizations were adapted from the Suite for Automated Global bioSurveillance. Findings indicated that early warning of outbreaks affecting fewer than 50 patients will require analysis at subnational levels, especially for common syndrome groups. Developing MARSS to improve sensitivity will require modification of underlying syndromic diagnosis codes, engineering to coordinate alerts among subdivisions, and enhanced algorithms. The bioterrorist threat in the Korean peninsula mandates these efforts.

Display Omitted We used syndromic data from 62 VHA hospitals and simulated alerting signals.We compared 7 detection methods on sensitivity and timeliness of alerting.Weekday/weekend stratification improved detection performance.Total visit adjustment increased sensitivity and timeliness.A Poisson regression provided the best sensitivity for high-count data. National syndromic surveillance systems require optimal anomaly detection methods. For method performance comparison, we injected multi-day signals stochastically drawn from lognormal distributions into time series of aggregated daily visit counts from the U.S. Centers for Disease Control and Prevention's BioSense syndromic surveillance system. The time series corresponded to three different syndrome groups: rash, upper respiratory infection, and gastrointestinal illness. We included a sample of facilities with data reported every day and with median daily syndromic counts ?1 over the entire study period. We compared anomaly detection methods of five control chart adaptations, a linear regression model and a Poisson regression model. We assessed sensitivity and timeliness of these methods for detection of multi-day signals. At a daily background alert rate of 1% and 2%, the sensitivities and timeliness ranged from 24 to 77% and 3.3 to 6.1days, respectively. The overall sensitivity and timeliness increased substantially after stratification by weekday versus weekend and holiday. Adjusting the baseline syndromic count by the total number of facility visits gave consistently improved sensitivity and timeliness without stratification, but it provided better performance when combined with stratification. The daily syndrome/total-visit proportion method did not improve the performance. In general, alerting based on linear regression outperformed control chart based methods. A Poisson regression model obtained the best sensitivity in the series with high-count data.

Objective To obtain feedback and seek future directions for an ISDS initiative to establish and update research questions in Informatics, Analytics, Communications, and Systems Research with the greatest perceived impact for improving surveillance practice. Introduction Over the past fifteen years, syndromic surveillance (SyS) has evolved from a set of ad hoc methods used mostly in post-disaster settings, then expanded with broad support and development because of bioterrorism concerns, and subsequently evolved to a mature technology that runs continuously to detect and monitor a wide range of health issues. Continued enhancements needed to meet the challenges of novel health threats with increasingly complex information sources will require technical advances focused on day-to-day public health needs. Since its formation in 2005, the International Society for Disease Surveillance (ISDS) has sought to clarify and coordinate global priorities in surveillance research. As part of a practitioner-driven initiative to identify current research priorities in SyS, ISDS polled its members about capabilities needed by SyS practitioners that could be improved as a result of research efforts. A taskforce of the ISDS Research Committee, consisting of national and global subject matter experts (SMEs) in SyS and ISDS professional staff, carried out the project. This panel will discuss the results and the preferred means to determine and communicate priorities in the future.

Objective Our objective was to compare the effectiveness of applying the historical limits method (HLM) to poison center (PC) call volumes with vs without stratifying by exposure type. Introduction The Centers for Disease Control and Prevention (CDC) uses the National Poison Data System (NPDS) to conduct surveillance of calls to United States PCs. PCs provide triage and treatment advice for hazardous exposures through a free national hotline. Information on demographics, health effects, implicated substance(s), medical outcome of the patient, and other variables are collected. CDC uses automated algorithms to identify anomalies in both pure call volume and specific clinical effect volume, and to identify calls reporting exposure to high priority agents. Pure and clinical effect volume anomalies are identified when an hourly call count exceeds a threshold based on historical data using HLM. 1 Clinical toxicologists and epidemiologists at the American Association of Poison Control Centers and CDC apply standardized criteria to determine if the anomaly identifies a potential incident of public health significance (IPHS) and to notify the respective health departments and local PCs as needed. Discussions with NPDS users and analysis of IPHS showed that alerting based on pure call volume yielded excessive false positives. A study using a 5-year NPDS call dataset assessed the positive predictive value (PPV) of the call volume-based approach. This study showed that less than 4% of anomalies were IPHS. 2 A low PPV can cause unnecessary waste of staff time and resources analyzing false positive anomalies. As an alternative to pure call volume-based detection where all calls to each PC are aggregated for anomaly detection, we considered separating calls by toxicologically-relevant exposure categories for more targeted anomaly detection. We hypothesized that this stratified approach would reduce the number of false positives. Methods We derived our exposure categories based on the criteria that the categories must: 1) relate to hazardous exposures of public health importance, 2) reflect categories based on clinical effects and treatment modalities, 3) avoid high priority exposures that may be triggered by single calls, 4) be compatible with exposure substance identification codes currently used by PCs and NPDS, and 5) include enough calls for meaningful tracking. We queried all calls reporting exposures to the proposed categories between January 1, 2009 and July 31, 2015 for ten PCs. We applied the HLM method after stratifying by exposure category and tabulated the number of alerts triggered for each category during the study period. We then applied the HLM method for the ten PCs on all combined exposure calls to represent the traditional non-stratified approach. We compared the combined alert burden generated by stratifying by exposure category with the alert burden for the non-stratified approach for varying time windows (1-, 2-, 4-, 8- and 24-hours). We conducted analysis in R. Results We derived a total of 20 exposure categories, including chemicals (n=4), drugs of abuse (n=6), pesticides (n=3), gas/fume/vapors (n=2), contaminated food/water (n=1), and others (n=4). Call counts during 2015 for these categories ranged from approximately 5,000 to 90,000. Table 1 shows the total number of alerts triggered for each method by time windows. There was a marked reduction of alert burden when first stratifying by exposure category for time windows shorter than eight hours compared to the alert burden for the non-stratified approach. Conclusions Stratification of call volume by exposure category and time window suggests potential improvement over traditional non- stratified approach by having a lower alert burden. Further work should focus on refining the exposure categories, refining the time window for surveillance, and assessing other detection performance metrics, such as sensitivity. Table 1: Alert burden comparison for the non-stratified vs stratified approach

Objective Our objective was to determine if the detection performance of current surveillance algorithms to detect call clusters is improved by stratifying by exposure category. Introduction The Centers for Disease Control and Prevention (CDC) uses the National Poison Data System (NPDS) to conduct surveillance of calls to United States poison centers (PCs) to identify clusters of reports of hazardous exposures and illnesses. NPDS stores basic information from PC calls including call type (information request only or call reporting a possible chemical exposure), exposure agent, demographics, clinical, and other variables. CDC looks for anomalies in PC data by using automated algorithms to analyze call and clinical effect volume, and by identifying calls reporting exposures to pre-specified high priority agents. Algorithms analyzing call and clinical effect volume identify anomalies when the number of calls exceeds a threshold using the historical limits method (HLM). Clinical toxicologists and epidemiologists at the American Association of Poison Control Centers and CDC apply standardized criteria to determine if the anomaly is a potential incident of public health significance (IPHS) and then notify the respective health departments and PCs as needed. Discussions with surveillance system users and analysis of past IPHS determined that call volume-based surveillance results in a high proportion of false positive anomalies. A study assessing the positive predictive value (PPV) of this approach determined that fewer than four percent of anomalies over a five-year period were IPHS. 1 A low PPV can cause an unnecessary waste of staff time and resources. We hypothesized that first stratifying call volume by exposure category would reduce the number of false positives. With the help of medical toxicologists, we created 20 toxicologically-relevant exposure categories to test this hypothesis. Methods To compare cluster detection performance between the two approaches, we used a historical testbed of hourly exposure call counts with and without initial stratification by exposure category from 10 selected PCs from Jan 1, 2006 - Jul 31, 2015. We ran the HLM for both non-stratified and stratified testbeds to estimate the monthly number of anomalies triggered (i.e., alert burden). Our target signals to assess detection performance consisted of call samples from three large public health events: the 2009 Salmonella food poisoning event from contaminated peanut butter, the 2012 Hurricane Sandy- associated carbon monoxide poisonings in New Jersey, and the 2014 Elk River contaminated water spill in West Virginia (WV). For each event, we chose 30 random calls one thousand times to obtain 1000 random sets of inject clusters. Each inject cluster was iteratively added into the testbed with and without initial stratification by exposure category. We then applied the HLM for each iteration to see if the algorithm identified the inject cluster. The sensitivity for each approach for each PC was calculated as the proportion of iterations where the algorithm identified the inject cluster. We reported median sensitivities from the ten PCs for each of the time windows of 1, 2, 4, 8, and 24 hours. Results Figure 1 summarizes results for the WV event with markers showing anomaly burden (x-axis) and sensitivity (y-axis) using the stratified ( Δ ) and the non-stratified (o) approach by different time windows (hrs). The results from the other two events are not shown but established similar patterns. Anomaly burden is shown as the estimated monthly anomaly count for each approach. For example, markers linked by the arrow show that with a 4-hour time window, the stratified approach achieves nearly perfect sensitivity with ~10 anomalies as the monthly anomaly burden while sensitivity of the non-stratified approach is below 20% with ~40 monthly anomalies. The stratified approach gave improved overall sensitivity across all time windows, and reduced anomaly burden for 1-, 2-, and 4-hour time windows. Conclusions We found a consistent detection advantage (higher sensitivity and lower anomaly burden) for the stratified vs traditional non- stratified approach for 1-, 2-, and 4-hour time windows. Further research should focus on refining the stratified approach and the specific surveillance parameters (such as time windows) that increase algorithm performance. Figure 1: Detection performance comparison: stratified vs non-stratified approach; 2014 Elk River contaminated water spill in West Virginia scenario

Syndromic surveillance has expanded since 2001 in both scope and geographic reach and has benefited from research studies adapted from numerous disciplines. The practice of syndromic surveillance continues to evolve rapidly. The International Society for Disease Surveillance solicited input from its global surveillance network on key research questions, with the goal of improving syndromic surveillance practice. A workgroup of syndromic surveillance subject matter experts was convened from February to June 2016 to review and categorize the proposed topics. The workgroup identified 12 topic areas in 4 syndromic surveillance categories: informatics, analytics, systems research, and communications. This article details the context of each topic and its implications for public health. This research agenda can help catalyze the research that public health practitioners identified as most important.

This paper continues an initiative conducted by the International Society for Disease Surveillance with funding from the Defense Threat Reduction Agency to connect near-term analytical needs of public health practice with technical expertise from the global research community. The goal is to enhance investigation capabilities of day-to-day population health monitors. A prior paper described the formation of consultancies for requirements analysis and dialogue regarding costs and benefits of sustainable analytic tools. Each funded consultancy targets a use case of near-term concern to practitioners. The consultancy featured here focused on improving predictions of asthma exacerbation risk in demographic and geographic subdivisions of the city of Boston, Massachusetts, USA based on the combination of known risk factors for which evidence is routinely available. A cross-disciplinary group of 28 stakeholders attended the consultancy on March 30-31, 2016 at the Boston Public Health Commission (BPHC). Known asthma exacerbation risk factors are upper respiratory virus transmission, particularly in school-age children, harsh or extreme weather conditions, and poor air quality. Meteorological subject matter experts described availability and usage of data sources representing these risk factors. Modelers presented multiple analytic approaches including mechanistic models, machine learning approaches, simulation techniques, and hybrids. Health department staff and local partners discussed surveillance operations, constraints, and operational system requirements. Attendees valued the direct exchange of information among public health practitioners, system designers, and modelers. Discussion finalized design of an 8-year de-identified dataset of Boston ED patient records for modeling partners who sign a standard data use agreement.

Analysis of available, potentially large data streams could have value in assisting risk identification efforts. We explored data from the American Society for the Prevention of Cruelty to Animals (ASPCA) Animal Poison Control Center hotline to identify weekly increases in: 1) call volume, 2) calls regarding certain intoxication products, 3) calls regarding certain species, and 4) calls with clinical signs associated with specific syndromes. Seasonal trends were observed in calls regarding pesticide intoxications. Human-centric product intoxication trends were observed in swine. These data support the value of passive data streams in risk identification, biosurveillance, and animal health monitoring.

Epidemiological modeling for infectious disease is important for disease management and its routine implementation needs to be facilitated through better description of models in an operational context. A standardized model characterization process that allows selection or making manual comparisons of available models and their results is currently lacking. A key need is a universal framework to facilitate model description and understanding of its features. Los Alamos National Laboratory (LANL) has developed a comprehensive framework that can be used to characterize an infectious disease model in an operational context. The framework was developed through a consensus among a panel of subject matter experts. In this paper, we describe the framework, its application to model characterization, and the development of the Biosurveillance Analytics Resource Directory (BARD; http://brd.bsvgateway.org/brd/), to facilitate the rapid selection of operational models for specific infectious/communicable diseases. We offer this framework and associated database to stakeholders of the infectious disease modeling field as a tool for standardizing model description and facilitating the use of epidemiological models.

Biosurveillance systems designed and deployed in the United States and abroad to allow public health authorities to monitor the health of their communities have significant design limitations despite their wide usage. One limitation is the lack of algorithmic solutions to combine local data sources for regional situation awareness. The objective of the current study is to develop and demonstrate the value of automated information fusion methods applied to the distributed neighboring public health sites. A prototype system consisting of distributed Bayesian models was designed to enable informed regional and local cognitive decision support response. The Intelligent Decision Support Network (IDSN) is composed of Bayesian Information Fusion Models (BIFMs) that target a particular syndrome or disease type. Using local data from county health departments in Northern Virginia for the time period between August 2005 and May 2007, we estimated the probability of a gastrointestinal (GI) outbreak in two ways: First, based on data from the local hospitals only; and second, based on the relative probability of outbreak by combining local hospital data and probabilities of GI events from the neighboring counties' BIFMs. Preliminary findings showed that the network of distributed models detected events that would be undetected without multi-jurisdictional data.

Background: Automated surveillance systems require statistical methods to recognize increases in visit counts that might indicate an outbreak. In prior work we presented methods to enhance the sensitivity of C2, a commonly used time series method. In this study, we compared the enhanced C2 method with five regression models. Methods: We used emergency department chief complaint data from US CDC BioSense surveillance system, aggregated by city (total of 206 hospitals, 16 cities) during 5/2008-4/2009. Data for six syndromes (asthma, gastrointestinal, nausea and vomiting, rash, respiratory, and influenza-like illness) was used and was stratified by mean count (1-19, 20-49, >=50 per day) into 14 syndrome-count categories. We compared the sensitivity for detecting single-day artificially-added increases in syndrome counts. Four modifications of the C2 time series method, and five regression models (two linear and three Poisson), were tested. A constant alert rate of 1% was used for all methods. Results: Among the regression models tested, we found that a Poisson model controlling for the logarithm of total visits (i.e., visits both meeting and not meeting a syndrome definition), day of week, and 14-day time period was best. Among 14 syndrome-count categories, time series and regression methods produced approximately the same sensitivity (

Algorithms for identifying public health threats or disease outbreaks are vulnerable to false alarms arising from sudden shifts in health-care utilization or data participation. This paper describes a method of reducing false alerts in automated public health surveillance algorithms, and in particular, automated syndromic surveillance algorithms, that rely on health-care utilization data. The technique is based on monitoring syndromic counts with reference to a suitable background, or reference, series of counts. The suitability of the background time series in decreasing the false-alarm rate will be shown to be related mathematically to the so-called mutual information that exists between the random variables representing the syndromic and background time series of counts. The method can be understood as a noise cancellation filter technique in which one noisy (reference) channel is used to cancel the background noise of the monitored (measured) channel. The issues discussed here may also be relevant to the appropriate use of rates in epidemiology and biostatistics. Copyright © 2011 John Wiley & Sons, Ltd.

Objective: We evaluated emergency department (ED) data, emergency medical services (EMS) data, and public utilities data for describing an outbreak of carbon monoxide (CO) poisoning following a windstorm.Methods: Syndromic ED data were matched against previously collected chart abstraction data. We ran detection algorithms on selected time series derived from all 3 data sources to identify health events associated with the CO poisoning outbreak. We used spatial and spatiotemporal scan statistics to identify geographic areas that were most heavily affected by the CO poisoning event.Results: Of the 241 CO cases confirmed by chart review, 190 (78.8%) were identified in the syndromic surveillance data as exact matches. Records from the ED and EMS data detected an increase in CO-consistent syndromes after the storm. The ED data identified significant clusters of CO-consistent syndromes, including zip codes that had widespread power outages. Weak temporal gastrointestinal (GI) signals, possibly resulting from ingestion of food spoiled by lack of refrigeration, were detected in the ED data but not in the EMS data. Spatial clustering of GI-based groupings in the ED data was not detected.Conclusions: Data from this evaluation support the value of ED data for surveillance after natural disasters. Enhanced EMS data may be useful for monitoring a CO poisoning event, if these data are available to the health department promptly.(Disaster Med Public Health Preparedness. 2011;5:37-45)

Dr Fricker’s paper addresses the practical, multidisciplinary problem of biosurveillance. This field has seen numerous books and journal articles in the past 10 years and is still evolving as a science. Based on his own reading, research, and contacts with those in public health practice (PHP), the author presents his view of the status and needed direction of this field. The remarks below are intended to augment this view from another perspective. Perspective: Following a background in mathematics and 20 years of developing and evaluating detection systems in non-health-related disciplines, I have worked on biosurveillance systems since 2000 while acquiring knowledge in epidemiology and biostatistics related to public health. The recent years have been devoted to the design, development, and evaluation of ESSENCE biosurveillance systems and to consulting for the U.S. Centers for Disease Control and Prevention, now my half-time employer. As a member since 2005 of the Board of Directors of the International Society for Disease Surveillance (ISDS), my role has been liaison to the ISDS Research Committee, which has an international roster of over 130 public health practitioners and academic, government, and industrial researchers. Dr Fricker’s remarks about the need for cross-disciplinary collaboration resonated with efforts over the past 2 years between the ISDS Research and PHP committees to promote research efforts with near-term PHP utility. These efforts have spawned several webinar and conference sessions, most notably a widely advertised and attended event entitled ‘Technical Challenges from the Public Health Practice Community’, whose recording and related products are freely available [1]. The remarks below are heavily influenced by the observations of PHP members from these events and by project-related contacts.

This paper describes the problem of public health monitoring for waterborne disease outbreaks using disparate evidence from health surveillance data streams and environmental sensors. We present a combined monitoring approach along with examples from a recent project at the Johns Hopkins University Applied Physics Laboratory in collaboration with the U.S. Environmental Protection Agency. The project objective was to build a module for the Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE) to include water quality data with health indicator data for the early detection of waterborne disease outbreaks. The basic question in the fused surveillance application is 'What is the likelihood of the public health threat of interest given recent information from available sources of evidence?' For a scientific perspective, we formulate this question in terms of the estimation of positive predictive value customary in classical epidemiology, and we present a solution framework using Bayesian Networks (BN). An overview of the BN approach presents advantages, disadvantages, and required adaptations needed for a fused surveillance capability that is scalable and robust relative to the practical data environment. In the BN project, we built a top-level health/water-quality fusion BN informed by separate waterborne-disease-related networks for the detection of water contamination and human health effects. Elements of the art of developing networks appropriate to this environment are discussed with examples. Results of applying these networks to a simulated contamination scenario are presented.

Introduction: We document a funded effort to bridge the gap between constrained scientific challenges of public health surveillance and methodologies from academia and industry. Component tasks are the collection of epidemiologists’ use case problems, multidisciplinary consultancies to refine them, and dissemination of problem requirements and shareable datasets. We describe an initial use case and consultancy as a concrete example and challenge to developers. Materials and Methods: Supported by the Defense Threat Reduction Agency Biosurveillance Ecosystem project, the International Society for Disease Surveillance formed an advisory group to select tractable use case problems and convene inter-disciplinary consultancies to translate analytic needs into well-defined problems and to promote development of applicable solution methods. The initial consultancy’s focus was a problem originated by the North Carolina Department of Health and its NC DETECT surveillance system: Derive a method for detection of patient record clusters worthy of follow-up based on free-text chief complaints and without syndromic classification. Results: Direct communication between public health problem owners and analytic developers was informative to both groups and constructive for the solution development process. The consultancy achieved refinement of the asyndromic detection challenge and of solution requirements. Participants summarized and evaluated solution approaches and discussed dissemination and collaboration strategies.Conclusions: A solution meeting the specification of the use case described above could improve human monitoring efficiency with expedited warning of events requiring follow-up, including otherwise overlooked events with no syndromic indicators. This approach can remove obstacles to collaboration with efficient, minimal data-sharing and without costly overhead.

We provided emergency department providers with a real-time laboratory-based influenza surveillance tool, and evaluated the utility and acceptability of the surveillance information using provider surveys. The majority of emergency department providers found the surveillance data useful and indicated the additional information impacted their clinical decision making regarding influenza testing and treatment.

This panel will expand the development process and use case collection of the ISDS Technical Conventions Committee, whose purpose is to bridge the gap between the analytic needs of public health practitioners and the expertise of the research community for the enhancement of disease surveillance.Panellists will present use cases of current and ongoing concern at local, state, and national levels. The audience will be prompted with questions on the relevance, aspects of common utility, technical feasibility, and data availability issues related to each use case, and derived or novel ideas will be solicited.

As a part of the Korea-US joint Biosurveillance Portal project, we have developed the military syndromic surveillance system with electronic medical records from military hospitals for early identification of bio-terror related events among Armed Forces in Korea. Respiratory, Gastrointestinal, Botulism, Dermatologic, Neurologic, Hemorrhagic and Fever syndromes were defined by different ICD-10 codes and their alert thresholds were developed based on the characteristics of time series derived using daily counts of ICD-10 codes for each syndrome.

The session will explore past collaborations between the statistician panelists and public health departments to highlight approaches that have and have not been effective and to recommend effective, sustainable relationship strategies for mutual advancement of practical disease surveillance and relevant academic research. Panelists will describe experiences working with health departments, including actual applications as examples. Issues discussed will include requirements analysis, scoping technical problems for health department utility, adaptation of traditional statistical methods, and management of changing data environments. Panelists will derive advice for public health practitioners seeking help in forming relationships, framing problems, communicating results, and seeking funding.

This presentation compares surveillance algorithms used in the National Poison Data System to identify incidents of public health significance with recently expanded filtering capabilities and with methods beyond the NPDS generalized historical limits model. Collected data series from 55 poison centers over 7 years include hourly counts of general call volumes and of substance-specific (e.g. CO exposure) calls. By applying current, modified, and novel methods to known and simulated clusters among these data, the authors will present the most efficient algorithms for identifying incidents of public health significance.

In order to provide real-time access to influenza test results, we created a laboratory-based surveillance system which automatically uploaded influenza test results from a rapid PCR-based influenza test, Xpert Flu, and the associated testing times and locations. On-site, type-specific results were available to physicians and uploaded for public health awareness within 100 minutes of patient nasopharyngeal swab. Expansion of this real-time capability to sentinel facilities could improve both local and national surveillance and response, reducing the need for syndromic influenza surveillance.

Since 2012, the ISDS Research Committee (RC) has been using an updated literature search process to identify literature related to disease surveillance. These articles are primarily in the medical literature, but one-third are published in other disciplines. ISDS members may not have access to those articles published outside the health sciences, making the ISDS article repository a valuable resource for those conducting work in surveillance.

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics, or ESSENCE II, uses syndromic and nontraditional health information to provide very early warning of abnormal health conditions in the National Capital Area (NCA). ESSENCE II is being developed for the Department of Defense Global Emerging Infections System and is the only known system to combine both military and civilian health care information for daily outbreak surveillance. The National Capital Area has a complicated, multijurisdictional structure that makes data sharing and integrated regional surveillance challenging. However, the strong military presence in all jurisdictions facilitates the collection of health care information across the region. ESSENCE II integrates clinical and nonclinical human behavior indicators as a means of identifying the abnormality as close to the time of onset of symptoms as possible. Clinical data sets include emergency room syndromes, private practice billing codes grouped into syndromes, and veterinary syndromes. Nonclinical data include absenteeism, nurse hotline calls, prescription medications, and over-the-counter self-medications. Correctly using information marked by varying degrees of uncertainty is one of the more challenging as pects of this program. The data (without personal identifiers) are captured in an electronic format, encrypted, archived, and processed at a secure facility. Aggregated information is then provided to users on secure Web sites. When completed, the system will provide automated capture, archiving, processing, and notification of abnormalities to epidemiologists and analysts. Outbreak detection methods currently include temporal and spatial variations of odds ratios, autoregressive modeling, cumulative summation, matched filter, and scan statistics. Integration of nonuniform data is needed to increase sensitivity and thus enable the earliest notification possible. The performance of various detection techniques was compared using results obtained from the ESSENCE II system.

Researchers working on the Department of Defense Global Emerging Infections System (DoD-GEIS) pilot system, the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE), have applied scan statistics for early outbreak detection using both traditional and nontraditional data sources. These sources include medical data indexed by International Classification of Disease, 9th Revision (ICD-9) diagnosis codes, as well as less-specific, but potentially timelier, indicators such as records of over-the-counter remedy sales and of school absenteeism. Early efforts employed the Kulldorff scan statistic as implemented in the SaTScan software of the National Cancer Institute. A key obstacle to this application is that the input data streams are typically based on time-varying factors, such as consumer behavior, rather than simply on the populations of the component subregions. We have used both modeling and recent historical data distributions to obtain background spatial distributions. Data analyses have provided guidance on how to condition and model input data to avoid excessive clustering. We have used this methodology in combining data sources for both retrospective studies of known outbreaks and surveillance of high-profile events of concern to local public health authorities. We have integrated the scan statistic capability into a Microsoft Access-based system in which we may include or exclude data sources, vary time windows separately for different data sources, censor data from subsets of individual providers or subregions, adjust the background computation method, and run retrospective or simulated studies.

The on-site beta test of system upgrade to the Early Notification of community-based Epidemics (ESSENCE) allowed user feedback to the development team to aid in the evaluation of an analytic combination of weighted clinical and syndromic evidence, known as fusion, and direct the completion of the upgrade prior to launch. The analytic fusion capability was improved particularly in the area of severity-related indicator data, improving algorithms, and user interfaces.

To understand and set priorities for research and best practices in public health monitoring, a group of ISDS members has developed an online survey to gather input from the ISDS community on the current landscape and prioritization of select technical issues in the field of biosurveillance. Through analysis, the Survey will identify respondents' perceptions of opportunities in the area of analytical methodologies. Results, both frequencies of responses and prioritization, for all respondents and by strata depending on demographic variables, will be available during the presentation.

While other surveillance systems may only use death and admissions as severity indicators, these serious events may overshadow the more subtle severity signals based on appointment type, disposition from an outpatient setting, and whether that patient had to return for care if they their condition has not improved. This abstract discusses how these additional data fields were utilized in a fusion model to improve the Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE).

Objective To describe a collaboration with the Johns Hopkins Applied Physics Laboratory (JHU APL), the North Carolina Division of Public Health (NC DPH), and the UNC Department of Emergency Medicine Carolina Center for Health Informatics (CCHI) to implement time-of-arrival analysis (TOA) for hospital emergency department (ED) data in NC DETECT to identify clusters of ED visits for which there is no pre-defined syndrome or sub-syndrome. Introduction TOA identifies clusters of patients arriving to a hospital ED within a short temporal interval. Past implementations have been restricted to records of patients with a specific type of complaint. The Florida Department of Health uses TOA at the county level for multiple sub-syndromes (1). In 2011, NC DPH, CCHI and CDC collaborated to enhance and evaluate this capability for NC DETECT, using NC DETECT data in BioSense 1.0 (2). After this successful evaluation based on exposure complaints, discussions were held to determine the best approach to implement this new algorithm into the production environment for NC DETECT. NC DPH was particularly interested in determining if TOA could be used for identifying clusters of ED visits not filtered by any syndrome or sub-syndrome. In other words, can TOA detect a cluster of ED visits relating to a public health event, even if symptoms from that event are not characterized by a predefined syndrome grouping? Syndromes are continuously added to NC DETECT but a syndrome cannot be created for every potential event of public health concern. This TOA approach is the first attempt to address this issue in NC DETECT. The initial goal is to identify clusters of related ED visits whose keywords, signs and/or symptoms are NOT all expressed by a traditional syndrome, e.g. rash, gastrointestinal, and flu-like illnesses. The goal instead is to identify clusters resulting from specific events or exposures regardless of how patients present – event concepts that are too numerous to pre-classify. Methods In late 2011, NC DPH and JHU APL signed a Software License Agreement and soon thereafter CCHI received the TOA software package. In May 2012, the TOA controller was adapted and set up to run against ED visit data for all NC DETECT hospitals. The TOA looks for clusters in all ED visits by hospital based solely on arrival time in both 30-minute and 60-minute intervals. There is no pre-classification of the chief complaints or triage notes into syndromes. TOA alerts are viewable on the NC DETECT Web application and, as of August 2012, users are able to document any actions taken on these alerts. Results From April 15, 2012 to July 31, 2012, TOA generated 173 alerts across all 115 hospitals reporting to NC DETECT. The TOA identified a group of scabies-related ED visits that was not captured in another syndrome. The TOA also identified clusters identified by hospitals as disaster-related which included misspellings that had not been previously identified, e.g. “diaster” and “disater,” as well as events involving out-of-town groups that will not be identified spatially (Table 1). This preliminary review of TOA alerts did not evaluate TOA for false negatives. Conclusions Our preliminary review of TOA shows that this algorithm approach can be helpful for identifying clusters of ED visits that are not captured by existing syndromes and can be used to identify hospital coding schemes for disaster events. The TOA will continue to be monitored in our production environment and evaluated for additional effectiveness. We will also explore tools that will display counts of terms within a TOA alert to assist in signal investigation.

Objective The objective was to use syndromic surveillance data from the North Carolina Disease Event Tracking and Epidemiologic Collection Tool NCDETECT and from BioSense to quantify the burden on North Carolina (NC) emergency departments of oral health-related visits more appropriate for care in a dental office (ED). Calculations were sought in terms of the Medicaid-covered visit rate relative to the Medicaid-eligible population by age group and by county. Introduction Concern over oral health-related ED visits stems from the increasing number of unemployed and uninsured, the cost burden of these visits, and the unavailability of indicated dental care in EDs [1]. Of particular interest to NC state public health planners are Medicaid-covered visits. Syndromic data in biosurveillance systems offer a means to quantify these visits overall and by county and age group. Methods Using BioSense data received from NCDETECT, 60.8 million records from 12.9 million ED visits were collected, covering all NC visits for state fiscal years (SFY) 2008–2010. Roughly 4% of visits were dropped because of patient residence zip codes missing or outside NC. A careful multi-step procedure involving both dentist consultants and data analysis was used to derive classification criteria for visits whose main reason was a nontraumatic oral health problem [2]. This procedure yielded 243,970 visits by ∼174,600 patients based on hospital-specific patient identifiers. Nontraumatic oral health-related visits were collected in a study set with added fields for method of payment, patient residence county, and age group. Based on previous studies, consultant preferences, and NC Medicaid eligibility guidelines, selected age groups were 0–14, 15–19, 20–29, 30–49, 50+ years. Stratified counts of Medicaid-eligibles were obtained from the NC Dental Director by study year. Using these tables and the ED visit study set, rates of nontraumatic oral health-related Medicaid visits per 10,000 eligibles were tabulated by county and age group for each study year. Demographics of multiple-visit patients were also studied. Results Rates of ED oral health-related visits were substantially higher for young adults than for other age groups. From statewide rates in Table 1, this age factor was consistent across study years. County-level rates showed the same age pattern to varying degrees. Detailed analysis showed problem areas, with rates in 21 of 100 counties exceeding 60 per 10,000 eligibles for the 20–29 year age group. Plots and tables complemented understanding of the ED oral health visit burden by age and county. The state total ED burden for oral health problems was ∼2% (0.2% – 9.7% by county). Conclusions Judicious use of syndromic data with external information, such as the detailed Medicaid denominators and the Method of Payment codes for each visit above, can give quantified estimates for policy-related public health issues. In the current study, the derived oral health visit rates gave numerical detail to concerns about the use of NC EDs for nontraumatic oral health problems by low-income persons affected by the economic recession. Results also show rate variation by county and can be combined with access-to-care data to inform planning of effective local measures to improve access to dental services and thus reduce the ED visit burden.

Objective To improve the method of automated retrieval of surveillance-related literature from a wide range of indexed repositories. Introduction The ISDS Research Committee (RC) is an interdisciplinary group of researchers interested in various topics related to disease surveillance. The RC hosts a literature review process with a permanent repository of relevant journal articles and bimonthly calls that provide a forum for discussion and author engagement. The calls have led to workgroups and society-wide events, boosted interest in the ISDS Conference, and fostered networking among participants. Since 2007, the RC has identified and classified published articles using an automated search method with the aim of progressing ISDS’s mission of advancing the science and practice of disease surveillance by fostering collaboration and increasing awareness of innovations in the field of surveillance. The RC literature review efforts have provided an opportunity for interprofessional collaboration and have resulted in a repository of over 1,000 articles, but feedback from ISDS members indicated relevant articles were not captured by the existing methodology. The method of automated literature retrieval was thus refined to improve efficiency and inclusiveness of stakeholder interests. Methods The earlier literature review method was implemented from March 2007 to March 2012. PubCrawler [1] (articles indexed in Medline) and Google Scholar [2] search results were sent to the RC via automated e-mail. To refine this method, the RC developed search strings in PubMed [3], Embase [4], and Scopus [5], consisting of over 100 terms suggested by members. After evaluating these methods, we found that the Scopus search is the most comprehensive and improved the cross-disciplinary scope. Scopus results allowed filtering of 50–100 titles and abstracts in fewer than 30 minutes each week for the identification of relevant articles (Figure). Journal titles were categorized to assess the increased range of fields covered; categories include epidemiology, agriculture, economics, and medicine (51 categories total). Results Since implementing the new method, potentially relevant articles identified per month increased from an average of 19 (SD: 13; n= 31) to 159 (SD: 63; n= 3). Both methods identified articles in the health sciences, but the new search also captured articles in the life, physical, and social sciences. Between March 2007 and March 2012, articles selected were classified into an average of 10 different categories per literature review (SD: 4; n= 31) versus an average of 33 categories (SD: 5; n= 3) with the updated process. Conclusions The new search method improves upon the previous method – it captures relevant articles indexed in health science and other secondary databases beyond Medline. The new method has resulted in a greater number of relevant literature articles, from a broader range of disciplines, and in reduced amount of preparation time as compared to the results of the previous search method. This improvement may increase multi-disciplinary discussions and partnerships, but changes in online publishing pose challenges to continued access of the new range of articles.

Objective We examined the utility of combining surveillance data from the Departments of Defense (DoD) and Veterans Affairs (VA) for spatial cluster detection. Introduction The Joint VA/DoD BioSurveillance System for Emerging Biological Threats project seeks to improve situational awareness of the health of VA/DoD populations by combining their respective data. Each system uses a version of the Electronic Surveillance System for Early Notification of Community-Based Epidemics (ESSENCE); a combined version is being tested. The current effort investigated combining the datasets for disease cluster detection. We compared results of retrospective cluster detection studies using both separate and joined data. — Does combining datasets worsen the rate of background cluster determination? — Does combining mask clusters detected on the separate datasets? — Does combining find clusters that the separate datasets alone would miss? Methods Cluster determination runs were done with a spatial scan statistics implementation previously verified [1] by comparison with SaTScan software [2] using DoD data from the Biosense system. Input data files were extracted from a repository of outpatient records from both DoD and VA facilities covering 4 years beginning Jan. 1, 2007. This repository includes over 37 million DoD records and over 86 million VA records. Input files were matrices of daily Influenza-like-Illness (ILI) or gastrointestinal (GI) visit counts. Matrix rows were consecutive days, columns were patient residence zip codes, and entry (i, j) was the number of visits on day i from with zip code j. These files were made for DoD data, VA data, and combined data. For assessing the alerting burden from combining datasets, three sets of runs were executed using data from three regions, Baltimore/Washington D.C. (dominated by DoD data), Los Angeles (mainly VA data), and Tampa (representation of both). For each region, sets of 1672 daily runs were executed for ILI and GI syndrome data. Lastly, focused runs were done to investigate known outbreaks in New York (GI, Jan–Mar 2010), San Diego (ILI, Dec 2007–Apr 2008 and Fall 2009), and New Jersey (GI, Jan–Mar 2010). Results Combining the data sources increased the rate of significant cluster alerting by a manageable 1–10% across run sets. Some clusters found only when the data were combined persisted over several days and may have indicated small events not reported in either system; however, we were unable to validate minor events that may have occurred in past years. Retrospective looks at known outbreaks were successful in that clustering evidence found in separate DoD and VA runs persisted when data sets were combined. For the New York run, a West Point outbreak was seen in repeated clusters of combined data, beginning days before the event report. However, clustering did not consistently produce alerts before outbreak report dates. In the New Jersey DoD runs, repeated clusters indicated a 10-week GI outbreak at Fort Dix; adding VA data that dominated the record counts gave the same clusters with no added cases, so the DoD event was probably self-contained. The San Diego runs were aimed at detecting unusually severe influenza epidemics in February 2008 and in the fall of 2009, and numerous clusters were found but did not enhance regional disease tracking. Conclusions From the analysis, combining DoD and VA data enhances cluster detection capability without loss of sensitivity to events isolated in either population and with manageable effect on the customary alert rate. For cluster detection, there may be many geographic regions where a health monitor in one of the systems would benefit from combined data. More detailed outbreak information is needed to quantify the timeliness/sensitivity advantages of combining datasets. In events examined, clustering itself yielded an occasional but not consistent timeliness advantage.

Objective For a multi-source decision support application, we sought to match univariate alerting algorithms to surveillance data types to optimize detection performance. Introduction Temporal alerting algorithms commonly used in syndromic surveillance systems are often adjusted for data features such as cyclic behavior but are subject to overfitting or misspecification errors when applied indiscriminately. In a project for the Armed Forces Health Surveillance Center to enable multivariate decision support, we obtained 4.5 years of out-patient, prescription and laboratory test records from all US military treatment facilities. A proof-of-concept project phase produced 16 events with multiple evidence corroboration for comparison of alerting algorithms for detection performance. We used the representative streams from each data source to compare sensitivity of 6 algorithms to injected spikes, and we used all data streams from 16 known events to compare them for detection timeliness. Methods The six methods compared were: Holt-Winters generalized exponential smoothing method (1)automated choice between daily methods, regression and an exponential weighted moving average (2)adaptive daily Shewhart-type chartadaptive one-sided daily CUSUMEWMA applied to 7-day means with a trend correction; and7-day temporal scan statistic Sensitivity testing: We conducted comparative sensitivity testing for categories of time series with similar scales and seasonal behavior. We added multiples of the standard deviation of each time series as single-day injects in separate algorithm runs. For each candidate method, we then used as a sensitivity measure the proportion of these runs for which the output of each algorithm was below alerting thresholds estimated empirically for each algorithm using simulated data streams. We identified the algorithm(s) whose sensitivity was most consistently high for each data category. For each syndromic query applied to each data source (outpatient, lab test orders, and prescriptions), 502 authentic time series were derived, one for each reporting treatment facility. Data categories were selected in order to group time series with similar expected algorithm performance: Median > 100 < Median ≤ 10Median = 0Lag 7 Autocorrelation Coefficient ≥ 0.2Lag 7 Autocorrelation Coefficient < 0.2 Timeliness testing: For the timeliness testing, we avoided artificiality of simulated signals by measuring alerting detection delays in the 16 corroborated outbreaks. The multiple time series from these events gave a total of 141 time series with outbreak intervals for timeliness testing. The following measures were computed to quantify timeliness of detection: Median Detection Delay – median number of days to detect the outbreak.Penalized Mean Detection Delay –mean number of days to detect the outbreak with outbreak misses penalized as 1 day plus the maximum detection time. Results Based on the injection results, the Holt-Winters algorithm was most sensitive among time series with positive medians. The adaptive CUSUM and the Shewhart methods were most sensitive for data streams with median zero. Table 1 provides timeliness results using the 141 outbreak-associated streams on sparse (Median=0) and non-sparse data categories. [Insert table #1 here] Data median Detection Delay, days Holt-winters Regression EWMA Adaptive Shewhart Adaptive CUSUM 7-day Trend-adj. EWMA 7-day Temporal Scan Median 0 Median 3 2 4 2 4.5 2 Penalized Mean 7.2 7 6.6 6.2 7.3 7.6 Median >0 Median 2 2 2.5 2 6 4 Penalized Mean 6.1 7 7.2 7.1 7.7 6.6 The gray shading in the table 1 indicates methods with shortest detection delays for sparse and non-sparse data streams. The Holt-Winters method was again superior for non-sparse data. For data with median=0, the adaptive CUSUM was superior for a daily false alarm probability of 0.01, but the Shewhart method was timelier for more liberal thresholds. Conclusions Both kinds of detection performance analysis showed the method based on Holt-Winters exponential smoothing superior on non-sparse time series with day-of-week effects. The adaptive CUSUM and She-whart methods proved optimal on sparse data and data without weekly patterns.

Objective To develop a nontraumatic oral health classification that could estimate the burden of oral health-related visits in North Carolina (NC) Emergency Departments (EDs) using syndromic surveillance system data. Introduction Lack of access to regular dental care often results in costly, oral health visits to EDs that could otherwise have been prevented or managed by a dentist (1). Most studies on oral health-related visits to EDs have used a wide range of classifications from different databases, but none have used syndromic surveillance data. The volume, frequency, and included details of syndromic data enabled timely burden estimates of nontraumatic oral health visits for NC EDs. Methods Literature review, input by subject matter experts (SMEs), and analysis of syndromic data was used to create the nontraumatic oral health classification. BioSense, a near real-time, national-level, electronic health surveillance system was the source of the NC ED syndromic data. Visits with at least one oral health-related ICD-9-CM code were extracted for NC fiscal years 2008–2010. Univariate analyses of chief complaint (CC) and final diagnosis data along with SME consultation were used to determine the CC substrings and ‘white list’ of ICD-9-CM codes used as inclusion criteria to classify visits as oral health-related. These analyses and consultations also determined the trauma-related codes and substrings used to exclude visits. Results Table 1 shows all nontraumatic oral health-related ICD-9-CM codes used for the characterization. Codes likely related to the types of dental emergencies that routine dental care could not have prevented were excluded. Approximately 275,000 patient records were evaluated to determine the CC substrings. The final CC substrings chosen (Table 1) represented over 56% of visits in the candidate record dataset. Over 334,000 BioSense patient records were evaluated, and SMEs reviewed the 32 ICD-9-CM codes that co-occurred most commonly in visits containing oral health-related ICD-9-CM codes to determine which co-occurring ICD-9-CM codes (white list, Table 1) could be present and still maintain the main reason for the visit as an oral health-related problem. Trauma-related visit criteria used for exclusion were derived from a subset of BioSense sub-syndromes (Falls; Fractures and dislocation; Injury, NOS; Sprains and strains; and Motor vehicle traffic accidents) and from selected CC substrings (‘assault’, ‘fight’, and ‘brawl’). In summary, an ED visit had a nontraumatic oral health classification if it contained 1) an oral health-related CC substring with no trauma-related ICD-9-CM codes or CC substrings or 2) an oral health-related ICD-9 code accompanied by no oral health-related or trauma-related CC substrings and with no other diagnosis codes except for those on the whitelist. Conclusions There is increasing demand to determine ways to use syndromic surveillance data in an alternative way for population health surveillance. This use of BioSense data provided a practical classification of patient records for the tracking of nontraumatic oral health-related visits to NC EDs. Visit estimates created using this classification in combination with other pertinent information could prove useful to policymakers when deciding upon resource allocation aimed at reducing this unnecessary burden on the NC ED system. The large volume of records in syndromic surveillance systems offers substantial weight of evidence for alternative use in epidemiological studies; however, accurate classification of records is required to select cases of interest. While data volume precludes validation of every included record, a combination of human expertise and data analysis can provide credible classification criteria.

Objective The project involves analytic combination of multiple evidence sources to monitor health at hundreds of care facilities. A demonstration module featuring a population-based Bayes Network [1] was refined and expanded for application in the Department of Defense Electronic Surveillance System for Community-Based Epidemics (ESSENCE). Introduction The ESSENCE demonstration module was built to help DoD health monitors make routine decisions based on disparate evidence sources such as daily counts of ILI-related chief complaints, ratios of positive lab tests for influenza, patient age distribution, and counts of antiviral prescriptions [1]. The module was a population-based (rather than individual-based) Bayesian network (PBN) in that inputs were algorithmic results from these multiple aggregate data streams, and output was the degree of belief that the combined evidence required investigation. The module reduced total alerts substantially and retained sensitivity to the majority of documented outbreaks while clarifying underlying sources of evidence. The current effort was to advance the prototype to production by refining components of the fusion methodology to improve sensitivity while retaining the reduced alert rate. Methods The multi-level approach to sensitivity improvement included expanded syndromic queries, more data-sensitive algorithm selection, improved transformation of algorithm outputs to alert states, and hierarchical training of Bayesian networks. Components were tested individually, and the net result was iteratively refined with performance using documented outbreaks. We examined time series of classes of prescribed drugs and laboratory tests during known events and discussed outbreak-associated elements with domain experts to liberalize data queries. Algorithms were matched to data streams with injection testing applied to 4.5 years of data from 502 outpatient clinics. A hierarchical approach was applied for improved training and verification of PBNs for events related to categories of Influenza-like Illness, Gastrointestinal, Fever, Neurological, and Rash, chosen both for public health importance and for availability of multiple supporting data types. Hierarchical, modular training was applied to common subnetworks, such as a severity indicator PBN depending on case disposition, acute case indicators, complex evaluation/management codes, and patient bounce-backs, depicted in Figure 1. Conversion of individual algorithm outputs to belief states (e.g. “at least two red alerts/past 7 days”) was broadened using analysis of lags between data sources. With data from the known events, we calculated decision support thresholds for the parent-level PBN decision nodes with a stochastic optimization technique maximizing the ratio of alert rates during outbreak to non-outbreak periods. Results The expanded data queries, more stream-specific algorithm selection, generalized state transformation, and hierarchical PBN training detected 22 of an expanded collection of 24 documented outbreaks, with incremental improvement ongoing. The mean alert rate drop achieved by the Bayes Net was 87% (minimum of 85%) compared to the combined alerts of all component algorithms across syndromes and facilities. Conclusions Expansion and further technical validation upheld the PBN approach as a user-friendly means of analytic decision support given multiple, variably weighted evidence sources. The PBN affords not only sharply reduced alerting, but also transparent indication of evidence underlying each alert. The older algorithm approach remains available as backup. Beta testing of the resulting production system will drive further modification.

Background The U.S. Department of Veterans Affairs (VA) and Department of Defense (DoD) had more than 18 million healthcare beneficiaries in 2011. Both Departments conduct individual surveillance for disease events and health threats. Methods We performed joint and separate analyses of VA and DoD outpatient visit data from October 2006 through September 2010 to demonstrate geographic and demographic coverage, timeliness of influenza epidemic awareness, and impact on spatial cluster detection achieved from a joint VA and DoD biosurveillance platform. Results Although VA coverage is greater, DoD visit volume is comparable or greater. Detection of outbreaks was better in DoD data for 58% and 75% of geographic areas surveyed for seasonal and pandemic influenza, respectively, and better in VA data for 34% and 15%. The VA system tended to alert earlier with a typical H3N2 seasonal influenza affecting older patients, and the DoD performed better during the H1N1 pandemic which affected younger patients more than normal influenza seasons. Retrospective analysis of known outbreaks demonstrated clustering evidence found in separate DoD and VA runs, which persisted with combined data sets. Conclusion The analyses demonstrate two complementary surveillance systems with evident benefits for the national health picture. Relative timeliness of reporting could be improved in 92% of geographic areas with access to both systems, and more information provided in areas where only one type of facility exists. Combining DoD and VA data enhances geographic cluster detection capability without loss of sensitivity to events isolated in either population and has a manageable effect on customary alert rates.