Search

Introduction: Electronic influenza surveillance systems aid in health surveillance and clinical decisionmaking within the emergency department (ED). While major advances have been made in integrating clinical decision-making tools within the electronic health record (EHR), tools for sharing surveillance data are often piecemeal, with the need for data downloads and manual uploads to shared servers, delaying time from data acquisition to end-user. Real-time surveillance can help both clinicians and public health professionals recognize circulating influenza earlier in the season and provide ongoing situational awareness. Methods: We created a prototype, cloud-based, real-time reporting system in two large, academically affiliated EDs that streamed continuous data on a web-based dashboard within hours of specimen collection during the influenza season. Data included influenza test results (positive or negative) coupled with test date, test instrument geolocation, and basic patient demographics. The system provided immediate reporting to frontline clinicians and to local, state, and federal health department partners. Results: We describe the process, infrastructure requirements, and challenges of developing and implementing the prototype system. Key process-related requirements for system development included merging data from the molecular test (GeneXpert) with the hospitals’ EHRs, securing data, authorizing/ authenticating users, and providing permissions for data access refining visualizations for end-users. Conclusion: In this case study, we effectively integrated multiple data systems at four distinct hospital EDs, relaying data in near real time to hospital-based staff and local and national public health entities, to provide laboratory-confirmed influenza test results during the 2014-2015 influenza season. Future innovations need to focus on integrating the dashboard within the EHR and clinical decision tools.

Background A high proportion of health care services are persistently utilized by a small subpopulation of patients. To improve clinical outcomes while reducing costs and utilization, population health management programs often provide targeted interventions to patients who may become persistent high users/utilizers (PHUs). Enhanced prediction and management of PHUs can improve health care system efficiencies and improve the overall quality of patient care. Objective The aim of this study was to detect key classes of diseases and medications among the study population and to assess the predictive value of these classes in identifying PHUs. Methods This study was a retrospective analysis of insurance claims data of patients from the Johns Hopkins Health Care system. We defined a PHU as a patient incurring health care costs in the top 20% of all patients’ costs for 4 consecutive 6-month periods. We used 2013 claims data to predict PHU status in 2014-2015. We applied latent class analysis (LCA), an unsupervised clustering approach, to identify patient subgroups with similar diagnostic and medication patterns to differentiate variations in health care utilization across PHUs. Logistic regression models were then built to predict PHUs in the full population and in select subpopulations. Predictors included LCA membership probabilities, demographic covariates, and health utilization covariates. Predictive powers of the regression models were assessed and compared using standard metrics. Results We identified 164,221 patients with continuous enrollment between 2013 and 2015. The mean study population age was 19.7 years, 55.9% were women, 3.3% had ≥1 hospitalization, and 19.1% had 10+ outpatient visits in 2013. A total of 8359 (5.09%) patients were identified as PHUs in both 2014 and 2015. The LCA performed optimally when assigning patients to four probability disease/medication classes. Given the feedback provided by clinical experts, we further divided the population into four diagnostic groups for sensitivity analysis: acute upper respiratory infection (URI) (n=53,232; 4.6% PHUs), mental health (n=34,456; 12.8% PHUs), otitis media (n=24,992; 4.5% PHUs), and musculoskeletal (n=24,799; 15.5% PHUs). For the regression models predicting PHUs in the full population, the F1-score classification metric was lower using a parsimonious model that included LCA categories (F1=38.62%) compared to that of a complex risk stratification model with a full set of predictors (F1=48.20%). However, the LCA-enabled simple models were comparable to the complex model when predicting PHUs in the mental health and musculoskeletal subpopulations (F1-scores of 48.69% and 48.15%, respectively). F1-scores were lower than that of the complex model when the LCA-enabled models were limited to the otitis media and acute URI subpopulations (45.77% and 43.05%, respectively). Conclusions Our study illustrates the value of LCA in identifying subgroups of patients with similar patterns of diagnoses and medications. Our results show that LCA-derived classes can simplify predictive models of PHUs without compromising predictive accuracy. Future studies should investigate the value of LCA-derived classes for predicting PHUs in other health care settings.

Invited discussion paper for "A Review of Some Sampling and Aggregation Strategies for Basic Statistical Process Monitoring" by Zwetsloot and Woodall.

BACKGROUND The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) is a secure web-based tool that enables health care practitioners to monitor health indicators of public health importance for the detection and tracking of disease outbreaks, consequences of severe weather, and other events of concern. The ESSENCE concept began in an internally funded project at the Johns Hopkins University Applied Physics Laboratory, advanced with funding from the State of Maryland, and broadened in 1999 as a collaboration with the Walter Reed Army Institute for Research. Versions of the system have been further developed by Johns Hopkins University Applied Physics Laboratory in multiple military and civilian programs for the timely detection and tracking of health threats. OBJECTIVE This study aims to describe the components and development of a biosurveillance system increasingly coordinating all-hazards health surveillance and infectious disease monitoring among large and small health departments, to list the key features and lessons learned in the growth of this system, and to describe the range of initiatives and accomplishments of local epidemiologists using it. METHODS The features of ESSENCE include spatial and temporal statistical alerting, custom querying, user-defined alert notifications, geographical mapping, remote data capture, and event communications. To expedite visualization, configurable and interactive modes of data stratification and filtering, graphical and tabular customization, user preference management, and sharing features allow users to query data and view geographic representations, time series and data details pages, and reports. These features allow ESSENCE users to gather and organize the resulting wealth of information into a coherent view of population health status and communicate findings among users. RESULTS The resulting broad utility, applicability, and adaptability of this system led to the adoption of ESSENCE by the Centers for Disease Control and Prevention, numerous state and local health departments, and the Department of Defense, both nationally and globally. The open-source version of Suite for Automated Global Electronic bioSurveillance is available for global, resource-limited settings. Resourceful users of the US National Syndromic Surveillance Program ESSENCE have applied it to the surveillance of infectious diseases, severe weather and natural disaster events, mass gatherings, chronic diseases and mental health, and injury and substance abuse. CONCLUSIONS With emerging high-consequence communicable diseases and other health conditions, the continued user requirement–driven enhancements of ESSENCE demonstrate an adaptable disease surveillance capability focused on the everyday needs of public health. The challenge of a live system for widely distributed users with multiple different data sources and high throughput requirements has driven a novel, evolving architecture design.

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) is a secure web-based tool that enables health care practitioners to monitor health indicators of public health importance for detection and tracking of disease outbreaks, consequences of severe weather, and other events of concern. The ESSENCE concept began in an internally funded project at the Johns Hopkins University Applied Physics Laboratory (JHU/APL), advanced with funding from the State of Maryland, and broadened in 1999 as a collaboration with the Walter Reed Army Institute for Research. Versions of the system have been further developed by JHU/APL in multiple military and civilian programs for timely detection and tracking of health threats. Features of ESSENCE include spatial and temporal statistical alerting, custom querying, user-defined alert notifications, geographical mapping, remote data capture, and event communications. These features allow ESSENCE users to gather and organize the resulting wealth of information into a coherent view of population health status and communicate findings among users. The resulting broad utility, applicability and adaptability of this system led to adoption of ESSENCE by the Centers for Disease Control and Prevention (CDC), numerous state and local health departments, and the Department of Defense (DOD) both nationally and globally. With emerging high-consequence communicable diseases and other health conditions, the continued user-requirements-driven enhancements of ESSENCE demonstrate an adaptable disease surveillance capability focused on the everyday needs of public health. The challenge of a live system for widely distributed users with multiple different data sources and high throughput requirements has driven an novel, evolving architecture design.

Display Omitted Used data from12 national notifiable diseases and simulated alerting signals.Compared 4 detection methods by background alert rate, sensitivity, and timeliness.Long-term trend adjustment improved detection performance.The modified Historical Limits Method (HLM) outperformed the traditional HLM.The Farrington-like method provided the best detection performance overall. To compare the performance of the standard Historical Limits Method (HLM), with a modified HLM (MHLM), the Farrington-like Method (FLM), and the Serfling-like Method (SLM) in detecting simulated outbreak signals. We used weekly time series data from 12 infectious diseases from the U.S. Centers for Disease Control and Preventions National Notifiable Diseases Surveillance System (NNDSS). Data from 2006 to 2010 were used as baseline and from 2011 to 2014 were used to test the four detection methods. MHLM outperformed HLM in terms of background alert rate, sensitivity, and alerting delay. On average, SLM and FLM had higher sensitivity than MHLM. Among the four methods, the FLM had the highest sensitivity and lowest background alert rate and alerting delay. Revising or replacing the standard HLM may improve the performance of aberration detection for NNDSS standard weekly reports.

Objective Our project goal is to enhance the capability of automating health surveillance[MOU1] by US Department of Defense (DoD) epidemiologists. We employ software tools that build and train Bayesian networks (BNs) to facilitate the development of analytic fusion of multiple, disparate data sources comprising both syndromic and diagnostic data streams for rapid estimation of overall levels of concern for potential disease outbreaks. Working with previously developed heuristic BNs, we evaluate the ability of machine learning algorithms to detect outbreaks with greater accuracy. We use historical training data on the ability to detect outbreaks of influenza-like illness (ILI). Introduction The motivation for this project is to provide greater situational awareness to DoD epidemiologists monitoring the health of military personnel and their dependents. An increasing number of data sources of varying clinical specificity and timeliness are available to the staff. The challenge is to integrate all the information for a coherent, up-to-date view of population health. Developers at the Johns Hopkins Applied Physics Laboratory, in collaboration with medical epidemiologists at the Armed Forces Health Surveillance Branch, previously designed a multivariate decision support tool to add to the DoD implementation of the Electronic Surveillance System for Early Notification of Community-Based Epidemics (ESSENCE). Data sources included clinical encounter records including free-text chief complaints, filled prescription records, and laboratory test orders and results. Filtered data streams were derived from these sources for daily monitoring, and alerting algorithms were customized and applied to the resulting time series. We built BNs to derive overall levels of concern from the collection of data streams and algorithm outputs to derive, in the form of daily fusion alerts, the overall level of various outbreak concerns. Visualizations made apparent which data features accounted for these concerns, including drill-down to the level of patient record details. Advantages of the BN approach are this transparency and the capacity for assessments using incomplete data and incorporating novel and report-based data streams. The need for such fusion was nearly unanimous in a global survey of public health epidemiologists [1]. Our proof-of-concept system based on commercial BN software was well received by a cross-section of DoD health monitors. The new software tools we apply in this project use freely available R packages which provide more comprehensive tools for BN training and development. These results will allow us to improve the analytic fusion abilities of DoD ESSENCE, as well as in civilian surveillance systems Our testing procedures and results are presented below. Methods We employ a 3.75-year dataset (2006-2010) with information from 502 US medical treatment facilities including 289 hospitals. Our data include time series of daily counts and alerting algorithm outputs from each facility for syndrome groups based on a) chief complaints and diagnosis codes from clinic visits, b) groups of laboratory test orders and influenza test results, and c) selected groups of filled prescriptions. For each facility group, the challenge is to combine these data streams into a daily assessment of levels of concern for an ILI outbreak. The software developed in this project facilitates the formation, training, and testing of BNs for outbreak alerting based on the datasets above. Underlying each BN is a directed acyclic graph whose leaf nodes represent discrete states of concern for each data stream, ranging from general streams such as ILI-related chief complaints to specific ones such as positive flu test results. The states are derived from both daily stream counts and alerting algorithm outputs. Internal nodes represent mid-level combinations of indicators, such as ILI concern based only on clinic data, and parent nodes which represent the calculated level of concern based on all data sources. The connectivity between the nodes, and the orientation of the edges of the graph are determined by the heuristic relationships adopted in previous projects [2]. Each node is associated with a conditional probability table (CPT). Training a BN requires state assignments for every node in the underlying graph, which must be obtained from a trusted source. These assignments can generate CPTs at each node of the BN such that, when given only a set of evidence nodes, the levels of concern can be propagated up the network to provide the desired levels of concern at the decision nodes. Truth data for ILI outbreaks comes from two sources: 33 documented outbreaks with dates supplied by DoD surveillance reports or media articles and, because many modest-sized outbreaks are unreported, 81 unconfirmed data-derived events with algorithmic alerts across multiple data sources. We use data from the more numerous unconfirmed events to train the BNs. To avoid commercial software constraints, we use the free R package gRain [3] to create, train, and test the BNs. We test multiple BNs for multivariate ILI outbreak detection, all based on the same nodal structure with 18 parent, intermediate, and leaf nodes. Candidate BNs are created and trained using either a) CPTs determined with a multivariate stochastic search in the previous project, augmented with ground truth data or b) a heuristic lookup table of state combinations. Results Table 1 shows the high odds ratios calculated for candidate BNs. These statistics are calculated for decision node outputs for event vs non-event dates in the truth data with the constraint that every event is detected for at least one date. The machine learning advantage from the training data is evident from comparing the two rows. To show the advantage of fusing data sources, Table 2 gives analogous odds ratios based on single-stream alerting algorithms. Aside from the lower detection statistics, single streams offer no corroboration of statistical alerts. Conclusions Analytic fusion is essential for the efficient, timely use of a growing collection of complex, streaming information by a limited workforce of human health monitors. This project builds upon previous fusion capability for automated health surveillance by expediting new development and facilitating software implementation through open source tools. The detection results indicate a significant advantage in both sensitivity and alert rates of automated systems achievable with machine learning. Moreover, if basic challenges of multivariate data acquisition and determination of truth datasets for supervised learning can be met, further improvements are likely using BN structure discovery methods as well as other machine learning approaches. References 1. Hopkins RS, Tong CC, Burkom HS, et al. A Practitioner-Driven Research Agenda for Syndromic Surveillance. Public Health Rep. 2017;132(1_suppl):116s-126s. 2. Burkom H.S., Elbert Y., Ramac-Thomas L., Cuellar C., Hung V., Refinement of a Population-Based Bayesian Network for Fusion of Health Surveillance Data, Online Journal of Public Health Informatics, Vol 5, No 1 (2013), DOI: https://doi.org/10.5210/ojphi.v5i1.4413. 3. Hojsgaard S., Graphical Independence Networks with the gRain Package for R, Journal of Statistical Software, (2012) 46(10), DOI: 10.18637/jss.v046.i10.

Objective In a partnership between the Public Health Division of the Oregon Health Authority (OHA) and the Johns Hopkins Applied Physics Laboratory (APL), our objective was develop an analytic fusion tool using streaming data and report-based evidence to improve the targeting and timing of evidence-based interventions in the ongoing opioid overdose epidemic. The tool is intended to enable practical situational awareness in the ESSENCE biosurveillance system to target response programs at the county and state levels. Threats to be monitored include emerging events and gradual trends of overdoses in three categories: all prescription and illicit opioids, heroin, and especially high-mortality synthetic drugs such as fentanyl and its analogues. Traditional sources included emergency department (ED) visits and emergency management services (EMS) call records. Novel sources included poison center calls, death records, and report-based information such as bad batch warnings on social media. Using available data and requirements analyses thus far, we applied and compared Bayesian networks, decision trees, and other machine learning approaches to derive robust tools to reveal emerging overdose threats and identify at-risk subpopulations. Introduction Unlike other health threats of recent concern for which widespread mortality was hypothetical, the high fatality burden of opioid overdose crisis is present, steadily growing, and affecting young and old, rural and urban, military and civilian subpopulations. While the background of many public health monitors is mainly infectious disease surveillance, these epidemiologists seek to collaborate with behavioral health and injury prevention programs and with law enforcement and emergency medical services to combat the opioid crisis. Recent efforts have produced key terms and phrases in available data sources and numerous user-friendly dashboards allowing inspection of hundreds of plots. The current effort seeks to distill and present combined fusion alerts of greatest concern from numerous stratified data outputs. Near-term plans are to implement best-performing fusion methods as an ESSENCE module for the benefit of OHA staff and other user groups. Methods By analyzing historical OHA data, we formed features to monitor in each data source to adapt diagnosis codes and text strings suggested by CDC’s injury prevention division, published EMS criteria [Reference 1], and generic product codes from CDC toxicologists, with guidance from OHA Emergency Services Director David Lehrfeld and from Oregon Poison Center Director Sandy Giffen. These features included general and specific opioid abuse indicators such as daily counts of records labelled with the “poisoning” subcategory and containing “fentanyl” or other keywords in the free-text. Matrices of corresponding time series were formed for each of 36 counties and the entire state as inputs to region-specific fusion algorithms. To obtain truth data for detection, the OHA staff provided guidance and design help to generate plausible overdose threat scenarios that were quantified as realistic data distributions of monitored features accounting for time delays and historical distributions of counts in each data source. We sampled these distributions to create 1000 target sets for detection based on the event duration and affected counties for each event scenario. We used these target datasets to compare the detection performance of fusion detection algorithms. Tested algorithms included Bayesian Networks formed with the R package gRain, and also random forest, logistic regression, and support vector machine models implemented with the Python scikit-learn package using default settings. The first 800 days of the data were used for model training, and the last 400 days for testing. Model results were evaluated with the metrics: Sensitivity = (number of target event days signaled) / (all event days) and Positive predictive value (PPV) = (number of target event days signaled) / (all days signaled). These metrics were combined with specificity regarded as the expected fusion alert rate calculated from the historical dataset with no simulated cases injected. Results The left half of Figure 1 illustrates a threat scenario along Oregon’s I5 corridor in which string of fentanyl overdoses with a few fatalities affects the monitored data streams in three counties over a seven-day period. The right half of the figure charts the performance metrics for random forest and Bayesian network machine learning methods applied to both training and test datasets assuming total case counts of 50, 20, and 10 overdoses. Sensitivity values were encouraging, especially for the Bayesian networks and even for the 10-case scenario. Computed PPV levels suggested a manageable public health investigation burden. Conclusions The detection results were promising for a threat scenario of particular concern to OHA based on a data scenario deemed plausible and realistic based on historical data. Trust and acceptance from public health surveillance of outputs from supervised machine learning methods beyond traditional statistical methods will require user experience and similar evaluation with additional threat scenarios and authentic event data. Credible truth data can be generated for testing and evaluation of analytic fusion methods with the advantages of several years of historical data from multiple sources and the expertise of experienced monitors. The collaborative generation process may be standardized and extended to other threat types and data environments. Next steps include the addition to the analytic fusion capability of report-based data that can influence data interpretation, including mainstream and social media reports, events in neighboring regions, and law enforcement data. References 1. Rhode Island Enhanced State Opioid Overdose Surveillance (ESOOS) Case Definition for Emergency Medical Services (EMS), http://www.health.ri.gov/publications/guidelines/ESOOSCaseDefinitionForEMS.pdf, last accessed: Sept. 9, 2018.

Objective This abstract describes an ISDS initiative to bring together public health practitioners and analytics solution developers from both academia and industry to define a roadmap for the development of algorithms, tools, and datasets to improve the capabilities of current text processing algorithms to identify negated terms (i.e. negation detection). Introduction Despite considerable effort since the turn of the century to develop Natural Language Processing (NLP) methods and tools for detecting negated terms in chief complaints, few standardised methods have emerged. Those methods that have emerged (e.g. the NegEx algorithm [1]) are confined to local implementations with customised solutions. Important reasons for this lack of progress include (a) limited shareable datasets for developing and testing methods (b) jurisdictional data silos, and (c) the gap between resource-constrained public health practitioners and technical solution developers, typically university researchers and industry developers. To address these three problems ISDS, funded by a grant from the Defense Threat Reduction Agency, organized a consultancy meeting at the University of Utah designed to bring together (a) representatives from public health departments, (b) university researchers focused on the development of computational methods for public health surveillance, (c) members of public health oriented non-governmental organisations, and (d) industry representatives, with the goal of developing a roadmap for the development of validated, standardised and portable resources (methods and data sets) for negation detection in clinical text used for public health surveillance. Methods Free-text chief complaints remain a vital resource for syndromic surveillance. However, the widespread adoption of Electronic Health Records (and federal Meaningful Use requirements) has brought changes to the syndromic surveillance practice ecosystem. These changes have included the widespread use of EHR-generated chief complaint “pick lists” (i.e. pre-defined chief complaints that are selected by the user, rather than text strings input by the user at a keyboard), triage note templated text, and triage note free-text (typically much more comprehensive than traditional chief complaints). A key requirement for a negation detection algorithm is the ability to successfully and accurately process these new and challenging data streams. Preparations for the consultancy included an email thread and a shared website for published articles and data samples leading to a structured pre-consultancy call designed to inform participants regarding the purpose of the consultancy and to align expectations. Then, health department users were requested to provide data samples exemplifying negation issues in the classification process. Presenting developers were asked to explain their underlying ideas, details of method implementation, size and composition of corpora used for evaluation, and classification performance results. Results The consultancy was held on January 19th & 20th 2017 at the University of Utah’s Department of Biomedical Informatics, and consisted of 25 participants. Participants were drawn from various different sectors, with representation from ISDS (2), the Defense Threat Reduction Agency (1), universities and research institutes (10), public health departments (5), the Department of Veterans Affairs (4), non-profit organisations (2), and technology firms (1). Participants were drawn from a variety of different professional backgrounds, including research scientists, software developers, public health executives, epidemiologists, and analysts. Day 1 of the consultancy was devoted to providing an overview of NLP and current trends in negation detection, including a detailed description of widely used algorithms and tools for the negation detection task. Key questions included: Should our focus be chief complaints only, or should we widen our scope to emergency department triage notes?, How many other NLP tasks (e.g. reliable concept recognition) is it necessary to address on the road to improved negation detection? With this background established, Day 2 centered on presentations from five different United States local and regional health departments (King County WA, Boston MA, North Carolina, Georgia, and Tennessee) on the various approaches to text processing and negation detection utilized across several jurisdictions. Several key areas of focus emerged as a result of the consultancy discussion. First, there is a clear need for a large, easily accessible corpus of free-text chief complaints that can form a standardised testbed for negation detection algorithm development and evaluation. Annotated data, in this context, consists of chief complaints annotated for concepts (e.g. vomiting, pain in chest) and the negation status of those concepts. It is important that the annotation include both annotated clinical concepts and negation status to allow for the uniform evaluation and performance comparison of candidate negation detection algorithms. Further, the annotated corpus should consist of several thousand (as opposed to several hundred) distinct and representative chief complaints in order to compare algorithms against a sufficient variety and volume of negation patterns. Conclusions The consultancy was stimulating and eye-opening for both public health practitioner and technology developer attendees. Developers unfamiliar with the everyday health-monitoring context gained an appreciation of the difficulty of deriving useful indicators from chief complaints. Also highlighted was the challenge of processing triage notes and other free-text fields that are often unused for surveillance purposes. Practitioners were provided with concise explanations and evaluations of recent NLP approaches applicable to negation processing. The event afforded direct dialogue important for communication across professional cultures. Please note that a journal paper describing the consultancy has recently been published in the Online Journal of Public Health Informatics [2]. References [1] Chapman W, Bridewell W, Henbury P, Cooper G, Buchanan B. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform. 2001, 34(5):301-310. [2] Conway M, Mowery D, Ising A, Velupillai S, Doan S, Gunn J, Donovan M, Wiedeman C, Ballester L, Soetebier K, Tong C, Burkom H. Cross-disciplinary consultance to breige public health technical needs and analytic developers: negation detection use case. Online Journal of Public Health Informatics. 2018, 10(2)

Introduction The Risk Identification Unit (RIU) of the US Dept. of Agriculture’s Center for Epidemiology and Animal Health (CEAH) conducts weekly surveillance of national livestock health data and routine coordination with agricultural stakeholders. As part of an initiative to increase the number of species, health issues, and data sources monitored, CEAH epidemiologists are building a surveillance system based on weekly syndromic counts of laboratory test orders in consultation with Colorado State University laboratorians and statistical analysts from the Johns Hopkins University Applied Physics Laboratory. Initial efforts focused on 12 years of equine test records from three state labs. Trial syndrome groups were formed based on RIU experience and published literature. Exploratory analysis, stakeholder input, and laboratory workflow details were needed to modify these groups and filter the corresponding data to eliminate alerting bias. Customized statistical detection methods were sought for effective monitoring based on specialized laboratory information characteristics and on the likely presentation and animal health significance of diseases associated with each syndrome. Methods Data transformation and syndrome formation focused on test battery type, test name, submitter source organization, and specimen type. We analyzed time series of weekly counts of tests included in candidate syndrome groups and conducted an iterative process of data analysis and veterinary consultation for syndrome refinement and record filters. This process produced a rule set in which records were directly classified into syndromes using only test name when possible, and otherwise, the specimen type or related body system was used with test name to determine the syndrome. Test orders associated with government regulatory programs, veterinary teaching hospital testing protocols, or research projects, rather than clinical concerns, were excluded. We constructed a testbed for sets of 1000 statistical trials and applied a stochastic injection process assuming lognormally distributed incubation periods to choose an alerting algorithm with the syndrome-required sensitivity and an alert rate within the specified acceptable range for each resulting syndrome. Alerting performance of the EARS C3 algorithm traditionally used by CEAH was compared to modified C2, CuSUM, and EWMA methods, with and without outlier removal and adjustments for the total weekly number of non-mandatory tests. Results The equine syndrome groups adopted for monitoring were abortion/reproductive, diarrhea/GI, necropsy, neurological, respiratory, systemic fungal, and tickborne. Data scales, seasonality, and variance differed widely among the weekly time series. Removal of mandatory and regulatory tests reduced weekly observed counts significantly—by >80% for diarrhea/GI syndrome. The RIU group studied outcomes associated with each syndrome and called for detection of single-week signals for most syndromes with expected false-alert intervals >8 and

Driven by the growing importance of situational awareness of bioterrorism threats, the Republic of Korea (ROK) and the United States have constructed a joint military capability, called the Biosurveillance Portal (BSP), to enhance biosecurity. As one component of the BSP, we developed the Military Active Real-time Syndromic Surveillance (MARSS) system to detect and track natural and deliberate disease outbreaks. This article describes the ROK military health data infrastructure and explains how syndromic data are derived and made available to epidemiologists. Queries corresponding to 8 syndromes, based on published clinical effects of weaponized pathogens, were used to classify military hospital patient records to form aggregated daily syndromic counts. A set of ICD-10 codes for each syndrome was defined through literature review and expert panel discussion. A study set of time series of national daily counts for each syndrome was extracted from the Defense Medical Statistical Information System between January 1, 2011, and May 31, 2014. A stratified, adjusted cumulative summation algorithm was implemented for each syndrome group to signal alerts prompting investigation. The algorithm was developed by calculating sensitivity to sets of 1,000 artificial outbreak signals randomly injected in the dataset, with each signal injected in a separate trial. Queries and visualizations were adapted from the Suite for Automated Global bioSurveillance. Findings indicated that early warning of outbreaks affecting fewer than 50 patients will require analysis at subnational levels, especially for common syndrome groups. Developing MARSS to improve sensitivity will require modification of underlying syndromic diagnosis codes, engineering to coordinate alerts among subdivisions, and enhanced algorithms. The bioterrorist threat in the Korean peninsula mandates these efforts.

An editorial is presented which addresses the author's views about the evolution of public health surveillance in America since the late 1990s, and it mentions the author's work on surveillance system development at the Johns Hopkins University Applied Physics Laboratory and as a consultant to the U.S. Centers for Disease Control and Prevention. The American federal government's HSPD-10 directive on bioterrorism is examined, along with chronic diseases and advances in biotechnology.

Display Omitted We used syndromic data from 62 VHA hospitals and simulated alerting signals.We compared 7 detection methods on sensitivity and timeliness of alerting.Weekday/weekend stratification improved detection performance.Total visit adjustment increased sensitivity and timeliness.A Poisson regression provided the best sensitivity for high-count data. National syndromic surveillance systems require optimal anomaly detection methods. For method performance comparison, we injected multi-day signals stochastically drawn from lognormal distributions into time series of aggregated daily visit counts from the U.S. Centers for Disease Control and Prevention's BioSense syndromic surveillance system. The time series corresponded to three different syndrome groups: rash, upper respiratory infection, and gastrointestinal illness. We included a sample of facilities with data reported every day and with median daily syndromic counts ?1 over the entire study period. We compared anomaly detection methods of five control chart adaptations, a linear regression model and a Poisson regression model. We assessed sensitivity and timeliness of these methods for detection of multi-day signals. At a daily background alert rate of 1% and 2%, the sensitivities and timeliness ranged from 24 to 77% and 3.3 to 6.1days, respectively. The overall sensitivity and timeliness increased substantially after stratification by weekday versus weekend and holiday. Adjusting the baseline syndromic count by the total number of facility visits gave consistently improved sensitivity and timeliness without stratification, but it provided better performance when combined with stratification. The daily syndrome/total-visit proportion method did not improve the performance. In general, alerting based on linear regression outperformed control chart based methods. A Poisson regression model obtained the best sensitivity in the series with high-count data.

Objective To obtain feedback and seek future directions for an ISDS initiative to establish and update research questions in Informatics, Analytics, Communications, and Systems Research with the greatest perceived impact for improving surveillance practice. Introduction Over the past fifteen years, syndromic surveillance (SyS) has evolved from a set of ad hoc methods used mostly in post-disaster settings, then expanded with broad support and development because of bioterrorism concerns, and subsequently evolved to a mature technology that runs continuously to detect and monitor a wide range of health issues. Continued enhancements needed to meet the challenges of novel health threats with increasingly complex information sources will require technical advances focused on day-to-day public health needs. Since its formation in 2005, the International Society for Disease Surveillance (ISDS) has sought to clarify and coordinate global priorities in surveillance research. As part of a practitioner-driven initiative to identify current research priorities in SyS, ISDS polled its members about capabilities needed by SyS practitioners that could be improved as a result of research efforts. A taskforce of the ISDS Research Committee, consisting of national and global subject matter experts (SMEs) in SyS and ISDS professional staff, carried out the project. This panel will discuss the results and the preferred means to determine and communicate priorities in the future.

Objective Our objective was to compare the effectiveness of applying the historical limits method (HLM) to poison center (PC) call volumes with vs without stratifying by exposure type. Introduction The Centers for Disease Control and Prevention (CDC) uses the National Poison Data System (NPDS) to conduct surveillance of calls to United States PCs. PCs provide triage and treatment advice for hazardous exposures through a free national hotline. Information on demographics, health effects, implicated substance(s), medical outcome of the patient, and other variables are collected. CDC uses automated algorithms to identify anomalies in both pure call volume and specific clinical effect volume, and to identify calls reporting exposure to high priority agents. Pure and clinical effect volume anomalies are identified when an hourly call count exceeds a threshold based on historical data using HLM. 1 Clinical toxicologists and epidemiologists at the American Association of Poison Control Centers and CDC apply standardized criteria to determine if the anomaly identifies a potential incident of public health significance (IPHS) and to notify the respective health departments and local PCs as needed. Discussions with NPDS users and analysis of IPHS showed that alerting based on pure call volume yielded excessive false positives. A study using a 5-year NPDS call dataset assessed the positive predictive value (PPV) of the call volume-based approach. This study showed that less than 4% of anomalies were IPHS. 2 A low PPV can cause unnecessary waste of staff time and resources analyzing false positive anomalies. As an alternative to pure call volume-based detection where all calls to each PC are aggregated for anomaly detection, we considered separating calls by toxicologically-relevant exposure categories for more targeted anomaly detection. We hypothesized that this stratified approach would reduce the number of false positives. Methods We derived our exposure categories based on the criteria that the categories must: 1) relate to hazardous exposures of public health importance, 2) reflect categories based on clinical effects and treatment modalities, 3) avoid high priority exposures that may be triggered by single calls, 4) be compatible with exposure substance identification codes currently used by PCs and NPDS, and 5) include enough calls for meaningful tracking. We queried all calls reporting exposures to the proposed categories between January 1, 2009 and July 31, 2015 for ten PCs. We applied the HLM method after stratifying by exposure category and tabulated the number of alerts triggered for each category during the study period. We then applied the HLM method for the ten PCs on all combined exposure calls to represent the traditional non-stratified approach. We compared the combined alert burden generated by stratifying by exposure category with the alert burden for the non-stratified approach for varying time windows (1-, 2-, 4-, 8- and 24-hours). We conducted analysis in R. Results We derived a total of 20 exposure categories, including chemicals (n=4), drugs of abuse (n=6), pesticides (n=3), gas/fume/vapors (n=2), contaminated food/water (n=1), and others (n=4). Call counts during 2015 for these categories ranged from approximately 5,000 to 90,000. Table 1 shows the total number of alerts triggered for each method by time windows. There was a marked reduction of alert burden when first stratifying by exposure category for time windows shorter than eight hours compared to the alert burden for the non-stratified approach. Conclusions Stratification of call volume by exposure category and time window suggests potential improvement over traditional non- stratified approach by having a lower alert burden. Further work should focus on refining the exposure categories, refining the time window for surveillance, and assessing other detection performance metrics, such as sensitivity. Table 1: Alert burden comparison for the non-stratified vs stratified approach

Objective Our objective was to determine if the detection performance of current surveillance algorithms to detect call clusters is improved by stratifying by exposure category. Introduction The Centers for Disease Control and Prevention (CDC) uses the National Poison Data System (NPDS) to conduct surveillance of calls to United States poison centers (PCs) to identify clusters of reports of hazardous exposures and illnesses. NPDS stores basic information from PC calls including call type (information request only or call reporting a possible chemical exposure), exposure agent, demographics, clinical, and other variables. CDC looks for anomalies in PC data by using automated algorithms to analyze call and clinical effect volume, and by identifying calls reporting exposures to pre-specified high priority agents. Algorithms analyzing call and clinical effect volume identify anomalies when the number of calls exceeds a threshold using the historical limits method (HLM). Clinical toxicologists and epidemiologists at the American Association of Poison Control Centers and CDC apply standardized criteria to determine if the anomaly is a potential incident of public health significance (IPHS) and then notify the respective health departments and PCs as needed. Discussions with surveillance system users and analysis of past IPHS determined that call volume-based surveillance results in a high proportion of false positive anomalies. A study assessing the positive predictive value (PPV) of this approach determined that fewer than four percent of anomalies over a five-year period were IPHS. 1 A low PPV can cause an unnecessary waste of staff time and resources. We hypothesized that first stratifying call volume by exposure category would reduce the number of false positives. With the help of medical toxicologists, we created 20 toxicologically-relevant exposure categories to test this hypothesis. Methods To compare cluster detection performance between the two approaches, we used a historical testbed of hourly exposure call counts with and without initial stratification by exposure category from 10 selected PCs from Jan 1, 2006 - Jul 31, 2015. We ran the HLM for both non-stratified and stratified testbeds to estimate the monthly number of anomalies triggered (i.e., alert burden). Our target signals to assess detection performance consisted of call samples from three large public health events: the 2009 Salmonella food poisoning event from contaminated peanut butter, the 2012 Hurricane Sandy- associated carbon monoxide poisonings in New Jersey, and the 2014 Elk River contaminated water spill in West Virginia (WV). For each event, we chose 30 random calls one thousand times to obtain 1000 random sets of inject clusters. Each inject cluster was iteratively added into the testbed with and without initial stratification by exposure category. We then applied the HLM for each iteration to see if the algorithm identified the inject cluster. The sensitivity for each approach for each PC was calculated as the proportion of iterations where the algorithm identified the inject cluster. We reported median sensitivities from the ten PCs for each of the time windows of 1, 2, 4, 8, and 24 hours. Results Figure 1 summarizes results for the WV event with markers showing anomaly burden (x-axis) and sensitivity (y-axis) using the stratified ( Δ ) and the non-stratified (o) approach by different time windows (hrs). The results from the other two events are not shown but established similar patterns. Anomaly burden is shown as the estimated monthly anomaly count for each approach. For example, markers linked by the arrow show that with a 4-hour time window, the stratified approach achieves nearly perfect sensitivity with ~10 anomalies as the monthly anomaly burden while sensitivity of the non-stratified approach is below 20% with ~40 monthly anomalies. The stratified approach gave improved overall sensitivity across all time windows, and reduced anomaly burden for 1-, 2-, and 4-hour time windows. Conclusions We found a consistent detection advantage (higher sensitivity and lower anomaly burden) for the stratified vs traditional non- stratified approach for 1-, 2-, and 4-hour time windows. Further research should focus on refining the stratified approach and the specific surveillance parameters (such as time windows) that increase algorithm performance. Figure 1: Detection performance comparison: stratified vs non-stratified approach; 2014 Elk River contaminated water spill in West Virginia scenario

Syndromic surveillance has expanded since 2001 in both scope and geographic reach and has benefited from research studies adapted from numerous disciplines. The practice of syndromic surveillance continues to evolve rapidly. The International Society for Disease Surveillance solicited input from its global surveillance network on key research questions, with the goal of improving syndromic surveillance practice. A workgroup of syndromic surveillance subject matter experts was convened from February to June 2016 to review and categorize the proposed topics. The workgroup identified 12 topic areas in 4 syndromic surveillance categories: informatics, analytics, systems research, and communications. This article details the context of each topic and its implications for public health. This research agenda can help catalyze the research that public health practitioners identified as most important.

This paper continues an initiative conducted by the International Society for Disease Surveillance with funding from the Defense Threat Reduction Agency to connect near-term analytical needs of public health practice with technical expertise from the global research community. The goal is to enhance investigation capabilities of day-to-day population health monitors. A prior paper described the formation of consultancies for requirements analysis and dialogue regarding costs and benefits of sustainable analytic tools. Each funded consultancy targets a use case of near-term concern to practitioners. The consultancy featured here focused on improving predictions of asthma exacerbation risk in demographic and geographic subdivisions of the city of Boston, Massachusetts, USA based on the combination of known risk factors for which evidence is routinely available. A cross-disciplinary group of 28 stakeholders attended the consultancy on March 30-31, 2016 at the Boston Public Health Commission (BPHC). Known asthma exacerbation risk factors are upper respiratory virus transmission, particularly in school-age children, harsh or extreme weather conditions, and poor air quality. Meteorological subject matter experts described availability and usage of data sources representing these risk factors. Modelers presented multiple analytic approaches including mechanistic models, machine learning approaches, simulation techniques, and hybrids. Health department staff and local partners discussed surveillance operations, constraints, and operational system requirements. Attendees valued the direct exchange of information among public health practitioners, system designers, and modelers. Discussion finalized design of an 8-year de-identified dataset of Boston ED patient records for modeling partners who sign a standard data use agreement.

Analysis of available, potentially large data streams could have value in assisting risk identification efforts. We explored data from the American Society for the Prevention of Cruelty to Animals (ASPCA) Animal Poison Control Center hotline to identify weekly increases in: 1) call volume, 2) calls regarding certain intoxication products, 3) calls regarding certain species, and 4) calls with clinical signs associated with specific syndromes. Seasonal trends were observed in calls regarding pesticide intoxications. Human-centric product intoxication trends were observed in swine. These data support the value of passive data streams in risk identification, biosurveillance, and animal health monitoring.

A collaboration of veterinary epidemiologists, laboratorians, and statistical analysts has designed a laboratory-based health surveillance system for horses in Colorado. Initial efforts focused on 12 years of data from three state labs. Exploratory analysis, stakeholder input, and discovery of laboratory workflow details were applied to refine syndrome groups and filter test order records to eliminate alerting bias. To tailor alerting methods for each syndrome, we constructed an algorithm testbed and a stochastic injection process based on target signal types chosen by the epidemiologists based on known disease progression. Testbed results yielded recommended methods, settings and thresholds for each syndrome.

Epidemiological modeling for infectious disease is important for disease management and its routine implementation needs to be facilitated through better description of models in an operational context. A standardized model characterization process that allows selection or making manual comparisons of available models and their results is currently lacking. A key need is a universal framework to facilitate model description and understanding of its features. Los Alamos National Laboratory (LANL) has developed a comprehensive framework that can be used to characterize an infectious disease model in an operational context. The framework was developed through a consensus among a panel of subject matter experts. In this paper, we describe the framework, its application to model characterization, and the development of the Biosurveillance Analytics Resource Directory (BARD; http://brd.bsvgateway.org/brd/), to facilitate the rapid selection of operational models for specific infectious/communicable diseases. We offer this framework and associated database to stakeholders of the infectious disease modeling field as a tool for standardizing model description and facilitating the use of epidemiological models.

Biosurveillance systems designed and deployed in the United States and abroad to allow public health authorities to monitor the health of their communities have significant design limitations despite their wide usage. One limitation is the lack of algorithmic solutions to combine local data sources for regional situation awareness. The objective of the current study is to develop and demonstrate the value of automated information fusion methods applied to the distributed neighboring public health sites. A prototype system consisting of distributed Bayesian models was designed to enable informed regional and local cognitive decision support response. The Intelligent Decision Support Network (IDSN) is composed of Bayesian Information Fusion Models (BIFMs) that target a particular syndrome or disease type. Using local data from county health departments in Northern Virginia for the time period between August 2005 and May 2007, we estimated the probability of a gastrointestinal (GI) outbreak in two ways: First, based on data from the local hospitals only; and second, based on the relative probability of outbreak by combining local hospital data and probabilities of GI events from the neighboring counties' BIFMs. Preliminary findings showed that the network of distributed models detected events that would be undetected without multi-jurisdictional data.

Background: Automated surveillance systems require statistical methods to recognize increases in visit counts that might indicate an outbreak. In prior work we presented methods to enhance the sensitivity of C2, a commonly used time series method. In this study, we compared the enhanced C2 method with five regression models. Methods: We used emergency department chief complaint data from US CDC BioSense surveillance system, aggregated by city (total of 206 hospitals, 16 cities) during 5/2008-4/2009. Data for six syndromes (asthma, gastrointestinal, nausea and vomiting, rash, respiratory, and influenza-like illness) was used and was stratified by mean count (1-19, 20-49, >=50 per day) into 14 syndrome-count categories. We compared the sensitivity for detecting single-day artificially-added increases in syndrome counts. Four modifications of the C2 time series method, and five regression models (two linear and three Poisson), were tested. A constant alert rate of 1% was used for all methods. Results: Among the regression models tested, we found that a Poisson model controlling for the logarithm of total visits (i.e., visits both meeting and not meeting a syndrome definition), day of week, and 14-day time period was best. Among 14 syndrome-count categories, time series and regression methods produced approximately the same sensitivity (

Algorithms for identifying public health threats or disease outbreaks are vulnerable to false alarms arising from sudden shifts in health-care utilization or data participation. This paper describes a method of reducing false alerts in automated public health surveillance algorithms, and in particular, automated syndromic surveillance algorithms, that rely on health-care utilization data. The technique is based on monitoring syndromic counts with reference to a suitable background, or reference, series of counts. The suitability of the background time series in decreasing the false-alarm rate will be shown to be related mathematically to the so-called mutual information that exists between the random variables representing the syndromic and background time series of counts. The method can be understood as a noise cancellation filter technique in which one noisy (reference) channel is used to cancel the background noise of the monitored (measured) channel. The issues discussed here may also be relevant to the appropriate use of rates in epidemiology and biostatistics. Copyright © 2011 John Wiley & Sons, Ltd.

Objective: We evaluated emergency department (ED) data, emergency medical services (EMS) data, and public utilities data for describing an outbreak of carbon monoxide (CO) poisoning following a windstorm.Methods: Syndromic ED data were matched against previously collected chart abstraction data. We ran detection algorithms on selected time series derived from all 3 data sources to identify health events associated with the CO poisoning outbreak. We used spatial and spatiotemporal scan statistics to identify geographic areas that were most heavily affected by the CO poisoning event.Results: Of the 241 CO cases confirmed by chart review, 190 (78.8%) were identified in the syndromic surveillance data as exact matches. Records from the ED and EMS data detected an increase in CO-consistent syndromes after the storm. The ED data identified significant clusters of CO-consistent syndromes, including zip codes that had widespread power outages. Weak temporal gastrointestinal (GI) signals, possibly resulting from ingestion of food spoiled by lack of refrigeration, were detected in the ED data but not in the EMS data. Spatial clustering of GI-based groupings in the ED data was not detected.Conclusions: Data from this evaluation support the value of ED data for surveillance after natural disasters. Enhanced EMS data may be useful for monitoring a CO poisoning event, if these data are available to the health department promptly.(Disaster Med Public Health Preparedness. 2011;5:37-45)

Dr Fricker’s paper addresses the practical, multidisciplinary problem of biosurveillance. This field has seen numerous books and journal articles in the past 10 years and is still evolving as a science. Based on his own reading, research, and contacts with those in public health practice (PHP), the author presents his view of the status and needed direction of this field. The remarks below are intended to augment this view from another perspective. Perspective: Following a background in mathematics and 20 years of developing and evaluating detection systems in non-health-related disciplines, I have worked on biosurveillance systems since 2000 while acquiring knowledge in epidemiology and biostatistics related to public health. The recent years have been devoted to the design, development, and evaluation of ESSENCE biosurveillance systems and to consulting for the U.S. Centers for Disease Control and Prevention, now my half-time employer. As a member since 2005 of the Board of Directors of the International Society for Disease Surveillance (ISDS), my role has been liaison to the ISDS Research Committee, which has an international roster of over 130 public health practitioners and academic, government, and industrial researchers. Dr Fricker’s remarks about the need for cross-disciplinary collaboration resonated with efforts over the past 2 years between the ISDS Research and PHP committees to promote research efforts with near-term PHP utility. These efforts have spawned several webinar and conference sessions, most notably a widely advertised and attended event entitled ‘Technical Challenges from the Public Health Practice Community’, whose recording and related products are freely available [1]. The remarks below are heavily influenced by the observations of PHP members from these events and by project-related contacts.

This paper describes the problem of public health monitoring for waterborne disease outbreaks using disparate evidence from health surveillance data streams and environmental sensors. We present a combined monitoring approach along with examples from a recent project at the Johns Hopkins University Applied Physics Laboratory in collaboration with the U.S. Environmental Protection Agency. The project objective was to build a module for the Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE) to include water quality data with health indicator data for the early detection of waterborne disease outbreaks. The basic question in the fused surveillance application is 'What is the likelihood of the public health threat of interest given recent information from available sources of evidence?' For a scientific perspective, we formulate this question in terms of the estimation of positive predictive value customary in classical epidemiology, and we present a solution framework using Bayesian Networks (BN). An overview of the BN approach presents advantages, disadvantages, and required adaptations needed for a fused surveillance capability that is scalable and robust relative to the practical data environment. In the BN project, we built a top-level health/water-quality fusion BN informed by separate waterborne-disease-related networks for the detection of water contamination and human health effects. Elements of the art of developing networks appropriate to this environment are discussed with examples. Results of applying these networks to a simulated contamination scenario are presented.

This paper discusses further advances in making robust predictions with the Holt-Winters forecasts for a variety of syndromic time series behaviors and introduces a control-chart detection approach based on these forecasts. Using three collections of time series data, we compare biosurveillance alerting methods with quantified measures of forecast agreement, signal sensitivity, and time-to-detect. The study presents practical rules for initialization and parameterization of biosurveillance time series. Several outbreak scenarios are used for detection comparison. We derive an alerting algorithm from forecasts using Holt-Winters-generalized smoothing for prospective application to daily syndromic time series. The derived algorithm is compared with simple control-chart adaptations and to more computationally intensive regression modeling methods. The comparisons are conducted on background data from both authentic and simulated data streams. Both types of background data include time series that vary widely by both mean value and cyclic or seasonal behavior. Plausible, simulated signals are added to the background data for detection performance testing at signal strengths calculated to be neither too easy nor too hard to separate the compared methods. Results show that both the sensitivity and the timeliness of the Holt-Winters-based algorithm proved to be comparable or superior to that of the more traditional prediction methods used for syndromic surveillance.

Methods based on scan statistics are widely used in health-related applications to detect clusters of disease. The most common methods are based on the Bernoulli and Poisson models. Kulldorff (1997) derived the likelihood ratio test statistic for his scan method for both of these models. His scan statistic is widely used with freely available software, SaTScan™ (see Kulldorff, 2005). We provide an alternative derivation of the likelihood ratio test statistic in the Poisson case. Our derivation is simpler and more general in the sense that it applies when the incidences are not aggregated into subregional counts.

For robust detection performance, traditional control chart monitoring for biosurveillance is based on input data free of trends, day-of-week effects, and other systematic behaviour. Time series forecasting methods may be used to remove this behaviour by subtracting forecasts from observations to form residuals for algorithmic input. We describe three forecast methods and compare their predictive accuracy on each of 16 authentic syndromic data streams. The methods are (1) a non-adaptive regression model using a long historical baseline, (2) an adaptive regression model with a shorter, sliding baseline, and (3) the Holt-Winters method for generalized exponential smoothing. Criteria for comparing the forecasts were the root-mean-square error, the median absolute per cent error (MedAPE), and the median absolute deviation. The median-based criteria showed best overall performance for the Holt-Winters method. The MedAPE measures over the 16 test series averaged 16.5, 11.6, and 9.7 for the non-adaptive regression, adaptive regression, and Holt-Winters methods, respectively. The non-adaptive regression forecasts were degraded by changes in the data behaviour in the fixed baseline period used to compute model coefficients. The mean-based criterion was less conclusive because of the effects of poor forecasts on a small number of calendar holidays. The Holt-Winters method was also most effective at removing serial autocorrelation, with most 1-day-lag autocorrelation coefficients below 0.15. The forecast methods were compared without tuning them to the behaviour of individual series. We achieved improved predictions with such tuning of the Holt-Winters method, but practical use of such improvements for routine surveillance will require reliable data classification methods.

Introduction: We document a funded effort to bridge the gap between constrained scientific challenges of public health surveillance and methodologies from academia and industry. Component tasks are the collection of epidemiologists’ use case problems, multidisciplinary consultancies to refine them, and dissemination of problem requirements and shareable datasets. We describe an initial use case and consultancy as a concrete example and challenge to developers. Materials and Methods: Supported by the Defense Threat Reduction Agency Biosurveillance Ecosystem project, the International Society for Disease Surveillance formed an advisory group to select tractable use case problems and convene inter-disciplinary consultancies to translate analytic needs into well-defined problems and to promote development of applicable solution methods. The initial consultancy’s focus was a problem originated by the North Carolina Department of Health and its NC DETECT surveillance system: Derive a method for detection of patient record clusters worthy of follow-up based on free-text chief complaints and without syndromic classification. Results: Direct communication between public health problem owners and analytic developers was informative to both groups and constructive for the solution development process. The consultancy achieved refinement of the asyndromic detection challenge and of solution requirements. Participants summarized and evaluated solution approaches and discussed dissemination and collaboration strategies.Conclusions: A solution meeting the specification of the use case described above could improve human monitoring efficiency with expedited warning of events requiring follow-up, including otherwise overlooked events with no syndromic indicators. This approach can remove obstacles to collaboration with efficient, minimal data-sharing and without costly overhead.

We provided emergency department providers with a real-time laboratory-based influenza surveillance tool, and evaluated the utility and acceptability of the surveillance information using provider surveys. The majority of emergency department providers found the surveillance data useful and indicated the additional information impacted their clinical decision making regarding influenza testing and treatment.

This panel will expand the development process and use case collection of the ISDS Technical Conventions Committee, whose purpose is to bridge the gap between the analytic needs of public health practitioners and the expertise of the research community for the enhancement of disease surveillance.Panellists will present use cases of current and ongoing concern at local, state, and national levels. The audience will be prompted with questions on the relevance, aspects of common utility, technical feasibility, and data availability issues related to each use case, and derived or novel ideas will be solicited.

As a part of the Korea-US joint Biosurveillance Portal project, we have developed the military syndromic surveillance system with electronic medical records from military hospitals for early identification of bio-terror related events among Armed Forces in Korea. Respiratory, Gastrointestinal, Botulism, Dermatologic, Neurologic, Hemorrhagic and Fever syndromes were defined by different ICD-10 codes and their alert thresholds were developed based on the characteristics of time series derived using daily counts of ICD-10 codes for each syndrome.

The session will explore past collaborations between the statistician panelists and public health departments to highlight approaches that have and have not been effective and to recommend effective, sustainable relationship strategies for mutual advancement of practical disease surveillance and relevant academic research. Panelists will describe experiences working with health departments, including actual applications as examples. Issues discussed will include requirements analysis, scoping technical problems for health department utility, adaptation of traditional statistical methods, and management of changing data environments. Panelists will derive advice for public health practitioners seeking help in forming relationships, framing problems, communicating results, and seeking funding.

This presentation compares surveillance algorithms used in the National Poison Data System to identify incidents of public health significance with recently expanded filtering capabilities and with methods beyond the NPDS generalized historical limits model. Collected data series from 55 poison centers over 7 years include hourly counts of general call volumes and of substance-specific (e.g. CO exposure) calls. By applying current, modified, and novel methods to known and simulated clusters among these data, the authors will present the most efficient algorithms for identifying incidents of public health significance.

In order to provide real-time access to influenza test results, we created a laboratory-based surveillance system which automatically uploaded influenza test results from a rapid PCR-based influenza test, Xpert Flu, and the associated testing times and locations. On-site, type-specific results were available to physicians and uploaded for public health awareness within 100 minutes of patient nasopharyngeal swab. Expansion of this real-time capability to sentinel facilities could improve both local and national surveillance and response, reducing the need for syndromic influenza surveillance.

Since 2012, the ISDS Research Committee (RC) has been using an updated literature search process to identify literature related to disease surveillance. These articles are primarily in the medical literature, but one-third are published in other disciplines. ISDS members may not have access to those articles published outside the health sciences, making the ISDS article repository a valuable resource for those conducting work in surveillance.

The presented method is a practical, understandable way to monitor single care facilities for chief complaint clusters of concern based on unusually high occurrence of rare or common terms that need not be related to syndromes. Routine implementation requires a human monitor to inspect the relevant CCs make follow-up decisions. Using 7 years of patient records from 15 hospitals, our approach pools CCs into contiguous time blocks and uses a statistical hypothesis test to seek current terms that are anomalous relative to their occurrence in a large sliding baseline. Sets of anomalous terms are then presented for further investigation.

The Electronic Surveillance System for the Early Notification of Community-Based Epidemics, or ESSENCE II, uses syndromic and nontraditional health information to provide very early warning of abnormal health conditions in the National Capital Area (NCA). ESSENCE II is being developed for the Department of Defense Global Emerging Infections System and is the only known system to combine both military and civilian health care information for daily outbreak surveillance. The National Capital Area has a complicated, multijurisdictional structure that makes data sharing and integrated regional surveillance challenging. However, the strong military presence in all jurisdictions facilitates the collection of health care information across the region. ESSENCE II integrates clinical and nonclinical human behavior indicators as a means of identifying the abnormality as close to the time of onset of symptoms as possible. Clinical data sets include emergency room syndromes, private practice billing codes grouped into syndromes, and veterinary syndromes. Nonclinical data include absenteeism, nurse hotline calls, prescription medications, and over-the-counter self-medications. Correctly using information marked by varying degrees of uncertainty is one of the more challenging as pects of this program. The data (without personal identifiers) are captured in an electronic format, encrypted, archived, and processed at a secure facility. Aggregated information is then provided to users on secure Web sites. When completed, the system will provide automated capture, archiving, processing, and notification of abnormalities to epidemiologists and analysts. Outbreak detection methods currently include temporal and spatial variations of odds ratios, autoregressive modeling, cumulative summation, matched filter, and scan statistics. Integration of nonuniform data is needed to increase sensitivity and thus enable the earliest notification possible. The performance of various detection techniques was compared using results obtained from the ESSENCE II system.