Search

Systematic reviews are extremely time-consuming. The goal of this work is to assess work savings and recall for a publication type filtering strategy that uses the output of two machine learning models, Multi-Tagger and web RCT Tagger, applied retrospectively to 10 systematic reviews on drug effectiveness. Our filtering strategy resulted in mean work savings of 33.6% and recall of 98.3%. Of 363 articles finally included in any of the systematic reviews, 7 were filtered out by our strategy, but 1 "error" was actually an article using a publication type that the SR team had not pre-specified as relevant for inclusion. Our analysis suggests that automated publication type filtering can potentially provide substantial work savings with minimal loss of included articles. Publication type filtering should be personalized for each systematic review and might be combined with other filtering or ranking methods to provide additional work savings for manual triage.

Objective: Secondary use of electronic health record (EHR) data is enabled by accurate and complete retrieval of the relevant patient cohort, which requires searching both structured and unstructured data. Clinical text poses difficulties to searching, although chart notes incorporate structure that may facilitate accurate retrieval. Methods: We developed rules identifying clinical document sections, which can be indexed in search engines that allow faceted searches, such as Lucene or Essie, an NLM search engine. We developed 22 clinical cohorts and two queries for each cohort, one utilizing section headings and the other searching the whole document. We manually evaluated a subset of retrieved documents to compare query performance. Results: Querying by section had lower recall than whole-document queries (0.83 vs 0.95), higher precision (0.73 vs 0.54), and higher F1 (0.78 vs 0.69). Conclusion: This evaluation suggests that searching specific sections may improve precision under certain conditions and often with loss of recall.

A literature search to identify relevant studies is one of the first steps in performing a systematic review (SR) in support of evidence-based medicine. To maximize efficiency, the search must find practically all relevant studies and retrieve few that are irrelevant; however, this level of precision is seldom attained. Therefore, many articles must be manually examined for relevance. To better understand the limitations of current search tools as applied to SR, we characterized the most common reasons that papers retrieved by SR searches were excluded from the review.The textual reasons given for retrieved but excluded articles were extracted from 6,743 SRs performed by 54 Cochrane Collaboration review groups. The frequencies of different exclusion reasons were analyzed, and we developed a taxonomy summarizing these reasons.Almost 65% of articles were excluded because the means of comparison were inappropriate. Of these, about 72% were due to the randomized controlled trial (RCT) design being required but not employed by the excluded study. Mismatching interventions and outcomes and incorrect population characteristics were also common reasons for exclusion.Currently available search methods do not adequately address the most common exclusion reasons for systematic review, even those based primarily on study design.

Just as translational medicine follows a long winding path from bench-to-bedside, so can Evidence-Based Medicine be envisioned as comprising a multi-step pipeline, from building evidence from raw data through synthesizing best practices and providing clinical decision support in a process described as the "evidence pyramid". 1 At one end, a heterogeneous mix of clinical and experimental studies including clinical trials, case reports, animal models and retrospective analyses are published as new knowledge. Then, experts collect and assess high-quality relevant evidence on specific issues and publish their conclusions (e.g., regarding efficacy and safety of treatments) as systematic reviews and meta-analyses. Finally, when an expert consensus has been reached, this must reach the attention of policy makers within the profession, the government and insurance companies, resulting in new practice guidelines and altered clinical practice within hospitals and clinics. At each stage, this process requires a large investment of time and effort from many individuals with a wide range of expertise. Our panel will discuss the variety of innovative approaches that are being taken by different informatics research groups to improve each step within the evidence based medicine pipeline. These approaches are, in part, devoted to making existing data collection and synthesis practices faster and more efficient, but they also involve re-imagining and re-engineering the processes by which evidence is accumulated, evaluated and applied.

Quality measurement has been slow to make a major impact in health care. Initial measures were too simple to affect outcomes of importance. Incentive programs such as Meaningful Use encourage better measures, but in process may become more complex. We evaluated the measures selected for Meaningful Use in two ways: we counted unique concept identifiers, taxonomies, and aggregated concepts as measures of complexity; and we surveyed informatics professionals to assess difficulty. There were 20,316 unique concept identifiers, 35 taxonomies, and 317 aggregated concepts across the 45 measures. Half the respondents reported measures at least moderately difficult. The number of identifiers was associated with fewer implementations (r=-.37); rating-of-difficulty was associated with more taxonomies (r=.24). The impact on accuracy may be substantial when moving to measures intended to be more relevant to clinical outcomes but requiring the use of more taxonomies, unused structured concept identifiers, or concepts only in free text fields.

To determine whether a factorized version of the complement naïve Bayes (FCNB) classifier can reduce the time spent by experts reviewing journal articles for inclusion in systematic reviews of drug class efficacy for disease treatment.The proposed classifier was evaluated on a test collection built from 15 systematic drug class reviews used in previous work. The FCNB classifier was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Weight engineering (WE) techniques were added to reduce underestimation for Medical Subject Headings (MeSH)-based and Publication Type (PubType)-based features. Cross-validation experiments were performed to evaluate the classifier's parameters and performance.Work saved over sampling (WSS) at no less than a 95% recall was used as the main measure of performance.The minimum workload reduction for a systematic review for one topic, achieved with a FCNB/WE classifier, was 8.5%; the maximum was 62.2% and the average over the 15 topics was 33.5%. This is 15.0% higher than the average workload reduction obtained using a voting perceptron-based automated citation classification system.The FCNB/WE classifier is simple, easy to implement, and produces significantly better results in reducing the workload than previously achieved. The results support it being a useful algorithm for machine-learning-based automation of systematic reviews of drug class efficacy for disease treatment.