Search

The third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 17–18 October 2017 in London, England. ICASR is an interdisciplinary group whose goal is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. The group seeks to facilitate the development and widespread acceptance of automated techniques for systematic reviews. The meeting’s conclusion was that the most pressing needs at present are to develop approaches for validating currently available tools and to provide increased access to curated corpora that can be used for validation. To that end, ICASR’s short-term goals in 2018–2019 are to propose and publish protocols for key tasks in systematic reviews and to develop an approach for sharing curated corpora for validating the automation of the key tasks.

This study examines the impact of cefepime and APP-β (antipseudomonal penicillin/ β-lactamase inhibitor combinations) on Gram-negative bacterial colonization and resistance in two Australian ICUs. While resistance did not cumulatively increase, cefepime (but not APP-β treatment) was associated with acquisition of antibiotic resistant Enterobacteriaceae, consistent with an ecological effect. Analysis of the resident gut E. coli population in a subset of patients showed an increase in markers of horizontal gene transfer after cefepime exposure that helps explain the increase in APP-β resistance and reminds us that unmeasured impacts on the microbiome are key outcome determinants that need to be fully explored. Electronic supplementary material The online version of this article (10.1186/s40635-018-0185-2) contains supplementary material, which is available to authorized users.

Background Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools. Discussion We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see “others” in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments. Conclusion We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.

The second meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 3–4 October 2016 in Philadelphia, Pennsylvania, USA. ICASR is an interdisciplinary group whose aim is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. Having automated tools for systematic review should enable more transparent and timely review, maximizing the potential for identifying and translating research findings to practical application. The meeting brought together multiple stakeholder groups including users of summarized research, methodologists who explore production processes and systematic review quality, and technologists such as software developers, statisticians, and vendors. This diversity of participants was intended to ensure effective communication with numerous stakeholders about progress toward automation of systematic reviews and stimulate discussion about potential solutions to identified challenges. The meeting highlighted challenges, both simple and complex, and raised awareness among participants about ongoing efforts by various stakeholders. An outcome of this forum was to identify several short-term projects that participants felt would advance the automation of tasks in the systematic review workflow including (1) fostering better understanding about available tools, (2) developing validated datasets for testing new tools, (3) determining a standard method to facilitate interoperability of tools such as through an application programming interface or API, and (4) establishing criteria to evaluate the quality of tools’ output. ICASR 2016 provided a beneficial forum to foster focused discussion about tool development and resources and reconfirm ICASR members’ commitment toward systematic reviews’ automation.

Objectives To examine the use of supervised machine learning to identify biases in evidence selection and determine if citation information can predict favorable conclusions in reviews about neuraminidase inhibitors. Study Design and Setting Reviews of neuraminidase inhibitors published during January 2005 to May 2013 were identified by searching PubMed. In a blinded evaluation, the reviews were classified as favorable if investigators agreed that they supported the use of neuraminidase inhibitors for prophylaxis or treatment of influenza. Reference lists were used to identify all unique citations to primary articles. Three classification methods were tested for their ability to predict favorable conclusions using only citation information. Results Citations to 4,574 articles were identified in 152 reviews of neuraminidase inhibitors, and 93 (61%) of these reviews were graded as favorable. Primary articles describing drug resistance were among the citations that were underrepresented in favorable reviews. The most accurate classifier predicted favorable conclusions with 96.2% accuracy, using citations to only 24 of 4,574 articles. Conclusion Favorable conclusions in reviews about neuraminidase inhibitors can be predicted using only information about the articles they cite. The approach highlights how evidence exclusion shapes conclusions in reviews and provides a method to evaluate citation practices in a corpus of reviews.

Background Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%. Methods We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. Results The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%. Conclusions Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

Background Clinical quality indicators are used to monitor the performance of healthcare services and should wherever possible be based on research evidence. Little is known however about the extent to which indicators in common use are based on research. The objective of this study is to measure the extent to which clinical quality indicators used in asthma management in children with outcome measurements can be linked to results in randomised controlled clinical trial (RCT) reports. This work is part of a broader research program to trial methods that improve the efficiency and accuracy of indicator development. Methods National-level indicators for asthma management in children were extracted from the National Quality Measures Clearinghouse database and the National Institute for Health and Care Excellence quality standards by two independent appraisers. Outcome measures were extracted from all published English language RCT reports for asthma management in children below the age of 12 published between 2005 and 2014. The two sets were then linked by manually mapping both to a common set of Unified Medical Language System (UMLS) concepts. Results The analysis identified 39 indicators and 562 full text RCTs dealing with asthma management in children. About 95% (37/39) of the indicators could be linked to RCT outcome measures. Conclusions It is possible to identify relevant RCT reports for the majority of indicators used to assess the quality of asthma management in childhood. The methods reported here could be automated to more generally support assessment of candidate indicators against the research evidence. Electronic supplementary material The online version of this article (doi:10.1186/s12913-017-2324-y) contains supplementary material, which is available to authorized users.

While it is important for the evidence supporting practice guidelines to be current, that is often not the case. The advent of living systematic reviews has made the concept of "living guidelines" realistic, with the promise to provide timely, up-to-date and high-quality guidance to target users. We define living guidelines as an optimization of the guideline development process to allow updating individual recommendations as soon as new relevant evidence becomes available. A major implication of that definition is that the unit of update is the individual recommendation and not the whole guideline. We then discuss when living guidelines are appropriate, the workflows required to support them, the collaboration between living systematic reviews and living guideline teams, the thresholds for changing recommendations, and potential approaches to publication and dissemination. The success and sustainability of the concept of living guideline will depend on those of its major pillar, the living systematic review. We conclude that guideline developers should both experiment with and research the process of living guidelines.

Genome is the carrier of somatic mutations during the development of cancer. The catalogue of simple somatic mutations (SSM) is a subgroup of somatic mutations. It includes single base substitutions, small deletions and insertions of T substitutions at NpCpG trinucleotide contexts. Skin cancer showed distinct mutational spectra. Breast cancer, bladder cancer, and cervical cancer were found to have similar mutational spectra. Acute myeloid leukemia and lung cancer from South Korea, and colorectal cancer from China show high density of single base substitutions per mega base in chromosome Y. To sum up, our study and findings will be thought provoking in studying SSM in cancer.

Systematic reviews are difficult to keep up to date, but failure to do so leads to a decay in review currency, accuracy, and utility. We are developing a novel approach to systematic review updating termed "Living systematic review" (LSR): systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs may be particularly important in fields where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions. We hypothesize that a continual approach to updating will achieve greater currency and validity, and increase the benefits to end users, with feasible resource requirements over time.

Objective: Quality improvement of health care requires robust measurable indicators to track performance. However identifying which indicators are supported by strong clinical evidence, typically from clinical trials, is often laborious. This study tests a novel method for automatically linking indicators to clinical trial registrations. Design: A set of 522 quality of care indicators for 22 common conditions drawn from the CareTrack study were automatically mapped to outcome measures reported in 13 971 trials from ClinicalTrials.gov. Intervention: Text mining methods extracted phrases mentioning indicators and outcome phrases, and these were compared using the Levenshtein edit distance ratio to measure similarity. Main Outcome Measure: Number of care indicators that mapped to outcome measures in clinical trials. Results: While only 13% of the 522 CareTrack indicators were thought to have Level I or II evidence behind them, 353 (68%) could be directly linked to randomized controlled trials. Within these 522, 50 of 70 (71%) Level I and II evidence-based indicators, and 268 of 370 (72%) Level V (consensus-based) indicators could be linked to evidence. Of the indicators known to have evidence behind them, only 5.7% (4 of 70) were mentioned in the trial reports but were missed by our method. Conclusions: We automatically linked indicators to clinical trial registrations with high precision. Whilst the majority of quality indicators studied could be directly linked to research evidence, a small portion could not and these require closer scrutiny. It is feasible to support the process of indicator development using automated methods to identify research evidence. Refereed/Peer-reviewed

Display Omitted Implemented a generic rule-based method to extract epidemiological characteristics.Created environmental health exposure specific dictionaries.Precision ranging from 81% to 100% across six characteristics.Automated extraction is a feasible approach towards automation of systematic reviews. IntroductionMost data extraction efforts in epidemiology are focused on obtaining targeted information from clinical trials. In contrast, limited research has been conducted on the identification of information from observational studies, a major source for human evidence in many fields, including environmental health. The recognition of key epidemiological information (e.g., exposures) through text mining techniques can assist in the automation of systematic reviews and other evidence summaries. MethodWe designed and applied a knowledge-driven, rule-based approach to identify targeted information (study design, participant population, exposure, outcome, confounding factors, and the country where the study was conducted) from abstracts of epidemiological studies included in several systematic reviews of environmental health exposures. The rules were based on common syntactical patterns observed in text and are thus not specific to any systematic review. To validate the general applicability of our approach, we compared the data extracted using our approach versus hand curation for 35 epidemiological study abstracts manually selected for inclusion in two systematic reviews. ResultsThe returned F-score, precision, and recall ranged from 70% to 98%, 81% to 100%, and 54% to 97%, respectively. The highest precision was observed for exposure, outcome and population (100%) while recall was best for exposure and study design with 97% and 89%, respectively. The lowest recall was observed for the population (54%), which also had the lowest F-score (70%). ConclusionThe generated performance of our text-mining approach demonstrated encouraging results for the identification of targeted information from observational epidemiological study abstracts related to environmental exposures. We have demonstrated that rules based on generic syntactic patterns in one corpus can be applied to other observational study design by simple interchanging the dictionaries aiming to identify certain characteristics (i.e., outcomes, exposures). At the document level, the recognised information can assist in the selection and categorization of studies included in a systematic review.

The three-dimensional microstructure of minerals and materials can be visualised in a non-destructive manner using X-ray microtomography. The digitised nature of the tomographic image allows us to generate finite element models which precisely detail the material’s microstructure. With a high degree of automation, high resolution models can be created quickly and with little user interaction. The geometry is taken from the microtomographic data, and loads and boundary conditions are applied to the model to simulate various conditions. The finite element analysis results show the deformation and stress distribution in the material. The technique allows us to study the relationship between microstructure and bulk properties of porous minerals, to characterise them in terms of their strength and stiffness, and to simulate their behaviour under known loading conditions. In this paper we present an application of micro-finite element analysis in the study of porous minerals. Micro-finite element analysis can be used to study the behaviour of a variety of minerals, and is especially useful when applied to materials that have a distinct microstructure that affects their bulk properties.

SummaryBuilding a sustainable health system in the 21st Century will require the reinvention of much of the present day system, and the intelligent use of information and communication technologies (ICT) to deliver high quality, safe, efficient and affordable health care. The Centre for Health Informatics (CHI) is Australia’s largest academic research group in this emerging discipline.Our research is underpinned by a planning process, based on different future scenarios for the health system, which helps us identify longer-term problems needing a sustained research effort. A research competency matrix is used to ensure that the Centre has the requisite core capabilities in the research methods and tools needed to pursue our research program.The Centre’s work is internationally recognized for its contributions in the development of intelligent search systems to support evidence-based healthcare, developing evaluation methodologies for ICT, and in understanding how communication shapes the safety and quality of health care delivery. Centre researchers also are working on safety models and standards for ICT in healthcare, mining complex gene micro array, medical literature and medical record data, building health system simulation methods to model the impact of health policy changes, and developing novel computational methods to automate the diagnosis of 3-D medical images.Any individual research group like CHI must necessarily focus on a few areas to allow it to develop sufficient research capacity to make novel and internationally significant contributions. As CHI approaches the end of its first decade, it is becoming clear that developing capacity becomes increasingly challenging as the research territory changes under our feet, and that the Centre will continue to evolve and shift its focus in the years to come.

Ferromagnetic embolization hyperthermia (FEH) is a novel treatment for liver cancer. Magnetic microspheres are injected into the hepatic artery and cluster in the periphery of tumours and are heated with externally applied magnetic fields. In order to more accurately simulate FEH, we modelled a three-dimensional heterogeneous distribution of heat sources. We constructed a fractal model of the vasculature in the periphery of a tumour. We used this model to compute the spatial distribution of the microspheres that lodge in capillaries. We used the distribution model as input to a finite-element heat transfer model of the FEH treatment. The overall appearance of the vascular tree is subjectively similar to that of the disorganized vascular network which encapsulates tumours. The microspheres are distributed in the tumour periphery in similar patterns to experimental observations. We expect the vasculature and microsphere deposition models to also be of interest to researchers of any targeted cancer therapies such as localized intra-arterial chemotherapy and selective internal radiotherapy. Our results show that heterogeneous microsphere distributions give significantly different results to those for a homogeneous model and thus are preferable when accurate results are required.

Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects. We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.

Background and Objectives Reports of randomized controlled trials (RCTs) should set findings within the context of previous research. The resulting network of citations would also provide an alternative search method for clinicians, researchers, and systematic reviewers seeking to base decisions on all available evidence. We sought to determine the connectedness of citation networks of RCTs by examining direct (referenced trials) and indirect (through references of referenced trials, etc) citation of trials to one another. Methods Meta-analyses were used to create citation networks of RCTs addressing the same clinical questions. The primary measure was the proportion of networks where following citation links between RCTs identifies the complete set of RCTs, forming a single connected citation group. Other measures included the number of disconnected groups (islands) within each network, the number of citations in the network relative to the maximum possible, and the maximum number of links in the path between two connected trials (a measure of indirectness of citations). Results We included 259 meta-analyses with a total of 2,413 and a median of seven RCTs each. For 46% (118 of 259) of networks, the RCTs formed a single connected citation group—one island. For the other 54% of networks, where at least one RCT group was not cited by others, 39% had two citation islands and 4% (10 of 257) had 10 or more islands. On average, the citation networks had 38% of the possible citations to other trials (if each trial had cited all earlier trials). The number of citation islands and the maximum number of citation links increased with increasing numbers of trials in the network. Conclusion Available evidence to answer a clinical question may be identified by using network citations created with a small initial corpus of eligible trials. However, the number of islands means that citation networks cannot be relied on for evidence retrieval.

Cancer is a heterogeneous disease caused, in part, by genetic and epigenetic alterations. These changes have been explored in studies of the pathogenesis of colorectal cancer (CRC) and have led to the identification of many biomarkers of disease progression. However, the number of biomarkers that have been incorporated into clinical practice is surprisingly small. We review the genetic and epigenetic mechanisms of colorectal cancer and discuss molecular markers recommended for use in early detection, screening, diagnosis, determination of prognosis, and prediction of treatment outcomes. We also review important areas for future research.

Introduction: Clinical quality indicators are necessary to monitor the performance of healthcare services. The development of indicators should, wherever possible, be based on research evidence to minimise the risk of bias which may be introduced during their development, because of logistic, ethical or financial constraints alone. The development of automated methods to identify the evidence base for candidate indicators should improve the process of indicator development. The objective of this study is to explore the relationship between clinical quality indicators for asthma management in children with outcome and process measurements extracted from randomised controlled clinical trial reports. Methods and analysis: National-level indicators for asthma management in children will be extracted from the National Quality Measures Clearinghouse (NQMC) database and the National Institute for Health and Care Excellence (NICE) quality standards. Outcome measures will be extracted from published English language randomised controlled trial (RCT) reports for asthma management in children aged below 12 years. The two sets of measures will be compared to assess any overlap. The study will provide insights into the relationship between clinical quality indicators and measurements in RCTs. This study will also yield a list of measurements used in RCTs for asthma management in children, and will find RCT evidence for indicators used in practice. Ethics and dissemination: Ethical approval is not necessary because this study will not include patient data. Findings will be disseminated through peer-reviewed publications. Refereed/Peer-reviewed

BACKGROUND Industry funding and financial conflicts of interest may contribute to bias in the synthesis and interpretation of scientific evidence. OBJECTIVE To examine the association between financial conflicts of interest and characteristics of systematic reviews of neuraminidase inhibitors. DESIGN Retrospective analysis. SETTING Reviews that examined the use of neuraminidase inhibitors in the prophylaxis or treatment of influenza, were published between January 2005 and May 2014, and used a systematic search protocol. MEASUREMENTS Two investigators blinded to all information regarding the review authors independently assessed the presentation of evidence on the use of neuraminidase inhibitors as favorable or not favorable. Financial conflicts of interest were identified using the index reviews, other publications, and Web-based searches. Associations between financial conflicts of interest, favorability assessments, and presence of critical appraisals of evidence quality were analyzed. RESULTS Twenty-six systematic reviews were identified, of which 13 examined prophylaxis and 24 examined treatment, accounting for 37 distinct assessments. Among assessments associated with a financial conflict of interest, 7 of 8 (88%) were classified as favorable, compared with 5 of 29 (17%) among those without a financial conflict of interest. Reviewers without financial conflicts of interest were more likely to include statements about the quality of the primary studies than those with financial conflicts of interest. LIMITATIONS The heterogeneity in populations and outcomes examined in the reviews precluded analysis of the contribution of selective inclusion of evidence on the discordance of the assessments made in the reviews. Many of the systematic reviews had overlapping authorship. CONCLUSION Reviewers with financial conflicts of interest may be more likely to present evidence about neuraminidase inhibitors in a favorable manner and recommend the use of these drugs than reviewers without financial conflicts of interest. PRIMARY FUNDING SOURCE Australian National Health and Medical Research Council.

Background In Evidence-Based Medicine, clinical practice guidelines and systematic reviews are crucial devices for medical practitioners in making clinical decision. Clinical practice guidelines are systematically developed statements to support health care decisions for specific circumstances whereas systematic reviews are summaries of evidence on clearly formulated clinical questions. Biomarkers are biological measurements (primarily molecular) that are used to diagnose, predict treatment outcomes and prognosticate disease and are increasingly used in randomized controlled trials (RCT). Methods We search PubMed for systematic reviews, RCTs, case reports and non-systematic reviews with and without mentions of biomarkers between years 1990–2011. We compared the frequency and growth rate of biomarkers and non-biomarkers publications. We also compared the growth of the proportion of biomarker-based RCTs with the growth of the proportion of biomarker-based systematic reviews. Results With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers. The annual growth rate of biomarkers publications is consistently higher than non-biomarkers publications, showing the growth in biomarkers research. From 20 years of systematic review publications indexed in PubMed, we identified a bias in systematic reviews against the inclusion of biomarker-based RCTs. Conclusions With the realisation of genome-based personalised medicine, biomarkers are becoming important for clinical decision making. The bias against the inclusion of biomarkers in systematic reviews leads to medical practitioners deprive of important information they require to address clinical questions. Sparse or weak evidence and lack of genetic training for systematic reviewers may contribute to this trend.

Summary Objectives To review the recent research literature in clinical decision support systems (CDSS). Methods A review of recent literature was undertaken, focussing on CDSS evaluation, consumers and public health, the impact of translational bioinformatics on CDSS design, and CDSS safety. Results In recent years, researchers have concentrated much less on the development of decision technologies, and have focussed more on the impact of CDSS in the clinical world. Recent work highlights that traditional process measures of CDSS effectiveness, such as document relevance are poor proxy measures for decision outcomes. Measuring the dynamics of decision making, for example via decision velocity, may produce a more accurate picture of effectiveness. Another trend is the broadening of user base for CDSS beyond front line clinicians. Consumers are now a major focus for biomedical informatics, as are public health officials, tasked with detecting and managing disease outbreaks at a health system, rather than individual patient level. Bioinformatics is also changing the nature of CDSS. Apart from personalisation of therapy recommendations, translational bioinformatics is creating new challenges in the interpretation of the meaning of genetic data. Finally, there is much recent interest in the safety and effectiveness of computerised physicianorderentry (CPOE) systems, given that prescribing and administration errors are a significant cause of morbidity and mortality. Of note, there is still much controversy surrounding the contention that poorly designed, implemented or used CDSS may actually lead to harm. Conclusions CDSS research remains an active and evolving area of research, as CDSS penetrate more widely beyond their traditional domain into consumer decision support, and as decisions become more complex, for example by involving sequence level genetic data.