Search

Background Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods Researchers manually annotated 70 items of information (‘entities’) in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the ‘FLAIR’ framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results The F1 evaluation score, derived from the false positive and false negative rates (range 0–1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05–0.88) compared with an average human annotator’s score of 0.75 (SD=0.15, range 0.38–1.00). The algorithm for assigning entities to study arms (e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper 1 ).

Reviewing complex interventions is challenging because they include many elements that can interact dynamically in a nonlinear manner. A systems perspective offers a way of thinking to help understand complex issues, but its application in evidence synthesis is not established. The aim of this project was to understand how and why systems perspectives have been applied in evidence synthesis. A methodological mapping review was conducted to identify papers using a systems perspective in evidence synthesis. A search was conducted in seven bibliographic databases and three search engines. A total of 101 papers (representing 98 reviews) met the eligibility criteria. Two categories of reviews were identified: (1) reviews using a "systems lens" to frame the topic, generate hypotheses, select studies, and guide the analysis and interpretation of findings (n = 76) and (2) reviews using systems methods to develop a systems model (n = 22). Several methods (e.g., systems dynamic modeling, soft systems approach) were identified, and they were used to identify, rank and select elements, analyze interactions, develop models, and forecast needs. The main reasons for using a systems perspective were to address complexity, view the problem as a whole, and understand the interrelationships between the elements. Several challenges for capturing the true nature and complexity of a problem were raised when performing these methods. This review is a useful starting point when designing evidence synthesis of complex interventions. It identifies different opportunities for applying a systems perspective in evidence synthesis, and highlights both commonplace and less familiar methods.

Background Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods Researchers manually annotated 70 items of information (‘entities’) in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the ‘FLAIR’ framework. The manually annotated intervention, population, setting and study entities were used to develop a deep-learning algorithm using multiple layers of long-short-term-memory (LSTM) components to predict smoking cessation outcomes. Results The F1 evaluation score, derived from the false positive and false negative rates (range 0-1), for the information extraction algorithm averaged 0.42 across different types of entity (SD=0.22, range 0.05-0.88) compared with an average human annotator’s score of 0.75 (SD=0.15, range 0.38-1.00). The algorithm for assigning entities to study arms (e.g., intervention or control) was not successful. This initial ML outcome prediction algorithm did not outperform prediction based just on the mean outcome value or a linear regression model. Conclusions While some success was achieved in using ML to extract information from reports of randomised trials of smoking cessation interventions, we identified major challenges that could be addressed by greater standardisation in the way that studies are reported. Outcome prediction from smoking cessation studies may benefit from development of novel algorithms, e.g., using ontological information to inform ML (as reported in the linked paper (1)).

This paper critically explores how survey and routinely collected data could aid in assessing the generalisability of public health evidence. We propose developing approaches that could be employed in understanding the relevance of public health evidence, and investigate ways of producing meta-analytic estimates tailored to reflect local circumstances, based on analyses of secondary data. Currently, public health decision makers face challenges in interpreting "global" review evidence to assess its meaning in local contexts. A lack of clarity on the definition and scope of generalisability, and the absence of consensus on its measurement, has stunted methodological progress. The consequence of failing to tackle generalisability means that systematic review evidence often fails to fulfil its potential contribution in public health decision making. Three approaches to address these problems are considered and emerging challenges discussed: (1) purposeful exploration after a review has been conducted, and we present a framework of potential avenues of enquiry and a worked example; (2) recalibration of the results to weight studies differentially based on their similarity to conditions in an inference population, and we provide a worked example using UK Census data to understand potential differences in the effectiveness of community engagement interventions among sites in England and Wales; (3) purposeful exploration before starting a review to ensure that the findings are relevant to an inference population. The paper aims to demonstrate how a more nuanced treatment of context in reviews of public health interventions could be achieved through greater engagement with existing large sources of secondary data.

Using text mining to aid the development of database search strings for topics described by diverse terminology has potential benefits for systematic reviews; however, methods and tools for accomplishing this are poorly covered in the research methods literature. We briefly review the literature on applications of text mining for search term development for systematic reviewing. We found that the tools can be used in 5 overarching ways: improving the precision of searches; identifying search terms to improve search sensitivity; aiding the translation of search strategies across databases; searching and screening within an integrated system; and developing objectively derived search strategies. Using a case study and selected examples, we then reflect on the utility of certain technologies (term frequency--inverse document frequency and Termine, term frequency, and clustering) in improving the precision and sensitivity of searches. Challenges in using these tools are discussed. The utility of these tools is influenced by the different capabilities of the tools, the way the tools are used, and the text that is analysed. Increased awareness of how the tools perform facilitates the further development of methods for their use in systematic reviews.

Background: This study describes an approach for the use of a specific type of qualitative evidence synthesis in the matrix approach, a mixed studies reviewing method. The matrix approach compares quantitative and qualitative data on the review level by juxtaposing concrete recommendations from the qualitative evidence synthesis against interventions in primary quantitative studies. However, types of qualitative evidence syntheses that are associated with theory building generate theoretical models instead of recommendations. Therefore, the output from these types of qualitative evidence syntheses cannot directly be used for the matrix approach but requires transformation. This approach allows for the transformation of these types of output. Method: The approach enables the inference of moderation effects instead of direct effects from the theoretical model developed in a qualitative evidence synthesis. Recommendations for practice are formulated on the basis of interactional relations inferred from the qualitative evidence synthesis. In doing so, we apply the realist perspective to model variables from the qualitative evidence synthesis according to the context-mechanism-outcome configuration. Findings: A worked example shows that it is possible to identify recommendations from a theory-building qualitative evidence synthesis using the realist perspective. We created subsets of the interventions from primary quantitative studies based on whether they matched the recommendations or not and compared the weighted mean effect sizes of the subsets. The comparison shows a slight difference in effect sizes between the groups of studies. The study concludes that the approach enhances the applicability of the matrix approach.

Using Toulmin's argumentation theory, we analysed the texts of systematic reviews in the area of workplace health promotion to explore differences in the modes of reasoning embedded in reports of narrative synthesis as compared with reports of meta-analysis. We used framework synthesis, grounded theory and cross-case analysis methods to analyse 85 systematic reviews addressing intervention effectiveness in workplace health promotion. Two core categories, or 'modes of reasoning', emerged to frame the contrast between narrative synthesis and meta-analysis: practical-configurational reasoning in narrative synthesis ('what is going on here? What picture emerges?') and inferential-predictive reasoning in meta-analysis ('does it work, and how well? Will it work again?'). Modes of reasoning examined quality and consistency of the included evidence differently. Meta-analyses clearly distinguished between warrant and claim, whereas narrative syntheses often presented joint warrant-claims. Narrative syntheses and meta-analyses represent different modes of reasoning. Systematic reviewers are likely to be addressing research questions in different ways with each method. It is important to consider narrative synthesis in its own right as a method and to develop specific quality criteria and understandings of how it is carried out, not merely as a complement to, or second-best option for, meta-analysis.

Background: School closures have been a recommended non-pharmaceutical intervention in pandemic response owing to the potential to reduce transmission of infection between children, school staff and those that they contact. However, given the many roles that schools play in society, closure for any extended period is likely to have additional impacts. Literature reviews of research exploring school closure to date have focused upon epidemiological effects; there is an unmet need for research that considers the multiplicity of potential impacts of school closures. Methods: We used systematic searching, coding and synthesis techniques to develop a systems-based logic model. We included literature related to school closure planned in response to epidemics large and small, spanning the 1918-19 ‘flu pandemic through to the emerging literature on the 2019 novel coronavirus. We used over 170 research studies and a number of policy documents to inform our model. Results: The model organises the concepts used by authors into seven higher level domains: children’s health and wellbeing, children’s education, impacts on teachers and other school staff, the school organisation, considerations for parents and families, public health considerations, and broader economic impacts. The model also collates ideas about potential moderating factors and ethical considerations. While dependent upon the nature of epidemics experienced to date, we aim for the model to provide a starting point for theorising about school closures in general, and as part of a wider system that is influenced by contextual and population factors. Conclusions: The model highlights that the impacts of school closures are much broader than those related solely to health, and demonstrates that there is a need for further concerted work in this area. The publication of this logic model should help to frame future research in this area and aid decision-makers when considering future school closure policy and possible mitigation strategies.

Abstract Background Comparisons between narrative synthesis and meta-analysis as synthesis methods in systematic reviews are uncommon within the same systematic review. We re-analysed a systematic review on the effects of plain packaging of tobacco on attractiveness. We sought to compare different synthesis approaches within the same systematic review and shed light on the comparative benefits of each approach. Methods In our re-analysis, we included results relating to attractiveness in included reports. We extracted findings from studies and converted all estimates of differences in attractiveness to Cohen’s d. We used multilevel meta-analysis to account for clustering of effect sizes within studies. Results Of the 19 studies reporting results on attractiveness, seven studies that included between-subjects analyses could be included in the meta-analysis. Plain packs were less attractive than branded packs (d = − 0.59, 95% CI [− 0.71, − 0.47]), with negligible but uncertain between-studies heterogeneity (I 2 = 0%, 95% CI [0.00, 70.81]) and high within-study heterogeneity (I 2 = 92.6%, 95% CI [91.04, 93.90]). Conclusions The meta-analysis found, similar to the narrative synthesis, that respondents typically rated plain packaging as less attractive than alternative (e.g. branded) tobacco packs. However, there were several trade-offs between analysis methods in the types and bodies of evidence each one contained and in the difference between partial precision and breadth of conclusions. Analysis methods were different in respect of the role of judgement and contextual variation and in terms of estimation and unexpected effect modification. In addition, we noted that analysis methods were different in how they accounted for heterogeneity and consistency.

Abstract Background Government policy increasingly supports engaging communities to promote health. It is critical to consider whether such strategies are effective, for whom, and under what circumstances. However, ‘community engagement’ is defined in diverse ways and employed for different reasons. Considering the theory and context we developed a conceptual framework which informs understanding about what makes an effective (or ineffective) community engagement intervention. Methods We conducted a systematic review of community engagement in public health interventions using: stakeholder involvement; searching, screening, appraisal and coding of research literature; and iterative thematic syntheses and meta-analysis. A conceptual framework of community engagement was refined, following interactions between the framework and each review stage. Results From 335 included reports, three products emerged: (1) two strong theoretical ‘meta-narratives’: one, concerning the theory and practice of empowerment/engagement as an independent objective; and a more utilitarian perspective optimally configuring health services to achieve defined outcomes. These informed (2) models that were operationalized in subsequent meta-analysis. Both refined (3) the final conceptual framework. This identified multiple dimensions by which community engagement interventions may differ. Diverse combinations of intervention purpose, theory and implementation were noted, including: ways of defining communities and health needs; initial motivations for community engagement; types of participation; conditions and actions necessary for engagement; and potential issues influencing impact. Some dimensions consistently co-occurred, leading to three overarching models of effective engagement which either: utilised peer-led delivery; employed varying degrees of collaboration between communities and health services; or built on empowerment philosophies. Conclusions Our conceptual framework and models are useful tools for considering appropriate and effective approaches to community engagement. These should be tested and adapted to facilitate intervention design and evaluation. Using this framework may disentangle the relative effectiveness of different models of community engagement, promoting effective, sustainable and appropriate initiatives.

Abstract Background Behaviour change is key to addressing both the challenges facing human health and wellbeing and to promoting the uptake of research findings in health policy and practice. We need to make better use of the vast amount of accumulating evidence from behaviour change intervention (BCI) evaluations and promote the uptake of that evidence into a wide range of contexts. The scale and complexity of the task of synthesising and interpreting this evidence, and increasing evidence timeliness and accessibility, will require increased computer support. The Human Behaviour-Change Project (HBCP) will use Artificial Intelligence and Machine Learning to (i) develop and evaluate a ‘Knowledge System’ that automatically extracts, synthesises and interprets findings from BCI evaluation reports to generate new insights about behaviour change and improve prediction of intervention effectiveness and (ii) allow users, such as practitioners, policy makers and researchers, to easily and efficiently query the system to get answers to variants of the question ‘What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what settings and why?’. Methods The HBCP will: a) develop an ontology of BCI evaluations and their reports linking effect sizes for given target behaviours with intervention content and delivery and mechanisms of action, as moderated by exposure, populations and settings; b) develop and train an automated feature extraction system to annotate BCI evaluation reports using this ontology; c) develop and train machine learning and reasoning algorithms to use the annotated BCI evaluation reports to predict effect sizes for particular combinations of behaviours, interventions, populations and settings; d) build user and machine interfaces for interrogating and updating the knowledge base; and e) evaluate all the above in terms of performance and utility. Discussion The HBCP aims to revolutionise our ability to synthesise, interpret and deliver evidence on behaviour change interventions that is up-to-date and tailored to user need and context. This will enhance the usefulness, and support the implementation of, that evidence.

Background: The extraction of data from the reports of primary studies, on which the results of systematic reviews depend, needs to be carried out accurately. To aid reliability, it is recommended that two researchers carry out data extraction independently. The extraction of statistical data from graphs in PDF files is particularly challenging, as the process is usually completely manual, and reviewers need sometimes to revert to holding a ruler against the page to read off values: an inherently time-consuming and error-prone process. Methods: To mitigate some of the above problems we integrated and customised two existing JavaScript libraries to create a new web-based graphical data extraction tool to assist reviewers in extracting data from graphs. This tool aims to facilitate more accurate and timely data extraction through a user interface which can be used to extract data through mouse clicks. We carried out a non-inferiority evaluation to examine its performance in comparison with participants’ standard practice for extracting data from graphs in PDF documents. Results: We found that the customised graphical data extraction tool is not inferior to users’ (N=10) prior standard practice. Our study was not designed to show superiority, but suggests that, on average, participants saved around 6 minutes per graph using the new tool, accompanied by a substantial increase in accuracy. Conclusions: Our study suggests that the incorporation of this type of tool in online systematic review software would be beneficial in facilitating the production of accurate and timely evidence synthesis to improve decision-making.

Background: The extraction of data from the reports of primary studies, on which the results of systematic reviews depend, needs to be carried out accurately. To aid reliability, it is recommended that two researchers carry out data extraction independently. The extraction of statistical data from graphs in PDF files is particularly challenging, as the process is usually completely manual, and reviewers need sometimes to revert to holding a ruler against the page to read off values: an inherently time-consuming and error-prone process. Methods: To mitigate some of the above problems we integrated and customised two existing JavaScript libraries to create a new web-based graphical data extraction tool to assist reviewers in extracting data from graphs. This tool aims to facilitate more accurate and timely data extraction through a user interface which can be used to extract data through mouse clicks. We carried out a non-inferiority evaluation to examine its performance in comparison to standard practice. Results: We found that the customised graphical data extraction tool is not inferior to users’ prior preferred current approaches. Our study was not designed to show superiority, but suggests that there may be a saving in time of around 6 minutes per graph, accompanied by a substantial increase in accuracy. Conclusions: Our study suggests that the incorporation of this type of tool in online systematic review software would be beneficial in facilitating the production of accurate and timely evidence synthesis to improve decision-making.

Abstract Background Complex or heterogeneous data pose challenges for systematic review and meta-analysis. In recent years, a number of new methods have been developed to meet these challenges. This qualitative interview study aimed to understand researchers’ understanding of complexity and heterogeneity and the factors which may influence the choices researchers make in synthesising complex data. Methods We conducted interviews with a purposive sample of researchers (N = 19) working in systematic review or meta-analysis across a range of disciplines. We analysed data thematically using a framework approach. Results Participants reported using a broader range of methods and data types in complex reviews than in traditional reviews. A range of techniques are used to explore heterogeneity, but there is some debate about their validity, particularly when applied post hoc. Conclusions Technical considerations of how to synthesise complex evidence cannot be isolated from questions of the goals and contexts of research. However, decisions about how to analyse data appear to be made in a largely informal way, drawing on tacit expertise, and their relation to these broader questions remains unclear.

Objectives: To examine evidence from studies exploring the relationship between childhood obesity and educational attainment. Design: A systematic review of secondary analyses and observational studies published in English after 1997 examining attainment as measured by grade point average or other validated measure, in children aged 6 to 16 years, in high-income countries. Methods: Eleven databases from the fields of public health, education and social science were searched, along with 19 specialist registers and catalogues. Hand searching of relevant journals, contacting of experts and citation searching were undertaken. Two reviewers used standardized tools to independently carry out data extraction and assess the quality of included studies. Evidence was synthesized in a narrative summary. Results: Twenty-nine studies were identified for inclusion. Overall, the evidence suggested that higher weight is weakly associated with lower educational attainment among children and young people. Differences between average attainment of overweight and non-overweight children were marginal, with potentially negligible real-world implications for test scores. Limited evidence suggested that little variation in achievement was explained by weight status alone. Almost half the studies found that other factors, such as socioeconomic status, may better explain much of the negative association between obesity and attainment. Conclusion: Theoretical and methodological inconsistencies were evident both within and between many of the studies. As such, the results of the included studies must be interpreted with caution. If the negative association between obesity and attainment is accepted, it still remains doubtful whether obesity is exerting a socially important effect upon educational attainment.

Driven by necessity in our own complex review, we developed alternative systematic ways of identifying relevant evidence where the key concepts are generally not focal to the primary studies' aims and are found across multiple disciplines--that is, hard-to-detect evidence. Specifically, we sought to identify evidence on community engagement in public health interventions that aim to reduce health inequalities. Our initial search strategy used text mining to identify synonyms for the concept "community engagement". We conducted a systematic search for reviews on public health interventions, supplemented by searches of trials databases. We then used information in the reviews' evidence tables to gather more information about the included studies than was evident in the primary studies' own titles or abstracts. We identified 319 primary studies cited in reviews after full-text screening. In this paper, we retrospectively reflect on the challenges and benefits of the approach taken. We estimate that more than a quarter of the studies that were identified would have been missed by typical searching and screening methods. This identification strategy was highly effective and could be useful for reviews of broad research questions, or where the key concepts are unlikely to be the main focus of primary research.

Background: Community engagement has been advanced as a promising way of improving health and reducing health inequalities; however, the approach is not yet supported by a strong evidence base. Objectives: To undertake a multimethod systematic review which builds on the evidence that underpins the current UK guidance on community engagement; to identify theoretical models underpinning community engagement; to explore mechanisms and contexts through which communities are engaged; to identify community engagement approaches that are effective in reducing health inequalities, under what circumstances and for whom; and to determine the processes and costs associated with their implementation. Data sources: Databases including the Cochrane Database of Systematic Reviews (CDSR), The Campbell Library, the Database of Abstracts of Reviews of Effects (DARE), the Health Technology Assessment (HTA) database, the NHS Economic Evaluation Database (NHS EED) and EPPI-Centre’s Trials Register of Promoting Health Interventions (TRoPHI) and Database of Promoting Health Effectiveness Reviews (DoPHER) were searched from 1990 to August 2011 for systematic reviews and primary studies. Trials evaluating community engagement interventions reporting health outcomes were included. Review methods: Study eligibility criteria: published after 1990; outcome, economic, or process evaluation; intervention relevant to community engagement; written in English; measured and reported health or community outcomes, or presents cost, resource, or implementation data characterises study populations or reports differential impacts in terms of social determinants of health; conducted in an Organisation for Economic Co-operation and Development (OECD) country. Study appraisal: risk of bias for outcome evaluations; assessment of validity and relevance for process evaluations; comparison against an economic evaluation checklist for economic evaluations. Synthesis methods: four synthesis approaches were adopted for the different evidence types: theoretical, quantitative, process, and economic evidence. Results: The theoretical synthesis identified key models of community engagement that are underpinned by different theories of changes. Results from 131 studies included in a meta-analysis indicate that there is solid evidence that community engagement interventions have a positive impact on health behaviours, health consequences, self-efficacy and perceived social support outcomes, across various conditions. There is insufficient evidence – particularly for long-term outcomes and indirect beneficiaries – to determine whether one particular model of community engagement is likely to be more effective than any other. There are also insufficient data to test the effects on health inequalities, although there is some evidence to suggest that interventions that improve social inequalities (as measured by social support) also improve health behaviours. There is weak evidence from the effectiveness and process evaluations that certain implementation factors may affect intervention success. From the economic analysis, there is weak but inconsistent evidence that community engagement interventions are cost-effective. By combining findings across the syntheses, we produced a new conceptual framework. Limitations: Differences in the populations, intervention approaches and health outcomes made it difficult to pinpoint specific strategies for intervention effectiveness. The syntheses of process and economic evidence were limited by the small (generally not rigorous) evidence base. Conclusions: Community engagement interventions are effective across a wide range of contexts and using a variety of mechanisms. Public health initiatives should incorporate community engagement into intervention design. Evaluations should place greater emphasis on long-term outcomes, outcomes for indirect beneficiaries, process evaluation, and reporting costs and resources data. The theories of change identified and the newly developed conceptual framework are useful tools for researchers and practitioners. We identified trends in the evidence that could provide useful directions for future intervention design and evaluation. Funding: The National Institute for Health Research Public Health Research programme.

BACKGROUND AND OBJECTIVES:Standardised or 'plain' tobacco packaging was introduced in Australia in December 2012 and is currently being considered in other countries. The primary objective of this systematic review was to locate, assess and synthesise published and grey literature relating to the potential impacts of standardised tobacco packaging as proposed by the guidelines for the international Framework Convention on Tobacco Control: reduced appeal, increased salience and effectiveness of health warnings, and more accurate perceptions of product strength and harm. METHODS:Electronic databases were searched and researchers in the field were contacted to identify studies. Eligible studies were published or unpublished primary research of any design, issued since 1980 and concerning tobacco packaging. Twenty-five quantitative studies reported relevant outcomes and met the inclusion criteria. A narrative synthesis was conducted. RESULTS:Studies that explored the impact of package design on appeal consistently found that standardised packaging reduced the appeal of cigarettes and smoking, and was associated with perceived lower quality, poorer taste and less desirable smoker identities. Although findings were mixed, standardised packs tended to increase the salience and effectiveness of health warnings in terms of recall, attention, believability and seriousness, with effects being mediated by the warning size, type and position on pack. Pack colour was found to influence perceptions of product harm and strength, with darker coloured standardised packs generally perceived as containing stronger tasting and more harmful cigarettes than fully branded packs; lighter coloured standardised packs suggested weaker and less harmful cigarettes. Findings were largely consistent, irrespective of location and sample. CONCLUSIONS:The evidence strongly suggests that standardised packaging will reduce the appeal of packaging and of smoking in general; that it will go some way to reduce consumer misperceptions regarding product harm based upon package design; and will help make the legally required on-pack health warnings more salient.