Search

Abstract Background There is increasing awareness among researchers and policymakers of the potential for healthcare interventions to have consequences beyond those initially intended. These unintended consequences or “spillover effects” result from the complex features of healthcare organisation and delivery and can either increase or decrease overall effectiveness. Their potential influence has important consequences for the design and evaluation of implementation strategies and for decision-making. However, consideration of spillovers remains partial and unsystematic. We develop a comprehensive framework for the identification and measurement of spillover effects resulting from changes to the way in which healthcare services are organised and delivered. Methods We conducted a scoping review to map the existing literature on spillover effects in health and healthcare interventions and used the findings of this review to develop a comprehensive framework to identify and measure spillover effects. Results The scoping review identified a wide range of different spillover effects, either experienced by agents not intentionally targeted by an intervention or representing unintended effects for targeted agents. Our scoping review revealed that spillover effects tend to be discussed in papers only when they are found to be statistically significant or might account for unexpected findings, rather than as a pre-specified feature of evaluation studies. This hinders the ability to assess all potential implications of a given policy or intervention. We propose a taxonomy of spillover effects, classified based on the outcome and the unit experiencing the effect: within-unit, between-unit, and diagonal spillover effects. We then present the INTENTS framework: Intended Non-intended TargEted Non-Targeted Spillovers. The INTENTS framework considers the units and outcomes which may be affected by an intervention and the mechanisms by which spillover effects are generated. Conclusions The INTENTS framework provides a structured guide for researchers and policymakers when considering the potential effects that implementation strategies may generate, and the steps to take when designing and evaluating such interventions. Application of the INTENTS framework will enable spillover effects to be addressed appropriately in future evaluations and decision-making, ensuring that the full range of costs and benefits of interventions are correctly identified.

Abstract Background Systems approaches are currently being advocated and implemented to address complex challenges in Public Health. These approaches work by bringing multi-sectoral stakeholders together to develop a collective understanding of the system, and then to identify places where they can leverage change across the system. Systems approaches are unpredictable, where cause-and-effect cannot always be disentangled, and unintended consequences – positive and negative – frequently arise. Evaluating such approaches is difficult and new methods are warranted. Methods Ripple Effects Mapping (REM) is a qualitative method which can capture the wider impacts, and adaptive nature, of a systems approach. Using a case study example from the evaluation of a physical activity-orientated systems approach in Gloucestershire, we: a) introduce the adapted REM method; b) describe how REM was applied in the example; c) explain how REM outputs were analysed; d) provide examples of how REM outputs were used; and e) describe the strengths, limitations, and future uses of REM based on our reflections. Results Ripple Effects Mapping is a participatory method that requires the active input of programme stakeholders in data gathering workshops. It produces visual outputs (i.e., maps) of the programme activities and impacts, which are mapped along a timeline to understand the temporal dimension of systems change efforts. The REM outputs from our example were created over several iterations, with data collected every 3–4 months, to build a picture of activities and impacts that have continued or ceased. Workshops took place both in person and online. An inductive content analysis was undertaken to describe and quantify the patterns within the REM outputs. Detailed guidance related to the preparation, delivery, and analysis of REM are included in this paper. Conclusion REM may help to advance our understanding and evaluation of complex systems approaches, especially within the field of Public Health. We therefore invite other researchers, practitioners and policymakers to use REM and continuously evolve the method to enhance its application and practical utility.

Abstract Background Systems thinking embraces the complexity of public health problems, including childhood overweight and obesity. It aids in understanding how factors are interrelated, and it can be targeted to produce favourable changes in a system. There is a growing call for systems approaches in public health research, yet limited practical guidance is available on how to evaluate public health programmes within complex adaptive systems. The aim of this paper is to present an evaluation framework that supports researchers in designing systems evaluations in a comprehensive and practical way. Methods We searched the literature for existing public health systems evaluation studies. Key characteristics on how to conduct a systems evaluation were extracted and compared across studies. Next, we overlaid the identified characteristics to the context of the Lifestyle Innovations Based on Youth Knowledge and Experience (LIKE) programme evaluation and analyzed which characteristics were essential to carry out the LIKE evaluation. This resulted in the Evaluation of Programmes in Complex Adaptive Systems (ENCOMPASS) framework. Results The ENCOMPASS framework includes five iterative stages: (1) adopting a system dynamics perspective on the overall evaluation design; (2) defining the system boundaries; (3) understanding the pre-existing system to inform system changes; (4) monitoring dynamic programme output at different system levels; and (5) measuring programme outcome and impact in terms of system changes. Conclusions The value of ENCOMPASS lies in the integration of key characteristics from existing systems evaluation studies, as well as in its practical, applied focus. It can be employed in evaluating public health programmes in complex adaptive systems. Furthermore, ENCOMPASS provides guidance for the entire evaluation process, all the way from understanding the system to developing actions to change it and to measuring system changes. By the nature of systems thinking, the ENCOMPASS framework will likely evolve further over time, as the field expands with more completed studies.

Abstract Background Emergency obstetric care training, using Advances in Labour and Risk Management (ALARM) International Program (AIP) was implemented in Ukraine, a country with universal access to skilled perinatal and obstetric care but restricted resources. A total of 577 providers (65.5% of total) from 28 maternal clinics attended a 5-day training session focused on the five main causes of maternal mortality, with hands-on skill workshops, pre- and post- tests, and an objective structured clinical examination. The effects of this emergency obstetric care training on maternal outcomes is the subject of this paper. Methods A non-randomized controlled trial was conducted. The pilot areas where the training was implemented consisted of 64 maternity clinics of which 28 were considered as cases and 36 non-participating clinics were the referents. Data on maternal outcomes were collected for a 2-year span (2004-2005) prior to the trainings, which took place 2006-2007 and again after implementation of the trainings, from 2008 to 2009. Information was collected from 189,852 deliveries. Outcomes for the study were incidences of operative delivery and postpartum hemorrhage. Non-parametric statistics, meta-analyses, and difference in difference (DID) estimation were used to assess the effect of the AIP on maternal indices. Results DID analysis showed that after the training, compared to the referents, the cases had significant reduction of blood transfusions (OR: 0.56; 95%CI: 0.48-0.65), plasma transfusions (OR: 0.70; 95%CI: 0.63-0.78), and uterus explorations (OR: 0.64; 95%CI: 0.59-0.69). We observed a non-significant reduction of postpartum hemorrhage ≥1000 ml (OR: 0.92; 95%CI: 0.81-1.04; P = 0.103). Utilization of vacuum extraction for vaginal delivery increased (OR: 2.86; 95%CI: 1.80-4.57), as well as forceps assisted delivery (OR: 1.80; 95%CI: 1.00-3.25) and cesarean section (OR: 1.11; 95%CI: 1.06-1.17). There was no change in the occurrence of postpartum hysterectomy and maternal mortality. Conclusions After one week of Emergency Obstetrics Care training of the obstetric staff in a setting with universal access to perinatal and obstetric care but restricted resources, an association with the reduction of postpartum hemorrhage related interventions was observed. The effects on the use of vacuum extraction and cesarean section were minimal. Trial registration Retrospectively registered 071212007807 from 07/12/2012.

Abstract Background There is increased recognition in clinical settings of the importance of documenting, understanding, and addressing patients’ social determinants of health (SDOH) to improve health and address health inequities. This study evaluated a pilot of a standardized SDOH screening questionnaire and workflow in an ambulatory clinic within a large integrated health network in Northern California. Methods The pilot screened for SDOH needs using an 11-question Epic-compatible paper questionnaire assessing eight SDOH and health behavior domains: financial resource, transportation, stress, depression, intimate partner violence, social connections, physical activity, and alcohol consumption. Eligible patients for the pilot receiving a Medicare wellness, adult annual, or new patient visits during a five-week period (February-March, 2020), and a comparison group from the same time period in 2019 were identified. Sociodemographic data (age, sex, race/ethnicity, and payment type), visit type, length of visit, and responses to SDOH questions were extracted from electronic health records, and a staff experience survey was administered. The evaluation was guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. Results Two-hundred eighty-nine patients were eligible for SDOH screening. Responsiveness by domain ranged from 55 to 67%, except for depression. Half of patients had at least one identified social need, the most common being stress (33%), physical activity (22%), alcohol (12%), and social connections (6%). Physical activity needs were identified more in females (81% vs. 19% in males, p

Abstract Background There is a growing need for methods that acknowledge and successfully capture the dynamic interaction between context and implementation of complex interventions. Case study research has the potential to provide such understanding, enabling in-depth investigation of the particularities of phenomena. However, there is limited guidance on how and when to best use different case study research approaches when evaluating complex interventions. This study aimed to review and synthesise the literature on case study research across relevant disciplines, and determine relevance to the study of contextual influences on complex interventions in health systems and public health research. Methods Systematic meta-narrative review of the literature comprising (i) a scoping review of seminal texts (n = 60) on case study methodology and on context, complexity and interventions, (ii) detailed review of empirical literature on case study, context and complex interventions (n = 71), and (iii) identifying and reviewing ‘hybrid papers’ (n = 8) focused on the merits and challenges of case study in the evaluation of complex interventions. Results We identified four broad (and to some extent overlapping) research traditions, all using case study in a slightly different way and with different goals: 1) developing and testing complex interventions in healthcare; 2) analysing change in organisations; 3) undertaking realist evaluations; 4) studying complex change naturalistically. Each tradition conceptualised context differently—respectively as the backdrop to, or factors impacting on, the intervention; sets of interacting conditions and relationships; circumstances triggering intervention mechanisms; and socially structured practices. Overall, these traditions drew on a small number of case study methodologists and disciplines. Few studies problematised the nature and boundaries of ‘the case’ and ‘context’ or considered the implications of such conceptualisations for methods and knowledge production. Conclusions Case study research on complex interventions in healthcare draws on a number of different research traditions, each with different epistemological and methodological preferences. The approach used and consequences for knowledge produced often remains implicit. This has implications for how researchers, practitioners and decision makers understand, implement and evaluate complex interventions in different settings. Deeper engagement with case study research as a methodology is strongly recommended.

Abstract Background Increasingly, strength-based approaches to health and wellbeing interventions with Aboriginal and Torres Strait Islander Australians are being explored. This is a welcome counter to deficit-based initiatives which can represent a non-Indigenous view of outcomes of interest. However, the evidence base is not well developed. This paper presents the protocol for evaluating a strengths-based initiative which provides life coaching services to Aboriginal and Torres Strait Islander community housing tenants. The study aims to evaluate the effect of life coaching on social and emotional wellbeing (SEWB) of tenants in three Victorian regions. Methods The More Than a Landlord (MTAL) study is a prospective cohort study of Aboriginal Housing Victoria tenants aged 16 years and over that embeds the evaluation of a life coaching program. All tenant holders in one metropolitan and two regional areas of Victoria are invited to participate in a survey of SEWB, containing items consistent with key categories of SEWB as understood and defined by Aboriginal and Torres Strait Islander peoples, and key demographics, administered by Aboriginal and Torres Strait Islander peer researchers at baseline, 6 and 18 months. Survey participants are then invited to participate in strengths based life coaching, using the GROW model, for a duration of up to 18 months. Indigenous life coaches provide tenants with structured support in identifying and making progress towards their goals and aspirations, rather than needs. The study aims to recruit a minimum of 200 survey participants of which it is anticipated that approximately 73% will agree to life coaching. Discussion The MTAL study is a response to Aboriginal and Torres Strait Islander community and organisational requests to build the evidence base for an initiative originally developed and piloted within an Aboriginal controlled organisation. The study design aligns with key principles for research in Indigenous communities in promoting control, decision making and capacity building. The MTAL study will provide essential evidence to evaluate the effectiveness of strengths-based initiatives in promoting SEWB in these communities and provide new evidence about the relationship between strengths, resilience, self-determination and wellbeing outcomes. Trial registration This trial was retrospectively registered with the ISRCTN Register on the 12/7/21 with the study ID: ISRCTN33665735 .

Abstract Given their inherent complexity, we need a better understanding of what is happening inside the “black box” of population health interventions. The theory-driven intervention/evaluation paradigm is one approach to addressing this question. However, barriers related to semantic or practical issues stand in the way of its complete integration into evaluation designs. In this paper, we attempt to clarify how various theories, models and frameworks can contribute to developing a context-dependent theory, helping us to understand the black box of population health interventions and to acknowledge their complexity. To achieve this goal, we clarify what could be referred to as “theory” in the theory-driven evaluation of the interventional system, distinguishing it from other models, frameworks and classical theories. In order to evaluate the interventional system with a theory-driven paradigm, we put forward the concept of interventional system theory (ISyT), which combines a causal theory and an action model. We suggest that an ISyT could guide evaluation processes, whatever evaluation design is applied, and illustrate this alternative method through different examples of studies. We believe that such a clarification can help to promote the use of theories in complex intervention evaluations, and to identify ways of considering the transferability and scalability of interventions.

Abstract In this paper, we discuss the importance for faculty to become familiar with the general guidelines for collecting, assembling and preparing a tenure and promotion (T&P) application or dossier at a Primarily Undergraduate Institution (PUI) and the critical role that mentoring plays throughout the T&P process. While key elements of the application process such as submission timelines and documentation guidelines are usually outlined in the faculty handbook of the specific institution, many aspects of assembling the dossier are not necessarily detailed in writing anywhere. Instead, there are important elements of the T&P process that typically rely on institutional knowledge and guidance that is often communicated informally. Junior faculty who have limited access to “informal communications” are at a significant disadvantage when they go through the T&P process even when they show accomplishments in teaching effectiveness, research, and service. The problem is especially important for women and underrepresented minority faculty in STEM disciplines that are less well represented among senior faculty in STEM. Senior faculty often serve as informal or formal mentors to their less seasoned colleagues. The goal of this article is to help demystify the T&P process by offering practical suggestions and describing some of the specific materials and steps that are an important part of documenting the development of a faculty member at a PUI.

Abstract Background China has successfully reduced tuberculosis (TB) incidence rate over the past three decades, however, challenges remain in improving the quality of TB diagnosis and treatment. In this paper, we assess the effects of the implementation of “China National Health Commission (NHC) and Gates Foundation TB Prevention and Control Project” on the quality of TB care in the three provinces. Methods We conducted the baseline study in 2016 and the final evaluations in 2019 in the 12 selected project counties. We obtained TB patients’ information from the TB Information Management System and reviewed medical records of TB cases in the TB designated hospitals. We compared TB diagnosis and treatment services with the national practice guideline and used Student’s t-test and Pearson χ 2 tests or Fisher’s exact tests to compare the difference before and after the project implementation. Results The percentage of sputum smear-negative (SS–) patients taking culture or rapid molecular test (RMT) doubled between 2015 and 2018 (from 35% to 87%), and the percentage of bacteriologically confirmed pulmonary TB cases increased from 36% to 52%. RMT has been widely used and contributed an additional 20% of bacteriologically confirmed TB cases in 2018. The percentage of TB patients taking drug susceptibility tests (DST) also doubled (from 40% in 2015 to 82% in 2018), and the proportion of TB patients receiving adequate diagnosis services increased from 85% to 96%. Among all SS– TB patients, over 86% received the recommended diagnostic services at the end of the study period, an improvement from 75% prior to the project implementation. However, the proportion of TB patients treated irrationally using second-line anti-TB drugs (SLDs) increased from 12.6% in 2015 to 19.9% in 2018. The regional disparities remained within the project provinces, albeit the gaps between them narrowed down for almost all indicators. Conclusions The quality of TB diagnosis services has been improved substantially, which is attributable to the coverage of new diagnosis technology. However, irrational use of SLDs remains a concern after the project implementation. Graphic abstract

Abstract Introduction The last two decades have seen a shift in former Soviet countries from highly specialized to more family medicine-focused systems. Medical education has slowly adjusted to these reforms, although the region is still at risk to have a chronic shortage of family doctors. This paper presents the evaluation of a new post-graduate family medicine program in Tajikistan, focused on competency-based training. The findings are relevant for policy makers, international organizations and practitioners participating in similar medical education reform programs. Methods We employed a quasi-experimental control group design and compared intervention residents, control group residents with traditional training, and 1st year residents with no training in two outcomes, clinical knowledge and competencies. We employed two objective measures, a written multiple-choice question test (MCQT) and an Objective Structured Clinical Examination (OSCE), respectively. We report reliability and validity of the measures along with ANOVA, planned contrasts and effect size estimates to examine differences across groups. Results We found statistically significant differences in both clinical knowledge and competencies between intervention and control groups. We also detected a large intervention effect size. Participants in the intervention outperformed control group participants in the two measures. Our analysis suggests that intervention and control group participants are comparable in terms of initial knowledge and competencies, strengthening the argument that the intervention caused the improvement in the program outcomes. Discussion Receiving tailored training and structured opportunities to practice knowledge and competencies in clinical settings have a positive effect on the education of family medicine doctors in Tajikistan. Our results support curriculum reform and investment in medical education in the form of longer and supervised on-the-job preparation designed to be more in line with international standards. We discuss suggestions for future studies and potential requirements to inform replicability in other countries. Conclusion Family medicine is well recognized as central to health systems throughout the world, but high quality residency training lags behind in some countries. Our study showed that investing in family medicine residency programs and structured training is effective in increasing critical clinical competencies. We encourage promoting comprehensive post graduate family medicine doctor training so that the goals of a family medicine centered health system are attainable.

Abstract Background Health and medical research funding agencies are increasingly interested in measuring the impact of funded research. We present a research impact case study for the first four years of an Australian National Health and Medical Research Council funded Centre of Research Excellence in Cardiovascular Outcomes Improvement (2016–2020). The primary aim of this paper was to explore the application of a research impact matrix to assess the impact of cardiovascular outcomes improvement research. Methods We applied a research impact matrix developed from a systematic review of existing methodological frameworks used to measure research impact. This impact matrix was used as a bespoke tool to identify and understand various research impacts over different time frames. Data sources included a review of existing internal documentation from the research centre and publicly available information sources, informal iterative discussions with 10 centre investigators, and confirmation of information from centre grant and scholarship recipients. Results By July 2019, the impact on the short-term research domain category included over 41 direct publications, which were cited over 87 times (median journal impact factor of 2.84). There were over 61 conference presentations, seven PhD candidacies, five new academic collaborations, and six new database linkages conducted. The impact on the mid-term research domain category involved contributions towards the development of a national cardiac registry, cardiovascular guidelines, application for a Medicare Benefits Schedule reimbursement item number, introduction of patient-reported outcome measures into several databases, and the establishment of nine new industry collaborations. Evidence of long-term impacts were described as the development and use of contemporary management for aortic stenosis, a cardiovascular risk prediction model and prevention targets in several data registries, and the establishment of cost-effectiveness for stenting compared to surgery. Conclusions We considered the research impact matrix a feasible tool to identify evidence of academic and policy impact in the short- to midterm; however, we experienced challenges in capturing long-term impacts. Cost containment and broader economic impacts represented another difficult area of impact to measure.

Abstract Background Men’s Sheds (“Sheds”) offer a unique opportunity to reach a captive audience of “hard-to-reach” men. However, attempts to engage Sheds in structured health promotion programmes must respect the ethos of Sheds as highly variable, autonomous, non-structured spaces. This paper captures the key methodologies used in “Sheds for Life’ (SFL), a men’s health initiative tailored to the Shed setting. Methods A hybrid effectiveness-implementation study design is used to test effectiveness and implementation outcomes across multiple levels (participant, provider, organisational and systems levels). A dynamic, iterative and collaborative process seeks to address barriers and translation into the real world context. Using a community-based participatory research approach and guided by established implementation frameworks, Shed members (‘Shedders’) assume the role of key decision makers throughout the evaluation process to promote the systematic uptake of SFL across Shed settings. The protocols pertaining to the development, design and implementation of SFL and the evaluation of impact on participants’ health and wellbeing outcomes up to 12 months are outlined. Conclusions There is a dynamic interplay between the intervention characteristics of SFL and the need to assess and understand the diverse contexts of Sheds and the wider implementation environment. A pragmatic and context-specific design is therefore favoured over a tightly controlled efficacy trial. Documenting the protocols used to evaluate and implement a complex multi-level co-developed intervention such as SFL helps to inform gender-specific, community-based men’s health promotion and translational research more broadly. Trial registration This study has been retrospectively registered with the ‘International Standard Randomised Controlled Trial Number’ registry ( ISRCTN79921361 ) as of the 5th of March 2021.

Abstract Background The United Nations Children’s Fund (UNICEF) published their Health Systems Strengthening (HSS) approach to meet its strategic goals of ending preventable maternal, newborn and child deaths and promoting the health and development of all children and reducing inequities in health in 2016. UNICEF commissioned the University of Melbourne’s Nossal Institute for Global Health to develop and deliver a pilot blended HSS program, involving 60 hours of online learning and 2 weeks of face-to-face teaching over a 6-month period. To assess the extent to which the HSS program had built the first 83 UNICEF 2017 graduates’ capabilities to apply HSS actions by 2017, UNICEF funded an independent evaluator from the University of Melbourne. Methods A mixed-methods assessment was conducted using: online surveys of graduates at: enrolment, completion, 6 months post-HSS program; nine focus groups with graduates at face-to-face workshops; and interviews with purposive samples of UNICEF graduates and graduate Senior Managers 12 months post-HSS program. Results The HSS program content, structure and mode of delivery was positively received. Graduates reported increased confidence taking HSS actions and multiple changes in work practices (e.g., increased systems thinking and using of health system-based approaches). Graduates’ Senior Manager interviews revealed mixed impressions of graduates applying HSS actions, partly explained by the fit between the HSS program learnings and UNICEF’s workplace environment. Key contextual factors influencing graduates applying HSS actions included: workload; limited opportunities to apply HSS actions; limited HSS examples; and variable support to apply HSS actions. Graduate and Senior Manager suggestions to optimise applying HSS actions included: linking HSS program content with UNICEF priorities; increasing opportunities for graduates to apply HSS actions; increasing access to HSS support. Conclusions The paper concludes by presenting HSS program and assessment suggestions from the 2017 UNICEF Pilot HSS program assessment and actions taken for the 2018 UNICEF staff cohorts by HSS program developers, funders and beneficiaries.

Abstract Background Interpersonal and communication skills (ICS) are important core competencies in medical education and certification. In this study, we identified self- and simulated patient (SP)-reported ratings of US first-year medical students’ ICS and the influence of age and gender on performance appraisal during the Objective-Structured Clinical Examination (OSCE). Methods OSCE participants, including 172 first-year medical students and 15 SPs were asked to evaluate the students’ ICS using the American Board of Internal Medicine–Patient-Satisfaction Questionnaire (ABIM–PSQ), electronically and via paper, respectively. Self- and SP-reported ratings of students’ ICS were presented as the median on a 5-point Likert-scale and as three categories defined as “good,” “very good,” and “inadequate.” Results SPs assessed all 172 students in the OSCE, while 43.6% of students assessed their own performance. The majority of students and SPs evaluated the students’ ICS as very good. 23.3% of SPs and 5.3% of students rated the medical students’ ability to encourage patient question-asking and answer questions as inadequate (P

Abstract Background Third-generation single molecule sequencing technologies can sequence long reads, which is advancing the frontiers of genomics research. However, their high error rates prohibit accurate and efficient downstream analysis. This difficulty has motivated the development of many long read error correction tools, which tackle this problem through sampling redundancy and/or leveraging accurate short reads of the same biological samples. Existing studies to asses these tools use simulated data sets, and are not sufficiently comprehensive in the range of software covered or diversity of evaluation measures used. Results In this paper, we present a categorization and review of long read error correction methods, and provide a comprehensive evaluation of the corresponding long read error correction tools. Leveraging recent real sequencing data, we establish benchmark data sets and set up evaluation criteria for a comparative assessment which includes quality of error correction as well as run-time and memory usage. We study how trimming and long read sequencing depth affect error correction in terms of length distribution and genome coverage post-correction, and the impact of error correction performance on an important application of long reads, genome assembly. We provide guidelines for practitioners for choosing among the available error correction tools and identify directions for future research. Conclusions Despite the high error rate of long reads, the state-of-the-art correction tools can achieve high correction quality. When short reads are available, the best hybrid methods outperform non-hybrid methods in terms of correction quality and computing resource usage. When choosing tools for use, practitioners are suggested to be careful with a few correction tools that discard reads, and check the effect of error correction tools on downstream analysis. Our evaluation code is available as open-source at https://github.com/haowenz/LRECE .

Abstract Background Despite an increasing number of training opportunities in implementation science becoming available, the demand for training amongst researchers and practitioners is unmet. To address this training shortfall, we developed the King’s College London ‘Implementation Science Masterclass’ (ISM), an innovative 2-day programme (and currently the largest of its kind in Europe), developed and delivered by an international faculty of implementation experts. Methods This paper describes the ISM and provides delegates’ quantitative and qualitative evaluations (gathered through a survey at the end of the ISM) and faculty reflections over the period it has been running (2014–2019). Results Across the 6-year evaluation, a total of 501 delegates have attended the ISM, with numbers increasing yearly from 40 (in 2014) to 147 (in 2019). Delegates represent a diversity of backgrounds and 29 countries from across the world. The overall response rate for the delegate survey was 64.5% (323/501). Annually, the ISM has been rated ‘highly’ in terms of delegates’ overall impression (92%), clear and relevant learning objectives (90% and 94%, respectively), the course duration (85%), pace (86%) and academic level 87%), and the support provided on the day (92%). Seventy-one percent of delegates reported the ISM would have an impact on how they approached their future work. Qualitative feedback revealed key strengths include the opportunities to meet with an international and diverse pool of experts and individuals working in the field, the interactive nature of the workshops and training sessions, and the breadth of topics and contexts covered. Conclusions Yearly, the UK ISM has grown, both in size and in its international reach. Rated consistently favourably by delegates, the ISM helps to tackle current training demands from all those interested in learning and building their skills in implementation science. Evaluation of the ISM will continue to be an annual iterative process, reflective of changes in the evidence base and delegates changing needs as the field evolves.

ABSTRACT Background Health researchers and funders are increasingly consulting with stakeholders to set their research agendas but these activities are rarely evaluated. The Cochrane Consumers and Communication Group (CCCG) conducted a priority-setting project for systematic reviews in partnership with stakeholders (consumers/patients, health professionals, policy-makers and others). In this paper, we aim to describe our evaluation of the project’s processes and outcomes. Methods We used a 10-element conceptual framework designed to evaluate processes (e.g. stakeholder engagement, use of explicit process) and outcomes (e.g. improved decision-making quality, stakeholder acceptance and understanding) of health priority-setting. Data sources included empirical data (feedback surveys, project documents and CCCG editorial policies) and CCCG staff reflections. Data were analysed using content analysis. Results The project met three and partially met two of the process elements, for example, by engaging key stakeholders throughout the project and using pre-determined and transparent methods that offered multiple and meaningful ways to contribute. The project met three and partially met two of the outcome elements. Stakeholders were satisfied with and accepted the process and an additional six Cochrane Review titles aligned with stakeholder priorities are now being conducted in partnership with stakeholders. The project has also directly influenced the editorial work of CCCG, for example, by shifting its organisational focus towards coproduction, and indirectly influenced the work of Cochrane’s prioritisation and coproduction activities. Some areas were identified as having room for improvement, for example, there was low participation by people from diverse backgrounds, stakeholders could contribute to most but not all project stages, and there was no formal way for stakeholders to appeal decisions at project end. In the 3 years since its completion, the Cochrane Reviews are nearing completion but none of the reviews have been published. Conclusion We demonstrated that our priority-setting methods were broadly in line with best practice and the project resulted in many positive outcomes beyond just identifying the top priorities for research. Our evaluation framework and recommendations for future evaluations may be of use to priority-setting researchers planning similar activities.

Abstract Background Complex global initiatives, like the Global Polio Eradication Initiative (GPEI), have prevented millions of paralyses and improved the health status of diverse populations. Despite the logistical challenges these initiatives must overcome at several levels, scant methods exist for systematically identifying and reaching a range of actors involved in their implementation. As a result, efforts to document the lessons learned from such initiatives are often incomplete. This paper describes the development and application of the Synthesis and Translation of Research and Innovations from Polio Eradication (STRIPE) systematic approach for identifying a comprehensive sample of actors involved in the GPEI. Results The survey for collecting lessons learned from the GPEI was conducted at the global level and within seven countries that represented GPEI operational contexts. Standard organizational and operational levels, as well as goals of program activities, were defined across contexts. Each survey iteration followed similar methodologies to theorize a target population or “universe” of all polio-related actors in the study area, enumerate a source population of specific individuals within the target population, and administer the survey to individuals within the source population. Based on the systematic approach used to obtain a comprehensive sample for lessons learned in GPEI, steps for obtaining a comprehensive sample for studying complex initiatives can be summarized as follows: (i) State research goal(s); (ii) Describe the program of interest; (iii) Define a sampling universe to meet these criteria; (iv) Estimate the size of the sampling universe; (v) Enumerate a source population within the universe that can be feasibly reached for sampling; (vi) Sample from the source population; and (vii) Reflect on the process to determine strength of inferences drawn. Conclusions The application of these methods can inform future evaluations of complex public health initiatives, resulting in better adoption of lessons learned, ultimately improving efficacy and efficiency, and resulting in significant health gains. Their use to administer the STRIPE lessons learned survey reflects experiences related to implementation challenges and strategies used to overcome barriers from actors across an extensive range of organizational, programming, and contextual settings.

Abstract Background Efforts to engage patients as partners in health research have grown and thereby the need for feedback and evaluation. In this pilot evaluation study, we aimed to 1) evaluate patient engagement in health research projects in Newfoundland and Labrador, Canada, and 2) learn more about how to best monitor and evaluate patient engagement. This paper presents the results of our participatory evaluation study and the lessons learned. The evaluation of the projects was driven by questions patients wanted answered. Methods We conducted a formative evaluation of patient engagement in health research projects. Projects spanned a variety of topics, target groups, research designs and methods of patient engagement. Participants included principal investigators (n = 6) and their patient partners (n = 14). Furthermore, graduate students (n = 13) working on their own research projects participated. Participants completed an online survey with closed and open-ended questions about their patient engagement efforts, experiences and preliminary outcomes. Patients were involved as co-investigators in the entire evaluation study. We used qualitative methods to evaluate our participatory process. Results The evaluation study results show that most patients and researchers felt prepared and worked together in various phases of the research process. Both groups felt that the insights and comments of patients influenced research decisions. They believed that patient engagement improved the quality and uptake of research. Students felt less prepared and were less satisfied with their patient engagement experience compared to researchers and their patient partners. Involvement of patient co-investigators in this evaluation resulted in learnings, transparency, validation of findings and increased applicability. Challenges were to select evaluation questions relevant to all stakeholders and to adapt evaluation tools to local needs. Conclusions Our findings show that researchers, patient partners and students value patient engagement in health research. Capacity building at the supervisor level in academic institutions is needed to better support students. Sufficient time is also needed to permit observable outcomes. Participatory evaluation may increase the relevance and usefulness of information, but it also raises issues such as who defines and designs the content of evaluation tools. A co-creation process is required to develop appropriate monitoring and evaluation strategies.

Abstract Background This research aims to explore an identified gap in implementation science methodology, that is, how to assess context in implementation research. Context is among the strongest influences on implementation success but is a construct that is poorly understood and reported within the literature. Consequently, there is little guidance on how to research context. This study addresses this issue by developing a method to account for the active role of context during implementation research. Through use of a case study, this paper demonstrates the value of using our context coding framework. Methods The developed context coding framework was guided by the sub-elements of the Consolidated Framework for Implementation Research (CFIR). Employing a constructivist approach, this framework builds on the CFIR and enables a deeper exploration of context at multiple levels of the health system. The coding framework enables the collation of various data sources such as organisational reports, culture audits, interview, survey, and observational data. It may be continuously updated as new data emerge and can be adapted by researchers as required. A pre-existing rating criterion has been integrated to the context coding framework to highlight the influence and relative strength of each contextual factor prior to and during implementation. Results It is anticipated that the context coding framework will facilitate a standardised approach to assessing context. This will provide a deeper understanding of how to account for the influence of context, ultimately providing guidance that should increase the likelihood of implementation success. The coding framework enables implementation progress to be monitored, facilitating the identification of contextual changes and variations across settings at different levels of the healthcare system. It is expected this framework will inform the selection of appropriate implementation strategies and enable the monitoring of such strategies regarding their impact on local context. Conclusions This research contributes to the extant literature by advancing methodologies for the consideration and assessment of context in implementation research. This context coding framework may be used in any setting to provide insight into the characteristics of particular contexts throughout implementation processes.

Abstract Background Health practitioners from different professions, and with differing competencies, need to collaborate to provide quality care. Competencies in interprofessional working need developing in undergraduate educational preparation. This paper reports the protocol for a systematic review of self-report instruments to assess interprofessional learning in undergraduate health professionals’ education. Methods We will search PubMed, Web of Science, CINAHL and ERIC from January 2010 onwards. A combination of search terms for interprofessional learning, health professions, psychometric properties, assessment of learning and assessment tools will be used. Two reviewers will independently screen all titles, abstracts and full-texts. Potential conflicts will be resolved through discussion. Quantitative and mixed-methods studies evaluating interprofessional learning in undergraduate health professions education (e.g. medicine, nursing, occupational and physical therapy, pharmacy and psychology) will be included. Methodological quality of each reported instrument, underpinning theoretical frameworks, and the effects of reported interventions will be assessed. The overall outcome will be the effectiveness of instruments used to assess interprofessional competence. Primary outcomes will be the psychometric properties (e.g. reliability, discriminant and internal validity) of instruments used. Secondary outcomes will include time from intervention to assessment, how items relate to specific performance/competencies (or general abstract constructs) and how scores are used (e.g. to grade students, to improve courses or research purposes). Quantitative summaries in tabular format and a narrative synthesis will allow recommendations to be made on the use of self-report instruments in practice. Discussion Many studies use self-report questionnaires as tools for developing meaningful interprofessional education activities and assessing students’ interprofessional competence. This systematic review will evaluate both the benefits and limitations of reported instruments and help educators and researchers (i) choose the most appropriate existing self-report instruments to assess interprofessional competence and (ii) inform the design and conduct of interprofessional competency assessment using self-report instruments. Systematic review registration Open Science Framework [ https://osf.io/vrfjn ].

Abstract Background The aim of this paper is to present the development of the Feel4Diabetes Healthy Diet Score and to evaluate its clinical validity. Methods Study population consisted of 3268 adults (63% women) from high diabetes risk families living in 6 European countries. Participants filled in questionnaires at baseline and after 1 year, reflecting the dietary goals of the Feel4Diabetes intervention. Based on these questions the Healthy Diet Score was constructed, consisting of the following components: breakfast, vegetables, fruit and berries, sugary drinks, whole-grain cereals, nuts and seeds, low-fat dairy products, oils and fats, red meat, sweet snacks, salty snacks, and family meals. Maximum score for each component was set based on its estimated relative importance regarding T2DM risk, higher score indicating better quality of diet. Clinical measurements included height, weight, waist circumference, heart rate, blood pressure, and fasting blood sampling, with analyses of glucose, total cholesterol, HDL-cholesterol, LDL-cholesterol, and triglycerides. Analysis of (co) variance was used to compare the Healthy Diet Score and its components between countries and sexes using baseline data, and to test differences in clinical characteristics between score categories, adjusted for age, sex and country. Pearson’s correlations were used to study the association between changes from baseline to year 1 in the Healthy Diet Score and clinical markers. To estimate reproducibility, Pearson’s correlations were studied between baseline and 1 year score, within the control group only. Results The mean total score was 52.8 ± 12.8 among women and 46.6 ± 12.8 among men (p

Abstract Background Despite numerous campaigns and interventions to end female genital mutilation (FGM), the practice persists across the world, including the European Union (EU). Previous interventions have focused mainly on awareness raising and legislation aimed at criminalizing the practice. Limited evidence exists on the effectiveness of interventions due in part to the lack of systematic evaluation of projects. This paper presents an evaluation of the REPLACE Approach, which is a new methodology for tackling FGM based on community-based behaviour change and intervention evaluation. Methods We developed, trialed and evaluated the REPLACE Approach through extensive engagement with eight FGM affected African diaspora communities in five EU countries. We employed qualitative and quantitative tools to obtain data to inform the development, implementation and evaluation of the Approach. These included community-based participatory action research, questionnaires and community readiness assessments. The research took place between 2010 and 2016. Results Findings suggested that the Approach has the capability for building the capacities of FGM affected communities to overturn social norms that perpetuate the practice. We observed that community-based action research is a useful methodology for collecting data in FGM intervention settings as it allows for effective community engagement to identify, educate and motivate influential community members to challenge the practice, as well as obtaining useful information on the beliefs and norms that shape the practice. We also found that community readiness assessments, pre and post intervention, were useful for tailoring interventions appropriately and for evaluating changes in attitudes and behaviour that may have resulted from the interventions. Conclusion This evaluation has demonstrated that the REPLACE Approach has the potential, over time, to bring about changes in norms and attitudes associated with FGM. Its strengths lay in the engagement with influential community members, in building the capacity and motivation of community members to undertake change, in recognising contextual differences in the barriers and enablers of FGM practice and in tailoring interventions to local community readiness to change, and then evaluating interventions to re-inform implementation. The next steps would therefore be to implement the Approach over a longer time frame to assess if it results in measurable change in behaviour.

Abstract Background A novel project sustainability framework is used to evaluate 65 off-grid solar photovoltaic (PV) energy system projects in Malawi. This study addresses PV projects serving rural public facilities, a solution known to have had historical issues with poor sustainability. A recent countrywide program targeting such facilities was evaluated against existing projects to determine whether this latest iteration offered better results. Method Sustainability is defined at the project-level with metrics justified under the main themes of technical, economic, social, and organizational. Data captured for each project is based on a custom survey and interview of key stakeholders. Projects are grouped according to age, project implementer, income level, and PV system size to compare relative results. Results The sustainability outlook for each project is evaluated. We find sustainability for most projects to be low. Social sustainability was weak with many projects due to low local community stakeholder engagement in terms of numbers of stakeholders, meeting frequency, and community contributions. Organizational challenges included a lack of key management positions in place and extremely limited training provisions. Furthermore, the evaluation highlights the economic health of the system to be key to sustainability, with the “healthiest” projects affording only 37% of expected operations and maintenance costs. Relative to expected demand, systems were found to be undersized for both panels (53% of required) and batteries (57%). Users reported achieving only 60% of their desired consumption. Poor sizing standards related to the lack of load profiles for first-time electricity users as well as poor quality components reduced the overall technical sustainability. Conclusion Rural public facilities with solar PV in Malawi are not well served by isolated community management. Improved sustainability requires the establishment of a robust financial framework prior to project development that includes formal local government support. This paper discusses ideal management frameworks and their ultimate implications for project developers, policy makers, and the research community.

Abstract Background Helpline services have become an increasingly popular mode of providing community access to information and expert information and advice in the health and welfare sector. This paper reports on data collected from 908 callers to UK-based breastfeeding helplines. Methods A mixed methods design was adopted utilising a structured interview schedule to elicit callers experiences of the help and support received. In this paper we report on a series of multiple regression models undertaken to elicit the variables associated with callers’ ‘overall satisfaction’ with the helpline service. Three models were constructed; 1) caller demographic/call characteristics; 2) attitudes and effectiveness of service characteristics and 3) impact of support on caller wellbeing. Results Overall, 74.6% of callers were very satisfied, and 19.8% were satisfied with the help and support received by the helpline service. The caller demographic/call characteristics found to have a significant relationship with overall satisfaction related to the ease of getting through to the helpline and whether the woman had previously breastfed. Service characteristics associated with overall satisfaction related to whether the information received was helpful and whether the support helped to resolve their issues. The extent to which the volunteer was perceived to have enough time, whether the support had encouraged them to continue breastfeeding, met the caller’s expectations and/or provided the support the caller needed were also significantly associated. Caller outcomes contributing significantly to overall satisfaction concerned callers feeling less stressed, more confident, reassured and determined to continue breastfeeding following the call. Consideration of the effect sizes indicated that key factors associated with overall satisfaction related to: volunteers having sufficient time to deal with the callers’ issues; the information being perceived as helpful; the volunteers providing the support the callers needed; and for callers to feel reassured following the call. Conclusion Overall, these results highlight the value of the breastfeeding helpline(s) in terms of providing rapid, targeted, realistic, practical, and responsive support that provides affirmation and encouragement. The benefits include confidence building and callers feeling reassured and motivated to continue breastfeeding. Care needs to be taken to ensure that helpline support is easily accessible to ensure that callers and their families can access support when needed. This may require consideration of extension to a 24 hour service.

Abstract Background An innovative program, the Practice Support Program (PSP), for full-service family physicians and their medical office assistants in primary care practices was recently introduced in British Columbia, Canada. The PSP was jointly approved by both government and physician groups, and is a dynamic, interactive, educational and supportive program that offers peer-to-peer training to physicians and their office staff. Topic areas range from clinical tools/skills to office management relevant to General Practitioner (GP) practices and “doable in real GP time”. PSP learning modules consist of three half-day learning sessions interspersed with 6–8 week action periods. At the end of the third learning session, all participants were asked to complete a pen-and-paper survey that asked them to rate (a) their satisfaction with the learning module components, including the content and (b) the perceived impact the learning has had on their practices and patients. Methods A total of 887 GPs (response rates ranging from 26.0% to 60.2% across three years) and 405 MOAs (response rates from 21.3% to 49.8%) provided responses on a pen-and-paper survey administered at the last learning session of the learning module. The survey asked respondents to rate (a) their satisfaction with the learning module components, including the content and (b) the perceived impact the learning has had on their practices and patients. The psychometric properties (Chronbach’s alphas) of the satisfaction and impact scales ranged from .82 to .94. Results Evaluation findings from the first three years of the PSP indicated consistently high satisfaction ratings and perceived impact on GP practices and patients, regardless of physician characteristics (gender, age group) or work-related variables (e.g., time worked in family practice). The Advanced Access Learning Module, which offers tools to improve office efficiencies, decreased wait times for urgent, regular and third next available appointments by an average of 1.2, 3.3, and by 3.4 days across all physicians. For the Chronic Disease Management module, over 87% of all GP respondents developed a CDM patient registry and reported being able to take better care of their patients. After attending the Adult Mental Health module: 94.1% of GPs agreed that they felt more comfortable helping patients who required mental health care; over 82% agreed that their skills and their confidence in diagnosing and treating mental health conditions had improved; and 41.0% agreed that their frequency of prescribing medications, if appropriate, had decreased. Additionally for the Adult Mental Health module, a 3–6 month follow-up survey of the GPs indicated that the implemented changes were sustained over time. Conclusion GP and medical office assistant participant ratings show that the PSP learning modules were consistently successful in providing GPs and their staff with new learning that was relevant and could be implemented and used in “real-GP-time”.

Abstract Background The provision of residential aged care is underpinned by information, and is reliant upon systems that adequately capture and effectively utilise and communicate this information. The aim of this study was to explicate and quantify the volume and method by which information is collected, exchanged within facilities and with external providers, and retrieved from facility information systems and hospitals. Methods A survey of staff (n = 119), including managers, health informatics officers (HIOs), quality improvement staff, registered nurses (RNs), enrolled nurses (ENs)/endorsed enrolled nurses (EENs) and assistants in nursing (AINs) was carried out in four residential aged care facilities in New South Wales and Victoria, Australia. Sites varied in size and displayed a range of information technology (IT) capabilities. The survey investigated how and by whom information is collected, retrieved and exchanged, and the frequency and amount of time devoted to these tasks. Descriptive analysis was performed using SPSS, and open responses to questions were coded into key themes. Results Staff completed a median of six forms each, taking a median of 30 min per shift. 68.8% of staff reported transferring information from paper to a computer system, which took a median of 30 min per shift. Handover and face-to-face communication was the most frequently used form of information exchange within facilities. There was a large amount of faxing and telephone communication between facility staff and General Practitioners and community pharmacists, with staff reporting sending a median of 2 faxes to pharmacy and 1.5 faxes to General Practitioners, and initiating 2 telephone calls to pharmacies and 1.5 calls to General Practitioners per shift. Only 38.5% of respondents reported that they always had information available at the point-of-care and only 35.4% of respondents reported that they always had access to hospital stay information of residents after hospital discharge. Conclusions This survey identified a high volume of information exchange activities, as well as inefficient procedures, such as the transfer of information from paper to computer systems and the reliance upon faxes for communication with external providers. These findings contribute to evidence for the need for interoperable IT systems to allow more efficient and reliable information exchange between facilities and external providers.

Abstract Background Cross-sector collaborative partnerships are a vital strategy in efforts to strengthen research-informed policy and practice and may be particularly effective at addressing the complex problems associated with chronic disease prevention. However, there is still a limited understanding of how such partnerships are implemented in practice and how their implementation contributes to outcomes. This paper explores the operationalisation and outcomes of knowledge mobilisation strategies within the Australian Prevention Partnership Centre — a research collaboration between policy-makers, practitioners and researchers. Methods The Centre’s programme model identifies six knowledge mobilisation strategies that are hypothesised to be essential for achieving its objectives. Using a mixed methods approach combining stakeholder interviews, surveys, participant feedback forms and routine process data over a 5-year period, we describe the structures, resources and activities used to operationalise these strategies and explore if and how they have contributed to proximal outcomes. Results Results showed that Centre-produced research, resources, tools and methods were impacting policy formation and funding. Policy-makers reported using new practical methodologies that were helping them to design, implement, evaluate and obtain funding for scaled-up policies and programmes, and co-creating compelling prevention narratives. Some strategies were better implemented and more impactful than others in supporting these outcomes, with variation in who they worked for. The activities used to effect engagement, capacity-building and partnership formation were mostly generating positive results, but co-production could be enhanced by greater shared decision-making. Considerably more work is needed to successfully operationalise knowledge integration and adaptive learning. Conclusions Describing how collaborative cross-sector research partnerships are operationalised in practice, and with what effects, can provide important insights into practical strategies for establishing and growing such partnerships and for maximising their contributions to policy. Findings suggest that the Centre has many strengths but could benefit from more inclusive and transparent governance and internal processes that facilitate dialogue about roles, expectations and co-production practices.

Abstract Background There has long been debate about the balance between fidelity to evidence-based interventions (EBIs) and the need for adaptation for specific contexts or particular patients. The debate is relevant to virtually all clinical areas. This paper synthesises arguments from both fidelity and adaptation perspectives to provide a comprehensive understanding of the challenges involved, and proposes a theoretical and practical approach for how fidelity and adaptation can optimally be managed. Discussion There are convincing arguments in support of both fidelity and adaptations, representing the perspectives of intervention developers and internal validity on the one hand and users and external validity on the other. Instead of characterizing fidelity and adaptation as mutually exclusive, we propose that they may better be conceptualized as complimentary, representing two synergistic perspectives that can increase the relevance of research, and provide a practical way to approach the goal of optimizing patient outcomes. The theoretical approach proposed, the “Value Equation,” provides a method for reconciling the fidelity and adaptation debate by putting it in relation to the value (V) that is produced. The equation involves three terms: intervention (IN), context (C), and implementation strategies (IS). Fidelity and adaptation determine how these terms are balanced and, in turn, the end product – the value it produces for patients, providers, organizations, and systems. The Value Equation summarizes three central propositions: 1) The end product of implementation efforts should emphasize overall value rather than only the intervention effects, 2) implementation strategies can be construed as a method to create fit between EBIs and context, and 3) transparency is vital; not only for the intervention but for all of the four terms of the equation. Summary There are merits to arguments for both fidelity and adaptation. We propose a theoretical approach, a Value Equation, to reconciling the fidelity and adaptation debate. Although there are complexities in the equation and the propositions, we suggest that the Value Equation be used in developing and testing hypotheses that can help implementation science move toward a more granular understanding of the roles of fidelity and adaptation in the implementation process, and ultimately sustainability of practices that provide value to stakeholders.

Abstract Background Communication between health care provider and patients in oncology presents challenges. Communication skills training have been frequently developed to address those. Given the complexity of communication training, the choice of outcomes and outcome measures to assess its effectiveness is important. The aim of this paper is to 1) perform a systematic review on outcomes and outcome measures used in evaluations of communication training, 2) discuss specific challenges and 3) provide recommendations for the selection of outcomes in future studies. Methods To identify studies and reviews reporting on the evaluation of communication training for health care professionals in oncology, we searched seven databases (Ovid MEDLINE, CENTRAL, CINAHL, EMBASE, PsychINFO, PsychARTICLES and Web of Science). We extracted outcomes assessed and the respective assessment methods. We held a two-day workshop with experts (n = 16) in communication theory, development and evaluation of generic or cancer-specific communication training and/or outcome measure development to identify and address challenges in the evaluation of communication training in oncology. After the workshop, participants contributed to the development of recommendations addressing those challenges. Results Out of 2181 references, we included 96 publications (33 RCTs, 2 RCT protocols, 4 controlled trials, 36 uncontrolled studies, 21 reviews) in the review. Most frequently used outcomes were participants’ training evaluation, their communication confidence, observed communication skills and patients’ overall satisfaction and anxiety. Outcomes were assessed using questionnaires for participants (57.3%), patients (36.0%) and observations of real (34.7%) and simulated (30.7%) patient encounters. Outcomes and outcome measures varied widely across studies. Experts agreed that outcomes need to be precisely defined and linked with explicit learning objectives of the training. Furthermore, outcomes should be assessed as broadly as possible on different levels (health care professional, patient and interaction level). Conclusions Measuring the effects of training programmes aimed at improving health care professionals’ communication skills presents considerable challenges. Outcomes as well as outcome measures differ widely across studies. We recommended to link outcome assessment to specific learning objectives and to assess outcomes as broadly as possible.

Abstract Background Trauma represents a major global health problem projected to increase in importance over the next decade. The majority of deaths occur in low- and middle-income countries (LMICs) where survival rates are lower than their high-income country (HIC) counterparts. Health system level changes in care for injured patients have been attributed to significant improvements in care quality and outcomes in HIC settings. There is a need for further research to assess trauma care health systems in LMICs to inform health system strengthening for the care of the injured. This study aims to conduct a narrative synthesis of a systematic search of the literature on the assessment of trauma care health systems in LMICs in order to inform the further development of trauma care health system assessment. Methods The review will include primary quantitative, qualitative or mixed method studies and secondary literature reviews. No restriction will be placed on language or date. Reports and publications identified from the grey literature including from relevant national and international health organisations will be included. Articles will be screened by two independent reviewers with a third reviewer resolving any persisting disagreement. The search will reveal heterogenous studies not suitable for meta-analysis. A narrative synthesis of the identified papers will be conducted to identify key methodological ideas and paradigms used to assess trauma care health systems. The analysis will consider how the differing methodological approaches could be adopted to understand barriers and delays to seeking, reaching and receiving care within a “Three Delays” framework. An iterative approach will be adopted to categorise identified articles, with the results presented as both within and across study analysis. Discussion The results of the review will be disseminated through publication in a peer-reviewed academic journal. The study forms part of a PhD project. The results will inform the development of a trauma care health system assessment applicable to LMICs. As this is a review of secondary data, no formal ethical approval is required. Systematic review registration PROSPERO CRD42018112990

Abstract Background While ‘dose’ is broadly understood as the ‘amount’ of an intervention, there is considerable variation in how this concept is defined. How we conceptualise, and subsequently measure, the dose of interventions has important implications for understanding how interventions produce their effects and are subsequently resourced and scaled up. This paper aims to explore the degree to which dose is currently understood as a distinct and well-defined implementation concept outside of clinical settings. Methods We searched four databases (MEDLINE, PsycINFO, EBM Reviews and Global Health) to identify original research articles published between 2000 and 2015 on health promotion interventions that contained the word ‘dose’ or ‘dosage’ in the title, abstract or keywords. We identified 130 articles meeting inclusion criteria and extracted data on how dose/dosage was defined and operationalised, which we then synthesised to reveal key themes in the use of this concept across health promotion interventions. Results Dose was defined in a variety of ways, including in relation to the amount of intervention delivered and/or received, the level of participation in the intervention and, in some instances, the quality of intervention delivery. We also observed some conflation of concepts that are traditionally kept separate (such as fidelity) either as slippage or as part of composite measures (such as ‘intervention dose’). Discussion Dose is not a well-defined or consistently applied concept in evaluations of health promotion interventions. While current approaches to conceptualisation and measurement of dose are suitable for interventions in organisational settings, they are less well suited to policies delivered at a population level. Dose often accompanies a traditional monotonic linear view of causality (e.g. dose response) which may or may not fully represent the intervention’s theory of how change is brought about. Finally, we found dose and dosage to be used interchangeably. We recommend a distinction between these terms, with ‘dosage’ having the advantage of capturing change to amount ‘dispensed’ over time (in response to effects achieved). Dosage therefore acknowledges the inevitable dynamic and complexity of implementation.

Abstract Background In this research paper we report on the quality of feedback provided in the logbooks of pre-clinical undergraduate students based on a model of ‘actionable feedback’. Feedback to clinical learners about their performance is crucial to their learning, which ultimately impacts on their development into competent clinicians. Due to students’ concerns regarding the inconsistency and quality of feedback provided by clinicians, a structured feedback improvement strategy to move feedback forward was added to the clinical skills logbook. The instrument was also extended for peer assessment. This study aims to assess the quality of feedback using the deliberate practice framework. Methods A feedback scoring system was used to retrospectively assess the quality of tutor and peer logbook feedback provided to second and third year medical students to identify deliberate practice components i.e. task, performance gap and action plan. The sample consisted of 425 second year and 600 third year feedback responses over a year. Results All three deliberate practice components were observed in the majority of the written feedback for both classes. The frequency was higher in peer (83%, 89%) than tutor logbook assessments (51%, 67%) in both classes respectively. Average tutor and peer task, gap and action feedback scores ranged from 1.84–2.07 and 1.93–2.21 respectively. The overall quality of feedback provided by the tutor and peer was moderate and less specific (average score

Abstract Background The goal of Lean Enterprise Transformation (LET) is to go beyond simply using Lean tools and instead embed Lean principles and practices in the system so that it becomes a fundamental, collective mindset of the entire enterprise. The Veterans Engineering Resource Center (VERC) launched the Veterans Affairs (VA) LET pilot program to improve quality, safety, and the Veteran’s experience. A national evaluation will examine the pilot program sites’ implementation processes, outcomes and impacts, and abilities to improve LET adoption and sustainment. This paper describes the evaluation design for the VA LET national evaluation and describes development of a conceptual framework to evaluate LET specifically in healthcare settings. Methods A targeted literature review of Lean evaluation frameworks was performed to inform the development of the conceptual framework. Key domains were identified by a multidisciplinary expert group and then validated with key stakeholders. The national evaluation design will examine LET implementation using qualitative, survey, and quantitative methods at ten VA facilities. Qualitative data include site visits, interviews, and field observation notes. Survey data include an employee engagement survey to be administered to front-line staff at all pilot sites. Quantitative data include site-level quality improvement metrics collected by the Veterans Services Support Center. Qualitative, quantitative, and mixed-methods analyses will be conducted to examine implementation of LET strategic initiatives and variations in implementation success across sites. Discussion This national evaluation of a large-scale LET implementation effort will provide insights helpful to other systems interested in embarking on a Lean journey. Additionally, we created a multi-faceted conceptual framework to capture the specific features of a Lean healthcare organization. This framework will guide this evaluation and may be useful as an assessment tool for other organizations interested in implementing Lean principles at an enterprise level.

Abstract Background Greenland struggles with a high prevalence of smoking, alcohol and drug abuse. In response to the increasing need for preventive initiatives, the first public health program Inuuneritta was introduced in 2007. Internationally, frameworks focus primarily on the implementation of a single, well-described intervention or program. However, with the increasing need and emergence of more holistic, integrated approaches, a need for research investigating the process of policy implementation from launch to action arises. This paper aims to augment the empirical evidence on the implementation of integrated health promotion programs within a governmental setting using the case of Inuuneritta II. In this study, the constraining and enabling determinants of the implementation processes within and across levels and sectors were examined. Methods Qualitative methods with a transdisciplinary approach were applied. Data collection consisted of six phases with different qualitative methods applied to gain a comprehensive overview and understanding of Inuuneritta II’s implementation process. These methods included: observations and focus group discussions at the community health worker (CHW) conference, telephone interviews, document analysis, and a workshop on results dissemination. Results Enabling determinants influencing the implementation process of Inuuneritta II positively were high motivation among adopters, local prevention committees supporting community health workers, and the initiation of the central prevention committee. In contrast, constraining determinants were ambiguous program aims, high turnovers, siloed budgets and work environments, and an inconsistent and neglected central prevention committee. Conclusion Inuuneritta II provided a substantial framework for an integrated health policy approach. However, having a holistic and comprehensive program enabling an integrated approach is not sufficient. Inuuneritta II’s integrated approach does not harmonise with the government’s inflexible organisational structure resulting in insufficient implementation.

Abstract Priority-setting (PS) for health research presents an opportunity for the relevant stakeholders to identify and create a list of priorities that reflects the country’s knowledge needs. Zambia has conducted several health research prioritisation exercises that have never been evaluated. Evaluation would facilitate gleaning of lessons of good practices that can be shared as well as the identification of areas of improvement. This paper describes and evaluates health research PS in Zambia from the perspectives of key stakeholders using an internationally validated evaluation framework. Methods This was a qualitative study based on 28 in-depth interviews with stakeholders who had participated in the PS exercises. An interview guide was employed. Data were analysed using NVIVO 10. Emerging themes were, in turn, compared to the framework parameters. Results Respondents reported that, while the Zambian political, economic, social and cultural context was conducive, there was a lack of co-ordination of funding sources, partners and research priorities. Although participatory, the process lacked community involvement, dissemination strategies and appeals mechanisms. Limited funding hampered implementation, monitoring and evaluation. Research was largely driven by the research funders. Conclusions Although there is apparent commitment to health research in Zambia, health research PS is limited by lack of funding, and consistently used explicit and fair processes. The designated national research organisation and the availability of tools that have been validated and pilot tested within Zambia provide an opportunity for focused capacity strengthening for systematic prioritisation, monitoring and evaluation. The utility of the evaluation framework in Zambia could indicate potential usefulness in similar low-income countries.

Abstract Background This paper reports on an online platform, People’s Open Access Education Initiative (Peoples-uni), as a means of enhancing access to master’s level public health education for health professionals. Peoples-uni seeks to improve population health in low- and middle-income countries by building public health capacity through e-learning at very low cost. We report here an evaluation of the Peoples-uni programme, conducted within the context of Sustainable Development Goal 4, which seeks to “ensure inclusive and quality education for all and promote lifelong learning” by 2030. The evaluation seeks to address the following three questions: (1) Did Peoples-uni meet its intended goals? (2) What were the different types of impacts that students experienced? (3) What suggestions for future changes in Peoples-uni did students recommend? Methods A mixed methods evaluation consisted of two parts, namely an online survey and a telephone interview. A total of 119 master’s level graduates were invited to participate; responses were obtained from 71 of those invited, giving a response rate of 60%. Respondents were spread across 31 countries. Interviews were conducted with 18 respondents. Results There was strong evidence that Peoples-uni had achieved its stated goals. Potential impacts on students included knowledge to enhance practice and appreciation of context, enhanced research capacity through knowledge of public health, critical thinking and evidence-based programming, and empowerment of students about the potential of education as a means of improving their lives. Accreditation through future partnerships with local universities was recommended by students. Conclusions Peoples-uni has been able to deliver a credible public health master’s level educational programme, with positive impacts on the students who graduated. Challenges are to find a way to accredit the programme to ensure its sustainability and to see how to take full advantage of the current, and future, graduates to turn this from an education programme into a capacity-building programme with real impact.

Abstract Background The implementation of Maternity Waiting Homes (MWHs) is a strategy to bring vulnerable women close to a health facility towards the end of their pregnancies. To date, while MWHs are a popular strategy, there is limited evidence on the role that MWHs play in reaching women most in need. This paper contributes to this topic by examining whether two program-supported MWHs in Malawi are reaching women in need and if there are changes in women reached over time. Methods Two rounds of exit interviews (2015 and 2017) were conducted with women within 3 months of delivery and included both MWH users and non-MWH users. These exit interviews included questions on sociodemographic factors, obstetric risk factors and use of health services. Bivariate statistics were used to compare MWH users and non-MWH users at baseline and endline and over time. Multivariable logistic regression was used to determine what factors were associated with MWH use, and Poisson regression was used to study factors associated with HIV knowledge. Descriptive data from discharge surveys were used to examine satisfaction with the MWH structure and environment over time. Results Primiparous women were more likely to use a MWH compared to women of parity 2 (p

Abstract Background Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) were funded by NIHR in England in 2008 and 2014 as partnerships between universities and surrounding health service organisations, focused on improving the quality of healthcare through the conduct and application of applied health research. The aim of this review is to synthesise learning from evaluations of the CLAHRCs. Methods Fifteen databases including CINAHL, MEDLINE, EMBASE and PsycINFO were searched to identify any evaluations of CLAHRCs. Current and archived CLAHRC websites and the reference lists of retrieved articles were scanned to identify any additional evaluations. Searches were restricted to English language only. Any publications from evaluations of the CLAHRCs were eligible for inclusion if they fulfilled at least one of three pre-specified inclusion criteria. A narrative synthesis was undertaken. Results Twenty-six evaluations (reported in 37 papers) were deemed eligible for inclusion. Evaluations focused on describing and exploring the formative partnerships, vision, values, structures and processes of CLAHRCs; the nature and role of boundaries; the deployment of knowledge brokers and hybrid roles to support knowledge mobilisation; patient and public involvement; and capacity building. The relative lack of data about the early impact of CLAHRCs on health care provision or outcomes is notable. Conclusions Much of the evaluative focus on CLAHRCs has been on how they have been organised and on the development of theory around their emergent properties. Evidence is lacking on the impact of CLAHRCs particularly in relation to the knowledge mobilisation processes and practices adopted. Further evaluation of CLAHRCs and other similar research and practice partnerships is warranted and should focus on which knowledge mobilisation approaches work where, how and why. Trial registration PROSPERO (Registration number: CRD42016042945).

Abstract Background Constant evaluation is important for maintaining and improving the quality of end-of-life care. We therefore conduct the fourth Japan Hospice and Palliative Evaluation Study (J-HOPE4) as a continuous evaluation study. In this present paper, we describe the design of J-HOPE4. The main purposes of J-HOPE4 are as follows:1) to evaluate the processes, structures, and outcomes of palliative care acute hospitals, palliative care units, and home hospice services; 2) to examine bereaved family members’ self-reported psychosocial conditions, such as grief and depression as bereavement outcomes;3) to provide data to ensure and improve the quality of care provided by participating institutions via feedback based on the results from each institution; and 4) provide clinical and academic information concerning the implications of various issues in palliative care by conducting additional studies. Methods We will conduct a cross-sectional, anonymous, self-reported questionnaire survey. In total, 190 institutions will participate in this study, meaning that 12,000 bereaved family members will be sent a questionnaire. Discussion This is one of the largest cross-sectional surveys involving hospice and palliative care, both in Japan and worldwide. Because this study will have a large sample size, the findings are expected to be generalizable to other settings.

Abstract Background January 2016 saw the publication of proposed revisions to the UK’s lower risk drinking guidelines but no sustained promotional activity. This paper aims to explore the impact of publishing guidelines without sustained promotional activity on reported guideline exposure and determinants of behaviour (capability, opportunity and motivation) proposed by the COM-B model. Methods Data were collected by a monthly repeat cross-sectional survey of adults (18+) resident in England over 15 months between November 2015 and January 2017 from a total of 16,779 drinkers, as part of the Alcohol Toolkit Study. Trends and associated 95% confidence intervals were described in the proportion of reported exposure to guidelines in the past month and measures of the capability, opportunity and motivation to consume alcohol within drinking guidelines. Results There was a rise in reported exposure to drinking guidelines in January 2016 (57.6–80.6%) which did not reoccur in January 2017. Following the increase in January 2016, reported exposure reduced slowly but remained significantly higher than in December 2015. In February 2016, there was an increase in measures of capability (31.1% reported tracking units of alcohol consumption and 87.8% considered it easier to drink safely) and opportunity (84.0% perceived their lifestyle as conducive to drinking within guidelines). This change was not maintained in subsequent months. Other measures showed marginal changes between January and February 2016 with no evidence of change in subsequent months. Conclusions Following the publication of revised drinking guideline in January 2016, there was a transient increase in exposure to guidelines, and capability and opportunity to drink within the guidelines that diminished over time. The transience and size of the changes indicate that behaviour change is unlikely. Well-designed, theory-based promotional campaigns may be required for drinking guidelines to be an effective public health intervention.

Abstract Background There are many challenges in delivering and evaluating knowledge for healthcare, but the lack of clear routes from knowledge to practice is a root cause of failures in safety within healthcare. Various types and sources of knowledge are relevant at different levels within the healthcare system. These need to be delivered in a timely way that is useful and actionable for those providing services or developing policies. How knowledge is taken up and used through networks and relationships, and the difficulties in attributing change to knowledge-based interventions, present challenges to understanding how knowledge into action (K2A) work influences healthcare outcomes. This makes it difficult to demonstrate the importance of K2A work, and harness support for its development and resourcing. This paper presents the results from a project commissioned by NHS Education for Scotland (NES) and Healthcare Improvement Scotland (HIS) to create an evaluation framework to help understand the NHS Scotland Knowledge into Action model. Methods The team took a developmental approach to creating an evaluation framework that would be useful and practical. This included a literature review to ensure the evaluation was evidence-based; adaptation of contribution analysis for K2A project; action research with K2A project leads to refine the work and develop suitable measures. Results Principles for evaluation and an evaluation framework based on contribution analysis were developed and implemented on a trial project. An outcomes chain was developed for the K2A programme and specific projects. This was used to design, collect and collate evidence of the K2A intervention. Data collected routinely by the intervention was supplemented with specific feedback measures from K2A project users. Conclusions The evaluation approach allowed for scrutiny of both processes and outcomes and was adaptable to projects on different scales. This framework has proved useful as a planning, reflecting and evaluation tool for K2A, and could be more widely used to evidence the ways in which knowledge to action work helps improve healthcare outcomes.

Abstract Background Interprofessional learning is gaining momentum in revolutionizing healthcare education. During the academic year 2015/16, seven undergraduate-entry health and social care programs from two universities in Hong Kong took part in an interprofessional education program. Based on considerations such as the large number of students involved and the need to incorporate adult learning principles, team-based learning was adopted as the pedagogy for the program, which was therefore called the interprofessional team-based learning program (IPTBL). The authors describe the development and implementation of the IPTBL program and evaluate the effectiveness of the program implementation. Methods Eight hundred and one students, who are predominantly Chinese, participated in the IPTBL. The quantitative design (a pretest-posttest experimental design) was utilized to examine the students’ gains on their readiness to engage in interprofessional education (IPE). Results Three instructional units (IUs) were implemented, each around a clinical area which could engage students from complementary health and social care disciplines. Each IU followed a team-based learning (TBL) process: pre-class study, individual readiness assurance test, team readiness assurance test, appeal, feedback, and application exercise. An electronic platform was developed and was progressively introduced in the three IUs. The students’ self-perceived attainment of the IPE learning outcomes was high. Across all four subscales of RIPLS, there was significant improvement in student’s readiness to engage in interprofessional learning after the IPTBL. A number of challenges were identified: significant time involvement of the teachers, difficulty in matching students from different programs, difficulty in making IPTBL count towards a summative assessment score, difficulty in developing the LAMS platform, logistics difficulty in managing paper TBL, and inappropriateness of the venue. Conclusions Despite some challenges in developing and implementing the IPTBL program, our experience showed that TBL is a viable pedagogy to be used in interprofessional education involving hundreds of students. The significant improvement in all four subscales of RIPLS showed the effects of the IPTBL program in preparing students for collaborative practice. Factors that contributed to the success of the use of TBL for IPE are discussed.

Abstract Background Translation encompasses the continuum from clinical efficacy to widespread adoption within the healthcare service and ultimately routine clinical practice. The Parenting, Eating and Activity for Child Health (PEACH™) program has previously demonstrated clinical effectiveness in the management of child obesity, and has been recently implemented as a large-scale community intervention in Queensland, Australia. This paper aims to describe the translation of the evaluation framework from a randomised controlled trial (RCT) to large-scale community intervention (PEACH™ QLD). Tensions between RCT paradigm and implementation research will be discussed along with lived evaluation challenges, responses to overcome these, and key learnings for future evaluation conducted at scale. Methods The translation of evaluation from PEACH™ RCT to the large-scale community intervention PEACH™ QLD is described. While the CONSORT Statement was used to report findings from two previous RCTs, the REAIM framework was more suitable for the evaluation of upscaled delivery of the PEACH™ program. Evaluation of PEACH™ QLD was undertaken during the project delivery period from 2013 to 2016. Results Experiential learnings from conducting the evaluation of PEACH™ QLD to the described evaluation framework are presented for the purposes of informing the future evaluation of upscaled programs. Evaluation changes in response to real-time changes in the delivery of the PEACH™ QLD Project were necessary at stages during the project term. Key evaluation challenges encountered included the collection of complete evaluation data from a diverse and geographically dispersed workforce and the systematic collection of process evaluation data in real time to support program changes during the project. Conclusions Evaluation of large-scale community interventions in the real world is challenging and divergent from RCTs which are rigourously evaluated within a more tightly-controlled clinical research setting. Constructs explored in an RCT are inadequate in describing the enablers and barriers of upscaled community program implementation. Methods for data collection, analysis and reporting also require consideration. We present a number of experiential reflections and suggestions for the successful evaluation of future upscaled community programs which are scarcely reported in the literature. Trials registration PEACH™ QLD was retrospectively registered with the Australian New Zealand Clinical Trials Registry on 28 February 2017 (ACTRN12617000315314).

Abstract Background Alcohol-related harm is a substantial burden on the community in Australia and internationally, particularly harm related to risky drinking practices of young people in the night-time economy. This protocol paper describes a study that will report on the changes in a wide range of health and justice outcome measures associated with major policy changes in the state of Queensland, Australia. A key element includes trading hours restrictions for licensed premises to 2 am for the state and 3 am in Safe Night Precincts (SNPs). Other measures introduced include drinks restrictions after midnight, increased patron banning measures for repeat offenders, mandatory ID scanning of patrons in late-night venues, and education campaigns. Methods The primary aim of the study is to evaluate change in the levels of harm due to these policy changes using administrative data (e.g., police, hospital, ambulance, and court data). Other study elements will investigate the impact of the Policy by measuring foot traffic volume in SNPs, using ID scanner data to quantify the volume of people entering venues and measure the effectiveness of banning notices, using patron interviews to quantify the levels of pre-drinking, intoxication and illicit drug use within night-time economy districts, and to explore the impacts of the Policy on business and live music, and costs to the community. Discussion The information gathered through this project aims to evaluate the effectiveness of the Policy and to draw on these findings to inform future prevention and enforcement approaches by policy makers, police, and venue staff.

Abstract Background Determining what constitutes “good practice” in the measurement of the costs and effects of health promotion and disease prevention measures is of particular importance. The aim of this paper was to gather expert knowledge on (economic) evaluations of health promotion and prevention measures for children and adolescents, especially on the practical importance, the determinants of project success, meaningful parameters for evaluations, and supporting factors, but also on problems in their implementation. This information is targeted at people responsible for the development of primary prevention or health promotion programs. Methods Partially structured open interviews were conducted by two interviewers and transcribed, paraphrased, and summarized for further use. Eight experts took part in the interviews. Results The interviewed experts saw evaluation as a useful tool to establish the effects of prevention programs, to inform program improvement and further development, and to provide arguments to decision making. The respondents’ thought that determinants of a program’s success were effectiveness with evidence of causality, cost benefit relation, target-group reach and sustainability. It was considered important that hard and soft factors were included in an evaluation; costs were mentioned only by one expert. According to the experts, obstacles to evaluation were lacking resources, additional labor requirements, and the evaluators’ unfamiliarity with a program’s contents. It was recommended to consider evaluation design before a program is launched, to co-operate with people involved in a program and to make use of existing structures. Conclusion While in in this study only a partial view of expert knowledge is represented, it could show important points to consider when developing evaluations of prevention programs. By considering these points, researchers could further advance towards a more comprehensive approach of evaluation targeting measures in children and adolescents.