Search

Abstract Background Evidence suggests that prehabilitation interventions, which optimise physical and mental health prior to treatment, can improve outcomes for surgical cancer patients and save costs to the health system through faster recovery and fewer complications. However, robust, theory-based evaluations of these programmes are needed. Using a theory of change (ToC) approach can guide evaluation plans by describing how and why a programme is expected to work. Theories of Change have not been developed for cancer prehabilitation programmes in the literature to date. This paper aims to provide an overview of the methodological steps we used to retrospectively construct a ToC for Prehab2Rehab (P2R), a cancer prehabilitation programme being implemented by the Cardiff and Vale University Health Board. Methods We used an iterative, participatory approach to develop the ToC. Following a literature review and document analysis, we facilitated a workshop with fourteen stakeholders from across the programme using a ‘backwards mapping’ approach. After the workshop, stakeholders had three additional opportunities to refine and validate a final working version of the ToC. Results Our process resulted in the effective and timely development of a ToC. The ToC captures how P2R’s interventions or activities are expected to bring about short, medium and long-term outcomes that, collectively, should result in the overarching desired impacts of the programme, which were improved patient flow and reduced costs to the health system. The process of developing a ToC also enabled us to have a better understanding of the programme and build rapport with key stakeholders. Conclusions The ToC has guided the design of an evaluation that covers the complexity of P2R and will generate lessons for policy and clinical practice on supporting surgical cancer patients in Wales and beyond. We recommend that evaluators apply a ToC development process at the outset of evaluations to bring together stakeholders and enhance the utilisation of the findings. This paper details a pragmatic, efficient and replicable process that evaluators could adopt to develop a ToC. Theory-informed evaluations may provide better evidence to develop and refine cancer prehabilitation interventions and other complex public health interventions.

Abstract Background Co-design with consumers and healthcare professionals is widely used in applied health research. While this approach appears to be ethically the right thing to do, a rigorous evaluation of its process and impact is frequently missing. Evaluation of research co-design is important to identify areas of improvement in the methods and processes, as well as to determine whether research co-design leads to better outcomes. We aimed to build on current literature to develop a framework to assist researchers with the evaluation of co-design processes and impacts. Methods A multifaceted, iterative approach, including three steps, was undertaken to develop a Co-design Evaluation Framework: 1) A systematic overview of reviews; 2) Stakeholder panel meetings to discuss and debate findings from the overview of reviews and 3) Consensus meeting with stakeholder panel. The systematic overview of reviews included relevant papers published between 2000 and 2022. OVID (Medline, Embase, PsycINFO), EBSCOhost (Cinahl) and the Cochrane Database of Systematic reviews were searched for papers that reported co-design evaluation or outcomes in health research. Extracted data was inductively analysed and evaluation themes were identified. Review findings were presented to a stakeholder panel, including consumers, healthcare professionals and researchers, to interpret and critique. A consensus meeting, including a nominal group technique, was applied to agree upon the Co-design Evaluation Framework. Results A total of 51 reviews were included in the systematic overview of reviews. Fifteen evaluation themes were identified and grouped into the following seven clusters: People (within co-design group), group processes, research processes, co-design context, people (outside co-design group), system and sustainment. If evaluation methods were mentioned, they mainly included qualitative data, informal consumer feedback and researchers’ reflections. The Co-Design Evaluation Framework used a tree metaphor to represent the processes and people in the co-design group (below-ground), underpinning system- and people-level outcomes beyond the co-design group (above-ground). To evaluate research co-design, researchers may wish to consider any or all components in the tree. Conclusions The Co-Design Evaluation Framework has been collaboratively developed with various stakeholders to be used prospectively (planning for evaluation), concurrently (making adjustments during the co-design process) and retrospectively (reviewing past co-design efforts to inform future activities).

Abstract Background Maintaining quality of care and improving the quality of life (QOL) of patients and their families are important issues in palliative care. Therefore, there is a need to continuously evaluate the quality and outcomes of the care provided. In Japan, the Japan hospice and palliative evaluation (J-HOPE) study series has been conducted every three to four years since 2010, and we will conduct the fifth J-HOPE study (J-HOPE5). In the present paper, we describe the protocol of the J-HOPE5 study. The main objectives are: (1) to evaluate the processes, structures and outcomes of care at hospices or palliative care units; (2) to examine bereaved family members’ self-reported psychosocial condition, such as grief and depression as bereavement outcomes; (3) to provide data to ensure and improve the quality of care provided by participating institutions via feedback based on the results from each institution; and (4) to provide clinical and academic information regarding the implications of various issues in palliative care by conducting specific research. Methods We will conduct a cross-sectional, anonymous, self-reported questionnaire survey. In total, 153 institutions have agreed to participate in this study, meaning that approximately 12,240 bereaved family members (n = 80/institution) will be sent a questionnaire. Discussion This is one of the largest cross-sectional bereavement surveys to evaluate the quality of specialized palliative care for patients with cancer, both in Japan and worldwide. The large sample size of this study will enable wide analyses of specific targets and topics.

Abstract Background The primary aim of this scoping review was to synthesise key domains and sub-domains described in existing clinical decision support systems (CDSS) implementation frameworks into a novel taxonomy and demonstrate most-studied and least-studied areas. Secondary objectives were to evaluate the frequency and manner of use of each framework, and catalogue frameworks by implementation stage. Methods A scoping review of Pubmed, Scopus, Web of Science, PsychInfo and Embase was conducted on 12/01/2022, limited to English language, including 2000–2021. Each framework was categorised as addressing one or multiple stages of implementation: design and development, evaluation, acceptance and integration, and adoption and maintenance. Key parts of each framework were grouped into domains and sub-domains. Results Of 3550 titles identified, 58 papers were included. The most-studied implementation stage was acceptance and integration, while the least-studied was design and development. The three main framework uses were: for evaluating adoption, for understanding attitudes toward implementation, and for framework validation. The most frequently used framework was the Consolidated Framework for Implementation Research. Conclusions Many frameworks have been published to overcome barriers to CDSS implementation and offer guidance towards successful adoption. However, for co-developers, choosing relevant frameworks may be a challenge. A taxonomy of domains addressed by CDSS implementation frameworks is provided, as well as a description of their use, and a catalogue of frameworks listed by the implementation stages they address. Future work should ensure best practices for CDSS design are adequately described, and existing frameworks are well-validated. An emphasis on collaboration between clinician and non-clinician affected parties may help advance the field.

Abstract Background The role of evaluation evidence in guiding health systems strengthening (HSS) investments at the global-level remains contested. A lack of rigorous impact evaluations is viewed by some as an obstacle to scaling resources. However, others suggest that power dynamics and knowledge hierarchies continue to shape perceptions of rigor and acceptability in HSS evaluations. This debate has had major implications on HSS resource allocation in global-level funding decisions. Yet, few studies have examined the relationship between HSS evaluation evidence and prioritization of HSS. In this paper, we explore the perspectives of key global health stakeholders, specifically around the nature of evidence sought regarding HSS and its potential impact on prioritization, the challenges in securing such evidence, and the drivers of intra- and inter-organizational divergences. We conducted a stakeholder analysis, drawing on 25 interviews with senior representatives of major global health organizations, and utilized inductive approaches to data analysis to develop themes. Results Our analysis suggests an intractable challenge at the heart of the relationship between HSS evaluations and prioritization. A lack of evidence was used as a reason for limited investments by some respondents, citing their belief that HSS was an unproven and potentially risky investment which is driven by the philosophy of HSS advocates rather than evidence. The same respondents also noted that the ‘holy grail’ of evaluation evidence that they sought would be rigorous studies that assess the impact of investments on health outcomes and financial accountability, and believed that methodological innovations to deliver this have not occurred. Conversely, others held HSS as a cross-cutting principle across global health investment decisions, and felt that the type of evidence sought by some funders is unachievable and not necessary – an ‘elusive quest’ – given methodological challenges in establishing causality and attribution. In their view, evidence would not change perspectives in favor of HSS investments, and evidence gaps were used as a ‘convenient excuse’. Respondents raised additional concerns regarding the design, dissemination and translation of HSS evaluation evidence. Conclusions Ongoing debates about the need for stronger evidence on HSS are often conducted at cross-purposes. Acknowledging and navigating these differing perspectives on HSS evaluation may help break the gridlock and find a more productive way forward.

Abstract Background On January 31st, 2023, the province of British Columbia (BC), Canada, was granted a federal exemption allowing adults (aged 18 +) to possess up to 2.5 g of select illicit drugs. The exemption will be in place for three years (2023–2026), marking the first formal decriminalization of illicit drug policy reform in Canada. BC’s decriminalization initiative is premised on several goals. This project seeks to evaluate each of these goals and their individual and combined contributions to determine the overall success of this policy. Methods The following protocol paper provides a detailed outline of a five-year (2022-2027) national evaluation of BC’s decriminalization initiative, as well as the specific objectives, methodologies, and planned analyses for eight interrelated sub-studies that comprise the evaluation. These sub-studies fall under the following five topical areas of research: 1) people who use drugs (PWUD), 2) the police and the criminal justice system, 3) the general public, 4) the health services system, and 5) an economic analysis. Additional research activities may also be explored. Results The overall evaluation and specific sub-study designs were informed by intensive stakeholder engagement. The evaluation was developed in collaboration with an international expert committee who came together to undertake a nominal group technique to decide on the final evaluation design and corresponding logic model. The evaluation will also employ an advisory board and individual sub-study working groups comprised of experts and PWUD who will oversee the development and implementation of the overall evaluation as well as each sub-study. Discussion This evaluation will draw on implementation science research practices to evaluate and understand the full impacts of this novel drug policy experiment. Results will be widely disseminated through manuscripts, reports, presentations, and infographics, which will be adapted and tailored for specific audiences. The protocol identifies several anticipated challenges and limitations. This evaluation’s evidence-based findings will be poised to offer pivotal insights that can shape and refine the discourse on drug policy and will serve as a critical resource for understanding the multifaceted impacts of decriminalization.

Abstract Background Addressing the upstream social determinants of health (e.g. built environment, education) can reduce the burden of non-communicable diseases. To do so effectively often requires system-wide collaboration. However, collaborating across multiple sectors, organizations and disciplines within a complex system can be challenging. ActEarly was a public health research consortium that aimed to improve child health by building an interdisciplinary, cross-city partnership to develop and/or evaluate upstream interventions, increase research capacity and improve collaboration between researchers, local authorities and communities. This paper explores ActEarly’s experiences of navigating complexity to identify mechanisms that supported its implementation and proposes recommendations for future intersectoral and interdisciplinary population health research collaborations. Methods We conducted a longitudinal qualitative study of ActEarly, integrating findings from inductive documentary analysis of internal documents (mainly meetings minutes and reports) (n = 114) and interviews (n = 70) with 45 consortium members at three different timepoints (2018, 2021, 2023). Participants worked across different organizations, cities, roles and levels of seniority in the consortium. Findings Clarity, Unity, Flexibility and Feasibility were seen as the key mechanisms required to support ActEarly’s implementation. Clear aims, governance structures and communication were necessary to manage the uncertainty of the complex system. A unified approach, characterized by strong relationships, having a shared vision and communal access to resources supported effective collaboration. Flexibility was required to adjust to different ways of working, respond to wider system events and manage the consortium. Establishing feasible aims that responded to the limitations of the system, the available resources and research infrastructure was required for teams to deliver the work. Conclusions Implementing multi-faceted programmes in a complex system can be challenging. We recommend that future whole-systems consortia seeking to improve population health build Clarity, Unity, Flexibility and Feasibility into their programmes, noting the complex interrelationships between these factors. Iterative reflections from all parties should support delivery amidst the uncertainty that comes with running a population health research collaboration, and strong leadership and governance should play a key role in ensuring that these are built into foundations the programme.

Abstract Background In 2016, a voluntary National Healthy Food and Drink Policy (hereafter, “the Policy”) was released to encourage public hospitals in New Zealand to provide food and drink options in line with national dietary guidelines. Five years later, eight (of 20) organisations had adopted it, with several preferring to retain or update their own institutional-level version. This study assessed staff and visitors’ awareness and support for and against the Policy, and collected feedback on perceived food environment changes since implementation of the Policy. Methods Cross-sectional electronic and paper-based survey conducted from June 2021 to August 2022. Descriptive statistics were used to present quantitative findings. Free-text responses were analysed following a general inductive approach. Qualitative and quantitative findings were compared by level of implementation of the Policy, and by ethnicity and financial security of participants. Results Data were collected from 2,526 staff and 261 visitors in 19 healthcare organisations. 80% of staff and 56% of visitors were aware of the Policy. Both staff and visitors generally supported the Policy, irrespective of whether they were aware of it or not, with most agreeing that “Hospitals should be good role models.” Among staff who opposed the Policy, the most common reason for doing so was freedom of choice. The Policy had a greater impact, positive and negative, on Māori and Pacific staff, due to more frequent purchasing onsite. Most staff noticed differences in the food and drinks available since Policy implementation. There was positive feedback about the variety of options available in some hospitals, but overall 40% of free text comments mentioned limited choice. 74% of staff reported that food and drinks were more expensive. Low-income staff/visitors and shift workers were particularly impacted by reduced choice and higher prices for healthy options. Conclusions The Policy led to notable changes in the healthiness of foods and drinks available in NZ hospitals but this was accompanied by a perception of reduced value and choice. While generally well supported, the findings indicate opportunities to improve implementation of food and drink policies (e.g. providing more healthy food choices, better engagement with staff, and keeping prices of healthy options low) and confirm that the Policy could be expanded to other public workplaces.

Abstract Background Medical education is a multifarious endeavour integrating a range of pedagogies and philosophies. Complexity as a science or theory (‘complexity’) signals a move away from a reductionist paradigm to one which appreciates that interactions in multi-component systems, such as healthcare systems, can result in adaptive and emergent outcomes. This examination of the nexus between medical education and complexity theory aims to discover ways that complexity theory can inform medical education and medical education research. Methods A structured literature review was conducted to examine the nexus between medical education and complexity; 5 databases were searched using relevant terms. Papers were included if they engaged fully with complexity as a science or theory and were significantly focused on medical education. All types of papers were included, including conceptual papers (e.g. opinion and theoretical discussions), case studies, program evaluations and empirical research. A narrative and thematic synthesis was undertaken to create a deep understanding of the use of complexity in medical education. Results Eighty-three papers were included; the majority were conceptual papers. The context and theoretical underpinnings of complexity as a relevant theory for medical education were identified. Bibliographic and temporal observations were noted regarding the entry of complexity into medical education. Complexity was relied upon as a theoretical framework for empirical studies covering a variety of elements within medical education including: knowledge and learning theories; curricular, program and faculty development; program evaluation and medical education research; assessment and admissions; professionalism and leadership; and learning for systems, about systems and in systems. Discussion There is a call for greater use of theory by medical educators. Complexity within medical education is established, although not widespread. Individualistic cultures of medicine and comfort with reductionist epistemologies challenges its introduction. However, complexity was found to be a useful theory across a range of areas by a limited number of authors and is increasingly used by medical educators and medical education researchers. This review has further conceptualized how complexity is being used to support medical education and medical education research. Conclusion This literature review can assist in understanding how complexity can be useful in medical educationalists' practice.

Plain English summary Children and young people (CYP) have a right to be involved in things that affect them, including research. There is growing interest in children and young people’s involvement in health research in the United Kingdom (UK), as well as understanding that what works for CYP is often different to what works for adults. This paper presents an evaluation of the Young Person's Advisory Group (YPAG) at Moorfields Eye Hospital in London. Evaluation uses research methods to find out how well a service or project is working and meeting the needs of those who use it, and how to improve things that could be better. A group of young evaluators, supported by adult researchers, designed and ran the evaluation; three of the young evaluators also helped write this paper. In our evaluation we used focus (discussion) groups and online surveys with young group members, their parents and carers, researchers who had worked with the group, the group’s facilitators (adults who help manage the group) and funders. We found that group members valued the social and creative aspects of the group as well as learning about research and developing skills and confidence. Learning was a two-way process, with both researchers and facilitators talking about how much they had learnt from working with the YPAG. All participants talked about the importance of feeling that CYP are making a difference to research, as well as of CYP’s right to be involved. Planning and support were important to the group working well, but we found that having the money and time to do this well was not always easy. And, while lots of researchers were keen to work with the group, and talked about how this had helped their research, we need to do more to engage researchers who have yet to be convinced. We also found that, while we wanted to ‘co-produce’ the group and share power for all big decisions, this was something we had to work towards, especially when group members were young and/or new to research and involvement. Co-producing an evaluation helped us to learn about the benefits and challenges of involving CYP in research, as well as how to involve them in evaluating that involvement.

Abstract Background Artificial intelligence (AI) is seen as one of the major disrupting forces in the future healthcare system. However, the assessment of the value of these new technologies is still unclear, and no agreed international health technology assessment-based guideline exists. This study provides an overview of the available literature in the value assessment of AI in the field of medical imaging. Methods We performed a systematic scoping review of published studies between January 2016 and September 2020 using 10 databases (Medline, Scopus, ProQuest, Google Scholar, and six related databases of grey literature). Information about the context (country, clinical area, and type of study) and mentioned domains with specific outcomes and items were extracted. An existing domain classification, from a European assessment framework, was used as a point of departure, and extracted data were grouped into domains and content analysis of data was performed covering predetermined themes. Results Seventy-nine studies were included out of 5890 identified articles. An additional seven studies were identified by searching reference lists, and the analysis was performed on 86 included studies. Eleven domains were identified: (1) health problem and current use of technology, (2) technology aspects, (3) safety assessment, (4) clinical effectiveness, (5) economics, (6) ethical analysis, (7) organisational aspects, (8) patients and social aspects, (9) legal aspects, (10) development of AI algorithm, performance metrics and validation, and (11) other aspects. The frequency of mentioning a domain varied from 20 to 78% within the included papers. Only 15/86 studies were actual assessments of AI technologies. The majority of data were statements from reviews or papers voicing future needs or challenges of AI research, i.e. not actual outcomes of evaluations. Conclusions This review regarding value assessment of AI in medical imaging yielded 86 studies including 11 identified domains. The domain classification based on European assessment framework proved useful and current analysis added one new domain. Included studies had a broad range of essential domains about addressing AI technologies highlighting the importance of domains related to legal and ethical aspects.

Abstract Background An increasing number of evaluations of social accountability (SA) interventions have been published in the past decade, however, reporting gaps make it difficult to summarize findings. We developed the Social Accountability Reporting for Research (SAR4Research) checklist to support researchers to improve the documentation of SA processes, context, study designs, and outcomes in the peer reviewed literature and to enhance application of findings. Methods We used a multi-step process, starting with an umbrella review of reviews on SA to identify reporting gaps. Next, we reviewed existing guidelines for reporting on behavioral interventions to determine whether one could be used in its current or adapted form. We received feedback from practitioners and researchers and tested the checklist through three worked examples using outcome papers from three SA projects. Results Our umbrella review of SA studies identified reporting gaps in all areas, including gaps in reporting on the context, intervention components, and study methods. Because no existing guidelines called for details on context and the complex processes in SA interventions, we used CONSORT-SPI as the basis for the SAR4Research checklist, and adapted it using other existing checklists to fill gaps. Feedback from practitioners, researchers and the worked examples suggested the need to eliminate redundancies, add explanations for items, and clarify reporting for quantitative and qualitative study components. Conclusions Results of SA evaluations in the peer-reviewed literature will be more useful, facilitating learning and application of findings, when study designs, interventions and their context are described fully in one or a set of papers. This checklist will help authors report better in peer-reviewed journal articles. With sufficient information, readers will better understand whether the results can inform accountability strategies in their own contexts. As a field, we will be better able to identify emerging findings and gaps in our understanding of SA.

Abstract Background Since 1996, an urban community-based organization whose primary mission is to serve diverse94 and emerging community health needs has provided screening, testing, overdose prevention and training, referrals, and access to treatment for substance use disorders (SUD) and communicable diseases such as HIV through its Life Points harm reduction program. Methods As a partner in a State survey in 2021, the community organization recruited a convenience sample of people who use drugs to participate in a survey focused on their substance use, healthcare, and barriers to SUD services. Community health workers conducted outreach and used an encrypted identifier to collect data from a convenience sample of harm reduction participants regarding demographics, legal justice, engagement in harm reduction and access to healthcare. Evaluators entered paper surveys into Qualtrics for reporting and summative analysis. Results A convenience sample of fifty-five people who use drugs were recruited and surveyed. The majority (86%, n = 47) were active participants in the agency Life Points (LP) harm reduction service. Participants’ average age was 42.9 years (SD = 11.5). About half (51%, n = 28) were male, 48% (n = 26) were female, and 2% (n = 1) was transgender. About two-thirds (67%, n = 37) of participants were White/Caucasian, 13% (n = 7) were Black/African-American, 11% (n = 6) were Hispanic and 7% (n = 4) were Multi-Racial. Regarding current substance use, 98% (n = 54) reported use of heroin, 51% (n = 28) reported crack, 47% (n = 26) cocaine, 25% (n = 14) alcohol, 24% (n = 13) opioids, and 15% (n = 8) marijuana. The majority, 87% (n = 48) said they had health care insurance and over two-thirds (69%, n = 37) said they had been arrested for a felony. Almost three quarters (71%, n = 39) reported receiving services from the Department of Health & Human Services. A higher percentage of females compared to males (65% and 29% respectively) reported engagement in community mental health services and 69% of females (n = 18) compared to 15% (n = 4) of males reported needing to participate in sex to meet basic social needs. Participants described social determinants of health as barriers to services, including access to food, legal justice and transportation. About 44% (n = 24) said they would consider enrolling in a drug treatment program in the next 30 days. Conclusion This sample was reflective of increased participation by White participants that began to appear about a decade ago. The majority of participants reported having healthcare insurance, which may be reflective of engagement with community health workers to access appropriate services. Community organizations and healthcare professionals should continue to explore social determinants of health that can impact the health of people who use drugs, including overcoming barriers to health care access such as investing in mobile unit outreach.

Abstract Background ALL_EARS@UoS is a patient and public involvement and engagement (PPIE) group for people with lived experience of hearing loss. The purpose of the group is to share experiences of hearing loss and hearing healthcare, inform research and improve services for patients at University of Southampton Auditory Implant Service. A year after inception, we wanted to critically reflect on the value and challenges of the group. Four members of ALL_EARS@UoS were recruited to an evaluation steering group. This paper reports the evaluation of the group using the UK Standards for Public Involvement. Methods An anonymous, mixed-methods questionnaire was co-designed and shared with members of ALL_EARS@UoS using an online platform. The questionnaire was designed to capture satisfaction, individual feedback through free-text answers, and demographic information. Descriptive statistics have been used to express the satisfaction and demographic data. Reflexive thematic analysis has been used to analyse the free-text responses. Group engagement and activity data over time were monitored and collected. Results The questionnaire response rate was 61% (11/18). Areas identified as strengths were ‘Communication’ and ‘Working together’. Five themes were developed from the thematic analysis; (1) Increased knowledge and awareness around the topic of hearing health for group members and wider society, (2) supporting research, (3) inclusivity within the group, (4) opportunity to make a difference for people in the future and (5) running of the group/group organisation. The data highlighted the value and challenges of PPIE. Members described feeling listened to and appreciation of being able to share experiences. Time of day and meeting format were identified as challenges as they affected who could attend the meetings. The ability to secure and maintain sufficient funding and time to support inclusive and diverse PPIE activities is a challenge for researchers. Conclusions We have identified how PPIE added value to both group members and researchers, emphasising the true benefit of PPIE. We have highlighted challenges we are facing and our plan to tackle these. We aim to continue to develop and sustain a group that reflects the diversity of the Deaf/deaf or hard of hearing community and of our local community.

Abstract Background By including the needs and perspectives of relevant stakeholders, co-creation is seen as a promising approach for tackling complex public health problems. However, recommendations and guidance on how to plan and implement co-creation are lacking. By identifying and analysing existing implementation and evaluation frameworks for public health, this study aims to offer key recommendations for professional stakeholders and researchers wanting to adopt a co-creation approach to public health interventions. Methods Firstly, PubMed and CINAHL databases were screened for articles introducing original implementation and evaluation frameworks for public health interventions. Backwards snowballing techniques were applied to the included papers. Secondly, identified frameworks were classified and relevant data extracted, including steps and constructs present in the frameworks. Lastly, recommendations were derived by conducting thematic analysis on the included frameworks. Results Thirty frameworks were identified and data related to their nature and scope extracted. The frameworks’ prominent steps and constructs were also retrieved. Recommendations related to implementation and evaluation in the context of co-creation were included. Conclusion When engaging in co-creation, we recommend including implementation considerations from an early stage and suggest adopting a systems thinking as a way to explore multiple levels of influence, contextual settings and systems from an early planning stage. We highlight the importance of partnering with stakeholders and suggest applying an evaluation design that is iterative and cyclical, which pays particular attention to the experience of the engaged co-creators.

Abstract Background This project (named Reinvent) aimed to promote Public Involvement (PI) in health research. Academics worked with a community group, the Eloquent Praise & Empowerment Dance Company, to develop a community partnership with young people from Black African, Asian and Caribbean heritage communities. The goal of this paper is to evaluate the Reinvent project for key learnings on how to engage and build partnerships with young people from Black African, Asian and Caribbean heritage communities. Methods Reinvent developed a steering group which consisted of five young people, one academic, a Race Equality Ambassador and the Director of Eloquent. The steering group co-produced an agenda for two workshops and the evaluation tools used. The content of the workshops included drama exercises, discussions on physical and mental health, nutrition and school-life, short introductions to the concepts of research and PI, and group work to critique and improve a video currently used to promote PI in health research to young people. The evaluation tools included using the ‘Cube’ evaluation framework, video-blogging and collecting anonymous feedback. Findings The responses to the ‘Cube’ evaluation framework were positive across all four domains (agenda, voice, contribute change) in both workshops. A few of the young people described having a better understanding of the meaning and practice of PI in a video-blog. The anonymous feedback suggested that the workshops had increased young people’s confidence in sharing their thoughts and opinions about health and PI. Conclusion Reinvent has shown that academic institutions and young people from an under-served community can partner to co-design workshops and apply evaluation tools. Working with young people in an environment in which they were comfortable, and by researchers joining in with the activities that the young people enjoyed (such as dance), enabled more informal and open conversations to develop. More work is needed to build upon this project so that young people can feel confident and supported to get involved in PI activities relating to research.

Abstract Based on the panel data of 31 provinces (municipalities and autonomous regions) in China from 2012 to 2019, this paper constructs the evaluation index system of basic medical and health services in China from seven dimensions: medical and health facilities, health expenditure, medical services, traditional Chinese medicine hospital services, maternal and child health care, people’s health and medical security, disease control and public health. The entropy method was used to measure the level of basic medical and health services in China, and its spatial differences and convergence characteristics were further investigated. In this study, we employ the entropy weight method, σ convergence, and β convergence as our primary methodologies. The entropy weight method is used to evaluate the variability of each indicator, determine the weights of indicators, and quantify the information content of the data. σ convergence illustrates the process by which the variance of a sample decreases over time. β convergence refers to the gradual approach of variables within an economic system towards their long-term equilibrium level over time. The results show that: (1) The scores of basic medical and health services in China’s four major regions (including Northeast, East, Central and West) remain in a relatively stable state, with small fluctuations and great room for improvement; (2) There are significant regional differences in the level of basic medical and health services in China, and the intra-regional differences are much greater than the inter-regional differences; (3) There is no significant σ convergence observed in China and its four major regions; however, there is a notable presence of β convergence.

Abstract Objectives To evaluate the impact a novel education programme - to improve research engagement, awareness, understanding and confidence - had on a diverse health and social care workforce. Barriers and facilitators to engagement were explored together with research capacity-building opportunities and ways to embed a research culture. The programme is entitled ‘Supporting The Advancement of Research Skills’ (STARS programme); the paper reports findings from a health and social care setting in England, UK. Methods A four-level outcome framework guided the approach to evaluation and was further informed by key principles of research capacity development and relevant theory. Quantitative data were collected from learners before and after engagement; these were analysed descriptively. Semi-structured online interviews were conducted with learners and analysed thematically. A purposive sample was achieved to include a diversity in age, gender, health and social care profession, and level of attendance (regular attendees, moderate attendees and non-attenders). Results The evaluation spanned 18 half-day workshops and 11 seminars delivered by expert educators. 165 (2% of total staff at Midlands Partnership University NHS Foundation Trust (MPFT)) staffs booked one or more education sessions; 128 (77%) including Allied Health Professionals (AHPs), psychologists, nursing and midwifery, and social workers attended one or more session. Key themes of engagement with teaching sessions, relevance and impact of training and promoting a research active environment were identified with relevant sub-themes. Positive impacts of training were described in terms of research confidence, intentions, career planning and application of research skills as a direct result of training. Lack of dedicated time for research engagement, work pressures and time commitments required for the programme were key barriers. Facilitators that facilitated engagement are also described. Conclusions Findings demonstrate the impact that a free, virtual and high-quality research education programme had at individual and organisational levels. The programme is the product of a successful collaboration between health and social care and academic organisations; this provides a useful framework for others to adapt and adopt. Key barriers to attendance and engagement spoke to system-wide challenges that an education programme could not address in the short-term. Potential solutions are discussed in relation to protecting staff time, achieving management buy-in, recognising research champions, and having a clear communication strategy.

Abstract Background The AgResults initiative tests the efficacy of results-based prizes to scale-up smallholder technology adoption. In Nigeria the project awarded a $18.75/ton prize for private sector actors who aggregated maize from smallholders that was treated by Aflasafe—a biocontrol that addresses liver cancer-causing aflatoxin contamination in maize. This paper examines the impact of AgResults initiative on smallholder farmers. Methods This evaluation estimates the causal effect of the AgResults program on farmer outcomes by comparing survey data from AgResults farmers to survey data from a matched comparison group of farmers. To improve balance, we use propensity score weights. In considering inestimable selection bias, we describe several key considerations, including the inclusion of comparison areas for treatment post-evaluation. Results The project increased Aflasafe adoption by 56% points, farmers earned 16% more net maize income on average. However, the majority of farmers in villages engaged by the project did not know about Aflasafe’s health benefits. This suggests that complimentary donor-directed efforts may still be needed to generate general awareness about the technologies whose benefit is not immediately visible to the smallholders. With the prize focused on aggregation, private sector actors may have reduced incentive to raise awareness about Aflasafe’s health benefits in case farmers held back Aflasafe-treated maize for consumption. Conclusions This paper highlights the potential of results-based prizes to engage the private sector in solving development problems. However, it also equally highlights the gaps that such an approach may have, arguing for the need for having complimentary efforts to address those gaps. This is particularly the case when the technology’s benefits are not perceived by the consumer (aflatoxins are not visible to the eye, and the health benefits are not immediate), or when the technology results in positive externalities (final consumers of Aflasafe-treated maize also benefit). Broader consumer awareness needs to be raised to promote continued development of the market for Aflasafe-treated maize, aflatoxin standards need to be enforced, and aflatoxin testing needs to be more easily available.

Abstract Background Due to the global digitalization, implementation of digital elements into daily work can support physiotherapists’ work but may also pose some challenges. Only little is known about physiotherapists’ attitude towards digitalization. This study primarily aimed to analyze physiotherapists’ attitude towards digitalization and to what extend digital tools have been implemented into their daily work. In second analysis, participants’ characteristics such as age, working place, gender and mode of survey participation were assessed. Methods A 12-main-item survey amongst voluntary course participants of one physiotherapeutic training center was conducted via paper-based as well as online questionnaires between July 2018 and June 2019 including questions on participants’ general as well as particular attitude towards digitalization, the use of (mobile) applications and possible advantages and disadvantages of the ongoing digital transformation. Sub-analysis was performed for age (≤40 years versus > 40 years), gender, mode of participation (paper vs. online) and working place (practice vs. hospital). Results Overall, 488 physiotherapists participated in the survey. In comparison of the age groups, younger participants had more concerns about data security (p = 0.042) and insufficient financial remuneration (p

Abstract Background Over 180,000 people use crack cocaine in England, yet provision of smoking equipment to support safer crack use is prohibited under UK law. Pipes used for crack cocaine smoking are often homemade and/or in short supply, leading to pipe sharing and injuries from use of unsafe materials. This increases risk of viral infection and respiratory harm among a marginalised underserved population. International evaluations suggest crack pipe supply leads to sustained reductions in pipe sharing and use of homemade equipment; increased health risk awareness; improved service access; reduction in injecting and crack-related health problems. In this paper, we introduce the protocol for the NIHR-funded SIPP (Safe inhalation pipe provision) project and discuss implications for impact. Methods The SIPP study will develop, implement and evaluate a crack smoking equipment and training intervention to be distributed through peer networks and specialist drug services in England. Study components comprise: (1) peer-network capacity building and co-production; (2) a pre- and post-intervention survey at intervention and non-equivalent control sites; (3) a mixed-method process evaluation; and (4) an economic evaluation. Participant eligibility criteria are use of crack within the past 28 days, with a survey sample of ~ 740 for each impact evaluation survey point and ~ 40 for qualitative process evaluation interviews. Our primary outcome measure is pipe sharing within the past 28 days, with secondary outcomes pertaining to use of homemade pipes, service engagement, injecting practice and acute health harms. Anticipated impact SIPP aims to reduce crack use risk practices and associated health harms; including through increasing crack harm reduction awareness among service providers and peers. Implementation has only been possible with local police approvals. Our goal is to generate an evidence base to inform review of the legislation prohibiting crack pipe supply in the UK. This holds potential to transform harm reduction service provision and engagement nationally. Conclusion People who smoke crack cocaine in England currently have little reason to engage with harm reduction and drug services. Little is known about this growing population. This study will provide insight into population characteristics, unmet need and the case for legislative reform. Trial registration : ISRCTN12541454 https://doi.org/10.1186/ISRCTN12541454

Abstract Background A robust evidence base is required to assist healthcare commissioners and providers in selecting effective and sustainable approaches to improve cancer diagnosis and treatment. Such evidence can be difficult to build, given the fast-paced and highly pressured nature of healthcare delivery, the absence of incentives, and the presence of barriers in conducting pragmatic yet robust research evaluations. Cancer Research UK (CRUK) has played an active part in building the evidence base through its funding of programmes to identify, evaluate and scale-up innovative approaches across the UK. The aim of this paper is to describe and explain the research design and intended approach and activities for two cancer services improvement projects in Scotland funded by CRUK. Methods A hybrid effectiveness-implementation study design will assess both the efficiency of the new pathways and their implementation strategies, with the aim of generating knowledge for scale-up. A range of implementation, service and clinical outcomes will be assessed as determined by the projects’ Theories of Change (ToCs). A naturalistic case study approach will enable in-depth exploration of context and process, and the collection and synthesis of data from multiple sources including routine datasets, patient and staff surveys, in-depth interviews and observational and other data. The evaluations are informed throughout by a patient/public representatives’ group, and by small group discussions with volunteer cancer patients. Discussion Our approach has been designed to provide a holistic understanding of how (well) the improvement projects work (in relation to their anticipated outcomes), and how they interact with their wider contexts. The evaluations will help identify barriers, facilitators, and unanticipated consequences that can impact scalability, sustainability and spread. By opting for a pragmatic, participatory evaluation research design, we hope to inform strategies for scaling up successful innovations while addressing challenges in a targeted manner.

Abstract Background Various questionnaires are used for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health (mHealth) services. Using the best one to meet the needs of an mHealth study is a challenge for researchers. Therefore, this study aimed to review and determine the frequently used questionnaires for evaluating the mentioned outcomes of mHealth services. Methods The PubMed database was searched for conducting this review in April 2021. Papers that used a referenced questionnaire to evaluate the satisfaction, usability, acceptance, or quality outcomes of mHealth were included. The first author’s name, year of publication, evaluation outcome, and evaluation questionnaire were extracted from relevant papers. Data were analyzed using descriptive statistics. Results In total, 247 papers were included in the study. Questionnaires were used for usability (40%), quality (34.5%), acceptance (8.5%), and satisfaction (4%) outcomes, respectively. System usability scale (36.5%), mobile application rating scale (35.5%), post study system usability questionnaire (6%), user mobile application rating scale (5%), technology acceptance model (4.5%), computer system usability questionnaire (2.5%), net promoter score (2%), health information technology usability evaluation scale (2%), the usefulness, satisfaction, and ease of use (1.5%), client satisfaction questionnaire (1.5%), unified theory of acceptance and use of technology (1.5%), questionnaire for user interaction satisfaction (1%), user experience questionnaire (1%), and after-scenario questionnaire (1%) were the most used questionnaires, respectively. Conclusion Despite the existence of special questionnaires for evaluating several outcomes of mHealth, general questionnaires with fewer items and higher reliability have been used more frequently. Researchers should pay more attention to questionnaires with a goal-based design.

Abstract Background Innovative tools are needed to complement the existing approach for malaria elimination. Gene drive mosquitoes are one potential new technology in the control of malaria vectors. Target Malaria is one of the research projects developing this technology, and in July 2019, the project proceeded to an important step for this evaluation pathway: the small-scale release of non-gene drive sterile male mosquitoes in a village in Burkina Faso. In addition to the entomological and laboratory work to prepare for this important milestone, significant community and stakeholder engagement work was done. The existing guidelines on gene drive mosquito provide an overall framework for such engagement work. However, they do not provide a road map on how to proceed or what benchmarks should be used to assess this work. Methods This study provides a review of engagement activities relevant to field trials on non-gene drive genetically-modified mosquitoes as well as an assessment framework—using both qualitative and quantitative studies as well as an audit procedure. The latter was implemented to evaluate whether the release activities could proceed with the appropriate level of agreement from the community. Results This paper shows the importance of this first phase of work to innovate and learn about engagement processes for responsible research in the field of genetic approaches for malaria vector control. The function of these assessments is crucial for the learning agenda. The assessments demonstrated ways to increase understanding and ensure effective progress with field studies and, therefore, the pathway for responsible research. Conclusion Gene drive technology is increasingly considered as a promising approach to control vector borne diseases, in particular malaria. Stakeholders’ involvement in this research process is one of the recurring requirements in international guidance documents. With this paper Target Malaria offers an opportunity to explore the practical achievements and challenges of stakeholder engagement during early phases of a technology evaluation, and in particular how it implemented an assessment framework to learn from its experience.

Abstract Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a “pipeline” approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to “take up the mantle” and tackle the complexity of real-world research translation and evaluation.

Abstract Background Qualitative Comparative Analysis (QCA) is a method for identifying the configurations of conditions that lead to specific outcomes. Given its potential for providing evidence of causality in complex systems, QCA is increasingly used in evaluative research to examine the uptake or impacts of public health interventions. We map this emerging field, assessing the strengths and weaknesses of QCA approaches identified in published studies, and identify implications for future research and reporting. Methods PubMed, Scopus and Web of Science were systematically searched for peer-reviewed studies published in English up to December 2019 that had used QCA methods to identify the conditions associated with the uptake and/or effectiveness of interventions for public health. Data relating to the interventions studied (settings/level of intervention/populations), methods (type of QCA, case level, source of data, other methods used) and reported strengths and weaknesses of QCA were extracted and synthesised narratively. Results The search identified 1384 papers, of which 27 (describing 26 studies) met the inclusion criteria. Interventions evaluated ranged across: nutrition/obesity (n = 8); physical activity (n = 4); health inequalities (n = 3); mental health (n = 2); community engagement (n = 3); chronic condition management (n = 3); vaccine adoption or implementation (n = 2); programme implementation (n = 3); breastfeeding (n = 2), and general population health (n = 1). The majority of studies (n = 24) were of interventions solely or predominantly in high income countries. Key strengths reported were that QCA provides a method for addressing causal complexity; and that it provides a systematic approach for understanding the mechanisms at work in implementation across contexts. Weaknesses reported related to data availability limitations, especially on ineffective interventions. The majority of papers demonstrated good knowledge of cases, and justification of case selection, but other criteria of methodological quality were less comprehensively met. Conclusion QCA is a promising approach for addressing the role of context in complex interventions, and for identifying causal configurations of conditions that predict implementation and/or outcomes when there is sufficiently detailed understanding of a series of comparable cases. As the use of QCA in evaluative health research increases, there may be a need to develop advice for public health researchers and journals on minimum criteria for quality and reporting.

Abstract The field of transdisciplinary sustainability research has brought forward a number of approaches aimed at fostering sustainability transformations and generating knowledge through collaborative experimentation in real-world settings. These cases are strongly embedded in their local context and thus the transfer of knowledge remains a key challenge. In this paper, we propose a case reporting approach that supports the structured and coherent reporting of such cases. This scheme is aimed at sustainability-oriented labs, where sustainability solutions are collaboratively developed through experimentation. The scheme focuses the reporting on local contexts, lab processes, and experiments. It is accompanied by a logic model and a set of four principles guiding the reporting procedure. The approach is designed to be general, in that it is applicable to diverse contexts and project designs, while its modularity allows the scheme to be adapted to the needs and specifics of each cases. The scheme was jointly developed and tested by a group of seven Urban Living Labs, each in their own unique context. With our approach we aim to contribute to knowledge transfer from and across cases of sustainability-oriented labs as emerging approaches in action-oriented research bridging the divide of case-based research and (meta) comparison.

Abstract Background The growth of data science and artificial intelligence offers novel healthcare applications and research possibilities. Patients should be able to make informed choices about using healthcare. Therefore, they must be provided with lay information about new technology. A team consisting of academic researchers, health professionals, and public contributors collaboratively co-designed and co-developed the new resource offering that information. In this paper, we evaluate this novel approach to co-production. Methods We used participatory evaluation to understand the co-production process. This consisted of creative approaches and reflexivity over three stages. Firstly, everyone had an opportunity to participate in three online training sessions. The first one focused on the aims of evaluation, the second on photovoice (that included practical training on using photos as metaphors), and the third on being reflective (recognising one’s biases and perspectives during analysis). During the second stage, using photovoice, everyone took photos that symbolised their experiences of being involved in the project. This included a session with a professional photographer. At the last stage, we met in person and, using data collected from photovoice, built the mandala as a representation of a joint experience of the project. This stage was supported by professional artists who summarised the mandala in the illustration. Results The mandala is the artistic presentation of the findings from the evaluation. It is a shared journey between everyone involved. We divided it into six related layers. Starting from inside layers present the following experiences (1) public contributors had space to build confidence in a new topic, (2) relationships between individuals and within the project, (3) working remotely during the COVID-19 pandemic, (4) motivation that influenced people to become involved in this particular piece of work, (5) requirements that co-production needs to be inclusive and accessible to everyone, (6) expectations towards data science and artificial intelligence that researchers should follow to establish public support. Conclusions The participatory evaluation suggests that co-production around data science and artificial intelligence can be a meaningful process that is co-owned by everyone involved.

Abstract Background Recent studies mention a need to investigate partnership roles and dynamics within patient and public involvement and engagement (PPIE) in health research, and how impact and outcomes are achieved. Many labels exist to describe involvement processes, but it is unknown whether the label has implications on partnerships and outcomes. This rapid review investigates how roles between patients, relatives and researchers in a broad variety of PPIE activities in health research are described in peer reviewed papers and explores what enables these partnerships. Methods Rapid review of articles published between 2012 and February 2022 describing, evaluating, or reflecting on experiences of PPIE in health research. All research disciplines and research areas were eligible. Four databases (Medline, Embase, PsychInfo and CINAHL) were searched between November 2021 and February 2022. We followed PRISMA guidelines and extracted descriptive factors: year, origin, research area and discipline, study focus, framework used and co-authorship. On a selection of articles, we performed a narrative analysis of partnership roles using Smits et al.’s. Involvement Matrix. Lastly, we performed a meta synthesis of reported enablers and outcomes of the partnerships. Patients and Relatives (PRs) have been involved in the whole rapid review process and are co-authors of this article. Results Seventy articles from various research disciplines and areas were included. Forty articles were selected for a narrative analysis of the role description of PRs and researchers, and a meta synthesis of enablers and outcomes. Most articles described researchers as decision-makers throughout the research cycle. PRs most often were partners when they were included as co-authors; they were mostly partners in the design, analysis, write-up, and dissemination stages. Enablers of partnerships included: PR training, personality of PRs and communication skills, trust, remuneration and time. Conclusions Researchers’ decision-making roles gives them control of where and when to include PRs in their projects. Co-authorship is a way of acknowledging patients’ contributions which may lead to legitimation of their knowledge and the partnership. Authors describe common enablers, which can help future partnership formation.

Abstract Background In 2020, the Health Resources and Services Administration’s HIV/AIDS Bureau funded an initiative to promote implementation of rapid antiretroviral therapy initiation in 14 HIV treatment settings across the U.S. The goal of this initiative is to accelerate uptake of this evidence-based strategy and provide an implementation blueprint for other HIV care settings to reduce the time from HIV diagnosis to entry into care, for re-engagement in care for those out of care, initiation of treatment, and viral suppression. As part of the effort, an evaluation and technical assistance provider (ETAP) was funded to study implementation of the model in the 14 implementation sites. Method The ETAP has used implementation science methods framed by the Dynamic Capabilities Model integrated with the Conceptual Model of Implementation Research to develop a Hybrid Type II, multi-site mixed-methods evaluation, described in this paper. The results of the evaluation will describe strategies associated with uptake, implementation outcomes, and HIV-related health outcomes for patients. Discussion This approach will allow us to understand in detail the processes that sites to implement and integrate rapid initiation of antiretroviral therapy as standard of care as a means of achieving equity in HIV care.

Abstract Background Stated principles in government policy documents serve as a set of values outlining how governments intend to work. As such, health planning principles should be reflected in health policy across the cycle of planning, implementation and evaluation. Such principles should be reflected in the process of governments commissioning and funding evaluation, and in the work of those commissioned to do evaluation on behalf of governments. Methods We reviewed health planning policy documents to identify principles Australian State and Territory and National governments stated as being important to the work they do within Aboriginal and Torres Strait Islander health contexts. Evaluation tenders and reports relating to Aboriginal and Torres Strait Islander health policy, programs and service for the period 1-Jan-2007 to 1-Jan-2017 were retrieved and assessed as to whether they embedded principles governments state as important. Results In Aboriginal and Torres Strait Islander health planning policy contexts, Australian governments outline shared responsibility, cultural competence, engagement, partnership, capacity building, equity, a holistic concept of health, accountability, and evidence-based as fundamental principles that will underpin the work they will do. In total, we identified 390 publicly advertised evaluation tenders, but were only able to retrieve 18 tenders and 97 reports. Despite strong rhetoric placing importance on the abovementioned principles, these were not consistently embedded in tenders released by government commissioners, nor in reports largely commissioned by governments. Principles most widely incorporated in documents were those corresponding to Closing the Gap - accountability, evidence-based and equity. Principles of holistic concept of health, capacity building, cultural competence and partnership do not appear well applied in evaluation practice. Conclusion Notwithstanding the tensions and criticism of current practice that sees dominant governments policing Aboriginal and Torres Strait Islander populations and defining what principles should inform health policy and evaluation practice, this paper reveals shortcomings in current evaluation practice. Firstly, this paper reveals a lack of transparency about current practice, with only 2% of tenders and 25% of reports in the public domain. Secondly, this paper reveals that governments do not ‘walk the talk’, particularly when it comes to principles relating to Aboriginal participation in health.

Plain English summary Patient and public involvement (PPI) improves the quality of health research and ensures that research is relevant to patients’ needs. Though PPI is increasingly evident in clinical and health services research, there are few examples in the research literature of effective PPI in translational and laboratory-based research. In this paper, we describe the development and evaluation of PPI in a multi-centre European project (EuroTEAM – Towards Early biomarkers in Arthritis Management) that included both translational and laboratory-based and psychosocial research. We found that although most PPI in EuroTEAM was centred around the psychosocial research, there were examples of PPI in the laboratory studies. As the project evolved, researchers became better at accommodating PPI and identifying PPI opportunities. It was generally agreed that PPI had a positive impact on the project overall, particularly on public engagement with the research. We concluded that the inclusion of both psychosocial and laboratory-based research in the same project facilitated PPI across all aspects of the research. In future projects, we would try to specify individual PPI activities in more detail at the project-planning stage, and better accommodate patient partners who are not native speakers of English. Abstract Background Patient and public involvement (PPI) enhances research quality and relevance and is central to contemporary health policy. The value of PPI has been recognised in rheumatology research, though there are limited examples of PPI in basic and translational science. The EU FP7 funded ‘EuroTEAM’ (Towards Early biomarkers in Arthritis Management) project was established to develop biomarker-based approaches to predict the future development of rheumatoid arthritis and incorporated psychosocial research to investigate the perceptions of ‘at risk’ individuals about predictive testing, and to develop informational resources about rheumatoid arthritis (RA) risk. Patient involvement was central to EuroTEAM from the inception of the project. The objective of this paper is to describe the development of PPI in EuroTEAM, formatively assess the impact of PPI from the perspectives of researchers and patient research partners (PRPs), reflect on successes and lessons learned, and formulate recommendations to guide future projects. Methods Two mixed-methods surveys (for PRPs and researchers) and a teleconference were undertaken to assess the impact of PPI on individual work packages and on EuroTEAM overall. Results There was consensus about the positive impact of PPI on the research and on the experiences of those involved. In particular, the positive impact of PPI on the personal development of researchers, and on effective public engagement with EuroTEAM research were highlighted. Researchers described adapting their practice in future projects to facilitate PPI. Spin-off projects and ongoing collaborations between PRPs and researchers reflected the value of PPI to participants. PPI was more frequently integrated in psychosocial research, though examples of PPI in laboratory/translational science were also described. PRPs asked for more opportunities to contribute meaningfully to basic scientific research and for more extensive feedback on their contributions. Conclusions The findings were used to formulate recommendations to guide effective involvement of patients in future similar projects, including identifying specific training requirements for PRPs and researchers, the identification of PRP focused tasks/deliverables at the project planning stage, and supporting access to involvement for all PRPs. Importantly, the distinctive multidisciplinary approach of EuroTEAM, incorporating both basic science and psychosocial research, facilitated patient involvement in the project overall.

Abstract Background Clinical pathways are one of the main tools to manage the health care’s quality and concerned with the standardization of care processes. They have been used to help frontline healthcare workers by presenting summarized evidence and generating clinical workflows involving a series of tasks performed by various people within and between work environments to deliver care. Integrating clinical pathways into Clinical Decision Support Systems (CDSSs) is a common practice today. However, in a low-resource setting (LRS), this kind of decision support systems is often not readily accessible or even not available. To fill this gap, we developed a computer aided CDSS that swiftly identifies which cases require a referral and which ones may be managed locally. The computer aided CDSS is designed primarily for use in primary care settings for maternal and childcare services, namely for pregnant patients, antenatal and postnatal care. The purpose of this paper is to assess the user acceptance of the computer aided CDSS at the point of care in LRSs. Methods For evaluation, we used a total of 22 parameters structured in to six major categories, namely “ease of use, system quality, information quality, decision changes, process changes, and user acceptance.” Based on these parameters, the caregivers from Jimma Health Center's Maternal and Child Health Service Unit evaluated the acceptability of a computer aided CDSS. The respondents were asked to express their level of agreement using 22 parameters in a think-aloud approach. The evaluation was conducted in the caregiver's spare-time after the clinical decision. It was based on eighteen cases over the course of two days. The respondents were then asked to score their level of agreement with some statements on a five-point scale: strongly disagree, disagree, neutral, agree, and strongly agree. Results The CDSS received a favorable agreement score in all six categories by obtaining primarily strongly agree and agree responses. In contrast, a follow-up interview revealed a variety of reasons for disagreement based on the neutral, disagree, and strongly disagree responses. Conclusions Though the study had a positive outcome, it was limited to the Jimma Health Center Maternal and Childcare Unit, and hence a wider scale evaluation and longitudinal measurements, including computer aided CDSS usage frequency, speed of operation and impact on intervention time are needed.

Abstract Background Young people with a sibling or parent who experiences mental health challenges have their own support needs. Most programs designed for this population lack a strong evidence base, and the involvement of young people in the development and evaluation of programs designed to support them is unclear or lacking. Methods This paper describes a protocol for a mixed methods, longitudinal, collaborative evaluation of a suite of programs delivered by The Satellite Foundation, a not-for-profit organisation for young people (5–25 years) who have a family member with mental health challenges. Young people’s lived experience and knowledge will guide the research approach. Institutional ethics approval has been obtained. Over a three year period, approximately 150 young people will be surveyed online on various wellbeing outcome measures, prior to, six and twelve months following program participation with data analysed using multi-level modelling. Groups of young people will be interviewed after participating in different Satellite programs each year. An additional group of young people will be interviewed individually over time. Transcripts will be analysed using thematic analysis. Young people’s creative artworks on their experiences will be included as part of the evaluation data. Discussion This novel, collaborative evaluation will provide vital evidence on young people’s experiences and outcomes during their time with Satellite. Findings will be used to inform future program development and policy. The approach used here may guide other researchers engaging in collaborative evaluations with community organisations.

Abstract Background The COVID-19 pandemic changed the way we work, spend, live, and learn. The impact was felt in the health sector where hospitals cancelled elective surgery, put on hold outpatient services, and implemented new social distancing procedures and telehealth systems, to enable hospitals to increase bed capacity. For medical students, these factors meant significant disruption to their clinical placements, remote delivery of their education, cessation of international and interstate placements, complicated by significant travel restrictions and border closures. There were concerns that final year students might be unable to graduate that year due to this lack of clinical exposure. Innovation As a result of this disruption in late March 2020 we developed an innovative 6 week ‘COVID-19 e-lective’ rotation, consisting of online modules, virtual clinical tutorials and a COVID project totalling the equivalent of 200 h of work. Results An evaluation was undertaken that found it to be remarkably successful in meeting the students’ learning needs and alleviating concerns about disrupted placements. It was also conducted during 2021 for all Year 4 students to help expand clinical placement opportunities. Outcomes This paper describes the e-lective, its innovations, its challenges, and its evaluation findings, for others to learn from.

Abstract Background Scientometric studies are one of the most important and useful tools to assess the research performance and knowledge impact of researchers. The aim of this study was to map out the scientific performance of the Iranian medical academics with respect to a detailed range of scientometric indicators. Methods Using scientometric approach, individual and scientific performance data of medical academic staff were extracted from the Iranian Scientometric Information Database (ISID). Total number of publications, total number of citations, citation per paper, h-index, international collaboration, self-citation, SJR decile, i10-Index, Quartile distribution were the studied scientometric variables. Out of the registered 19,023 academic staff, 746 were included in the study through simple random sampling method using random sample extraction function in STATA. Data were analyzed using STATA 14 statistical software package. Results Most of the included academicians were men (60%). A total of 13,682 articles were published by them until 2018, being cited 114,928 times with a mean of 5.77 citation per paper. H-index median was three and about 90% of the staff had an H-index below 10. Number of published papers, cite per paper and H-index metrics were significantly different with respect to gender, academic position/degree, and general field of study (p

Abstract Background Primary care reform has been on the political agenda in Canada and many industrialized countries for several decades; it is widely seen as the foundation for broader health system transformation. Federal investments in primary care, including major cash transfers to provinces and territories as part of a 10-year health care funding agreement in 2004, triggered waves of primary care reform across Canada. Nevertheless, Commonwealth Fund surveys show, Canada continues to lag behind other industrialized nations with respect to timely access to care, electronic medical record use and audit and feedback for quality improvement in primary care. This paper evaluates the pace and direction of primary care reform as well as the extent of resulting change in the organization and delivery of primary care in Ontario, Canada’s most populous province. Methods Qualitative and quantitative methods were used for this study. A literature review was conducted to analyze the core dimensions of primary care reform, the history of reform in Ontario, and the extent to which different dimensions are integrated into Ontario’s models. Quantitative data on the number of family physicians/general practitioners and patients enrolled in these models was examined over a 10-year period to determine the degree of change that has taken place in the organization and delivery of primary care in Ontario. Results There are 11 core reform dimensions that individually and collectively shift from conventional primary care toward the more expansive vision of primary health care. Assessment of Ontario’s models against these core dimensions demonstrate that there has been little substantive change in the organization and delivery of primary care over 10 years in Ontario. Conclusions Primary care reform is a multi-dimensional construct with different reform models bundling core dimensions in different ways. This understanding is important to move beyond the rhetoric of “reform” and to critically assess the pace and direction of change in primary care in Ontario and in other jurisdictions. The conceptual framework developed in this paper can assist decision-makers, academics and health care providers in all jurisdictions in evaluating the pace of change in the primary care sector, as well as other sectors.

Abstract Background While there has been progress in controlling the HIV epidemic, HIV still remains a disease of global concern. Some of the progress has been attributed to increased public awareness and uptake of public health interventions, as well as increased access to anti- retroviral treatment and the prevention of vertical HIV transmission. These interventions would not have been possible without substantial investments in HIV programs. However, donor fatigue introduces the need for low income countries to maximize the benefits of the available resources. This necessitates identification of priorities that should be funded. Evaluating prioritization processes would enable decision makers to assess the effectiveness of their processes, thereby designing intervention strategies. To date most evaluations have focused on cost-benefit analyses, which overlooks additional critical impacts of priority setting decisions. Kapiriri & Martin (2010) developed and validated a comprehensive framework for evaluating PS in low income countries. The objective of this paper report findings from a comprehensive evaluation of priority setting for HIV in Uganda, using the framework; and to identify lessons of good practice and areas for improvement. Methods This was a qualitative study based on forty interviews with decision makers and policy document review. Data were analysed using INVIVO 10, and based on the parameters in Kapiriri et al’s evaluation framework. Results We found that HIV enjoys political support, which contributes to the availability of resources, strong planning institutions, and participatory prioritization process based on some criteria. Some of the identified limitations included; undue donor and political influence, priorities not being publicized, and lack of mechanisms for appealing the decisions. HIV prioritization had both positive and negative impacts on the health system. Conclusions The framework facilitated a more comprehensive evaluation of HIV priority setting. While there were successful areas, the process could be strengthened by minimizing undue influence of external actors, and support the legitimate institutions to set priorities and implement them. These should also institute mechanisms for publicizing the decisions, appeals and increased accountability. While this paper looked at HIV, the framework is flexible enough to be used in evaluating priority setting for other health programs within similar context.

Abstract Background Alternative models of cancer follow-up care are needed to ameliorate pressure on services and better meet survivors’ long-term needs. This paper reports an evaluation of a service improvement initiative for the follow-up care of prostate cancer patients based on remote monitoring and supported self-management. Methods This multi-centred, historically controlled study compared patient reported outcomes of men experiencing the new Programme with men experiencing a traditional clinic appointment model of follow-up care, who were recruited in the period immediately prior to the introduction of the Programme. Data were collected by self-completed questionnaires, with follow up measurement at four and eight months post-baseline. The primary outcome was men’s unmet survivorship needs, measured by the Cancer Survivors’ Unmet Needs Survey. Secondary outcomes included cancer specific quality of life, psychological wellbeing and satisfaction with care. The analysis was intention to treat. Regression analyses were conducted for outcomes at each time point separately, controlling for pre-defined clinical and demographic variables. All outcome analyses are presented in the paper. Costs were compared between the two groups. Results Six hundred and twenty-seven men (61%) were consented to take part in the study (293 in the Programme and 334 in the comparator group.) Regarding the primary measure of unmet survivorship needs, 25 of 26 comparisons favoured the Programme, of which 4 were statistically significant. For the secondary measures of activation for self-management, quality of life, psychological well-being and lifestyle, 20 of 32 comparisons favoured the Programme and 3 were statistically significant. There were 22 items on the satisfaction with care questionnaire and 13 were statistically significant. Per participant costs (British pounds, 2015) in the 8 month follow up period were slightly lower in the programme than in the comparator group (£289 versus £327). The Programme was acceptable to patients. Conclusion The Programme is shown to be broadly comparable to traditional follow-up care in all respects, adding to evidence of the viability of such models.

Abstract Background Peer-research is steered and conducted by people with lived experience of the issues being researched. This paper explores the value of peer-research in two complex public health intervention evaluations in the UK. Methods Reports from 18 peer research projects, completed by residents from 12 communities in the UK taking part in two community empowerment interventions, were analysed using cross-case analysis. Results Undertaking peer research helped to build the evaluation and research skills within individual projects as well as providing data on other outcomes related to the programmes Theory of Change. Some peer researchers, however, felt unprepared for the activity despite support from the academic team and were unsatisfied with project outcomes. While peer research projects provided more opportunities for local residents to engage with the overall evaluations, there was an overreliance on people closely connected to the programmes to be peer researchers. The peer research projects explored topics that were broader than the aims and objectives of the overall programme evaluations. All provided insight into the context in which projects occurred, while some also informed understanding of programme change mechanisms. Conclusions Including peer research as part of complex public health intervention evaluations can help uncover important contextual and ecological details beyond the reach of more traditional evaluation data collection. Peer research can also empower and build research/evaluation capacity within communities, which is particularly pertinent for community empowerment interventions.

Abstract Background This paper explores the feasibility of delivering a music festival-based drug checking service in Australia, evaluating service design and stakeholder acceptability. Methods Questionnaire and interview data were collected from adult service users and key stakeholders. A mixed methods approach was used to analyse the data on implementation, impact and acceptability. Results The trial service tested 170 substances with more than 230 patrons (including individuals who attended in groups). Adult service users had an average age of 21 years. Voluntary participation in the evaluation resulted in 158 participants completing the pre-service questionnaire, most of whom also completed the post-service (147 participants). Eleven in-depth qualitative interviews were conducted with patrons in the weeks following the drug checking. Concordance between what the patron expected the drug to be and drug checking results occurred in 88 per cent (n = 139) of the sample. Evaluation results show that the experience of testing and the accompanying harm reduction brief interventions positively impacted on patrons’ self-reported drug harm reduction knowledge, trust of health providers and stated drug use intentions. The service was received positively by service users. Conclusion This is the first independent evaluation of a pilot drug checking service in Australia. Consideration of operational feasibility and self-reported behavioural change suggests that the program was successful, although communication about the interpretation of drug checking results could be improved. Future studies should develop strategies for follow-up and consider the applicability of behavioural change theory.

Abstract Psychometrically sound resilience outcome measures are essential to establish how health and care services or interventions can enhance the resilience of people living with dementia (PLWD) and their carers. This paper systematically reviews the literature to identify studies that administered a resilience measurement scale with PLWD and/or their carers and examines the psychometric properties of these measures. Electronic abstract databases and the internet were searched, and an international network contacted to identify peer-reviewed journal articles. Two authors independently extracted data. They critically reviewed the measurement properties from the available psychometric data in the studies, using a standardised checklist adapted for purpose. Fifty-one studies were included in the final review, which applied nine different resilience measures, eight developed in other populations and one developed for dementia carers in Thailand. None of the measures were developed for use with people living with dementia. The majority of studies (N = 47) focussed on dementia carers, three studies focussed on people living with dementia and one study measured both carers and the person with dementia. All the studies had missing information regarding the psychometric properties of the measures as applied in these two populations. Nineteen studies presented internal consistency data, suggesting seven of the nine measures demonstrate acceptable reliability in these new populations. There was some evidence of construct validity, and twenty-eight studies hypothesised effects a priori (associations with other outcome measure/demographic data/differences in scores between relevant groups) which were partially supported. The other studies were either exploratory or did not specify hypotheses. This limited evidence does not necessarily mean the resilience measure is not suitable, and we encourage future users of resilience measures in these populations to report information to advance knowledge and inform further reviews. All the measures require further psychometric evaluation in both these populations. The conceptual adequacy of the measures as applied in these new populations was questionable. Further research to understand the experience of resilience for people living with dementia and carers could establish the extent current measures -which tend to measure personal strengths -are relevant and comprehensive, or whether further work is required to establish a new resilience outcome measure.

Abstract Background Research capacity development (RCD) is considered fundamental to closing the evidence–practice gap, thereby contributing to health, wealth and knowledge for practice. Numerous frameworks and models have been proposed for RCD, but there is little evidence of what works for whom and under what circumstances. There is a need to identify mechanisms by which candidate interventions or clusters of interventions might achieve RCD and contribute to societal impact, thereby proving meaningful to stakeholders. Methods A realist synthesis was used to develop programme theories for RCD. Structured database searches were conducted across seven databases to identify papers examining RCD in a health or social care context (1998–2013). In addition, citation searches for 10 key articles (citation pearls) were conducted across Google Scholar and Web of Science. Of 214 included articles, 116 reported on specific interventions or initiatives or their evaluation. The remaining 98 articles were discussion papers or explicitly sought to make a theoretical contribution. A core set of 36 RCD theoretical and conceptual papers were selected and analysed to generate mechanisms that map across macro contexts (individual, team, organisational, network). Data were extracted by means of ‘If-Then’ statements into an Excel spreadsheet. Models and frameworks were deconstructed into their original elements. Results Eight overarching programme theories were identified featuring mechanisms that were triggered across multiple contexts. Three of these fulfilled a symbolic role in signalling the importance of RCD (e.g. positive role models, signal importance, make a difference), whilst the remainder were more functional (e.g. liberate talents, release resource, exceed sum of parts, learning by doing and co-production of knowledge). Outcomes from one mechanism produced changes in context to stimulate mechanisms in other activities. The eight programme theories were validated with findings from 10 systematic reviews (2014–2017). Conclusions This realist synthesis is the starting point for constructing an RCD framework shaped by these programme theories. Future work is required to further test and refine these findings against empirical data from intervention studies.

Abstract Objective The objective of this paper is to demonstrate the utility of the CFIR framework for evaluating broad-scale change by discussing the challenges to be addressed when planning the assessment of broad-scale change and the solutions developed by the evaluation team to address those challenges. The evaluation of implementation of Patient-centered Care and Cultural Transformation (PCC&CT) within the Department of Veterans Affairs (VA) will be used as a demonstrative example. Patient-Centered Care (PCC) is personalized health care that considers a patient’s circumstances and goals. The Department of Veterans Affairs (VA) is working towards implementing PCC throughout its healthcare system, comprised of multiple interventions with a singular long-term goal of cultural transformation, however little is known about the factors influencing its implementation. This paper discusses the issues that arose using CFIR to qualitatively assess the factors influencing implementation of cultural transformation. Results Application of CFIR to this broad-scale evaluation revealed three strategies recommended for use in evaluating implementation of broad-scale change: (1) the need for adapted definitions for CFIR constructs (especially due to new application to broad-scale change), (2) the use of a mixed deductive-inductive approach with thematic coding to capture emergent themes not encompassed by CFIR, and (3) its use for expedited analysis and synthesis for rapid delivery of findings to operational partners. This paper is among the first to describe use of CFIR to guide the evaluation of a broad-scale transformation, as opposed to discrete interventions. The processes and strategies described in this paper provide a detailed example and structured approach that can be utilized and expanded upon by others evaluating implementation of broad-scale evaluations. Although CFIR was the framework selected for this evaluation, the strategies described in this paper including: use of adapted definitions, use of mixed deductive-inductive approach, and the approach for expedited analysis and synthesis can be transferred and tested with other frameworks.

Abstract Background Transition into higher education has been identified as one of the most stressful periods for learners. Interventions targeting the transition phase such as near- peer mentoring might help address some of these challenges. We were however unable to identify a published systematic review of the literature describing outcomes of near-peer mentoring of medical students during the transition phase into medical school. The aim of this paper is to review the literature and describe the outcomes of near-peer mentoring schemes for first-year medical students in the transition phase. Methods A search of different electronic databases was carried out, using the search terms peer, buddy, mentor*, counsel*, advise*, tutor*, student, medical, school. 1861 articles were identified, however only 5 studies met the inclusion criteria- primary mentees should be first-years, and mentors must be inclusive of second-years but not limited to them. In reporting this paper, the PRISMA guidelines were followed. Results Published material on near-peer mentoring for medical students is scarce. Three outcomes for peer mentoring were identified- professional and personal development, stress reduction, and ease of transitioning. Incidentally, peer-mentoring was also found to have facilitated the development of personal and professional attitudes in the mentors. The quality of the evaluation methods in the studies was however low to moderate. Conclusion Near-peer-mentoring is a way of promoting professional and personal development. It is also promising to aid transition and maintain well-being of first-year medical students. However, larger, better quality longitudinal studies, are needed to ascertain its true value for these students.

Abstract Background and aim Recent UK policy has focussed on improving support for victims of domestic violence and abuse (DVA), in healthcare settings. DVA victims attending hospital are often at highest risk of harm, yet DVA support in hospitals has been inadequate. A targeted service supporting high risk DVA victims, was implemented at a hospital Trust in North West England. The service was provided by Independent Domestic Violence Advisors (IDVAs). This paper assesses the activity in the hospital-based IDVA service during the COVID-19 pandemicand addresses the research questions: What was the demand for the service? How did the service respond? What facilitated this response? Methods A mixed-methods study was undertaken. Quantitative data on referrals to the service were examined using simple descriptive statistics and compared to other DVA services. Semi-structured interviews were undertaken with IDVAs and other hospital staff involved with the service and the data subjected to thematic analysis. Results The quantitative analysis showed that referrals dropped at the start of lockdown, then increased and continued to rise; the qualitative findings reiterated this pattern. Referrals came from a range of departments across the Trust, with the majority from A&E. Pre-pandemic, the population supported by the service included higher proportions of males and people aged 40 and over than at other IDVA services; this continued during the pandemic. The qualitative findings indicated a flexible response during the pandemic, enabled by strong working relationships and by using workarounds. Conclusions The hospital-based IDVAs provided an efficient, flexible serviceduring the COVID-19 pandemic. Referrals increased during the first lockdown and subsequent relaxing of restrictions. Locating the IDVAs within a team working across the organisation, and building good working relationships facilitated an effective disclosure and referral route, which endured through social restrictions. The IDVAs supported high-risk victims who may otherwise not have been identified in traditional community-based DVA settings during the pandemic. Hospital-based IDVA services can broaden access by supporting vulnerable, at risk populations whose needs may not be identified at other services.

Abstract Background Given the magnitude and frequency of conflicts in the MENA region along with their devastating impact on health responses and outcomes, there exists a strong need to invest in contextualized, innovative, and accessible capacity building approaches to enhance leadership and skills in global health. The MENA region suffers from limited (1) continued educational and career progression opportunities, (2) gender balance, and (3) skill-mix among its health workforce, which require significant attention. Main text The Global Health Institute at the American University of Beirut incepted the Academy division to develop and implement various global health capacity building (GHCB) initiatives to address those challenges in fragile settings across low-and middle-income countries in the MENA region. These initiatives play a strategic role in this context, especially given their focus on being accessible through employing innovative learning modalities. However, there exists a dearth of evidence-based knowledge on best practices and recommendations to optimize the design, implementation, and evaluation of GHCB in fragile settings in the MENA region. The present paper describes the development of the evaluation of capacity building program (eCAP), implemented under the Academy division, to assess the effectiveness of its initiatives. eCAP is composed of 3 phases: (1) a situational assessment, followed by (2) production of multiple case studies, and finally (3) a meta-assessment leading to model development. The goal of eCAP is not only to inform the Academy’s operations, but also to synthesize produced knowledge into the formation of an evidence-based, scalable, and replicable model for GHCB in fragile settings. Conclusion eCAP is an important initiative for researchers, educators, and practitioners interested in GHCB in fragile settings. Several lessons can be learned from the outcomes it has yielded so far in its first two phases of implementation, ranging from the situational assessment to the production of evaluation case studies, which are expanded on in the manuscript along with pertinent challenges.

Abstract Background Latin America and the Caribbean (LAC) are among those regions most affected by the COVID-19 pandemic worldwide. The COVID-19 pandemic has strained health systems in the region. In this context of severe healthcare resource constraints, there is a need for systematic priority-setting to support decision-making which ensures the best use of resources while considering the needs of the most vulnerable groups. The aim of this paper was to provide a critical description and analysis of how health systems considered priority-setting in the COVID-19 response and preparedness plans of a sample of 14 LAC countries; and to identify the associated research gaps. Methods A documentary analysis of COVID-19 preparedness and response plans was performed in a sample of 14 countries in the LAC region. We assessed the degree to which the documented priority-setting processes adhered to established quality indicators of effective priority-setting included in the Kapiriri and Martin framework. We conducted a descriptive analysis of the degree to which the reports addressed the quality parameters for each individual country, as well as a cross-country comparison to explore whether parameters varied according to independent variables. Results While all plans were led and supported by the national governments, most included only a limited number of quality indicators for effective priority-setting. There was no systematic pattern between the number of quality indicators and the country’s health system and political contexts; however, the countries that had the least number of quality indicators tended to be economically disadvantaged. Conclusion This study adds to the literature by providing the first descriptive analysis of the inclusion of priority-setting during a pandemic, using the case of COVID-19 response and preparedness plans in the LAC region. The analysis found that despite the strong evidence of political will and stakeholder participation, none of the plans presented a clear priority-setting process, or used a formal priority-setting framework, to define interventions, populations, geographical regions, healthcare setting or resources prioritized. There is need for case studies that analyse how priority-setting actually occurred during the COVID-19 pandemic and the degree to which the implementation reflected the plans and the parameters of effective priority-setting, as well as the impact of the prioritization processes on population health, with a focus on the most vulnerable groups.

Abstract Background There is increasing awareness among researchers and policymakers of the potential for healthcare interventions to have consequences beyond those initially intended. These unintended consequences or “spillover effects” result from the complex features of healthcare organisation and delivery and can either increase or decrease overall effectiveness. Their potential influence has important consequences for the design and evaluation of implementation strategies and for decision-making. However, consideration of spillovers remains partial and unsystematic. We develop a comprehensive framework for the identification and measurement of spillover effects resulting from changes to the way in which healthcare services are organised and delivered. Methods We conducted a scoping review to map the existing literature on spillover effects in health and healthcare interventions and used the findings of this review to develop a comprehensive framework to identify and measure spillover effects. Results The scoping review identified a wide range of different spillover effects, either experienced by agents not intentionally targeted by an intervention or representing unintended effects for targeted agents. Our scoping review revealed that spillover effects tend to be discussed in papers only when they are found to be statistically significant or might account for unexpected findings, rather than as a pre-specified feature of evaluation studies. This hinders the ability to assess all potential implications of a given policy or intervention. We propose a taxonomy of spillover effects, classified based on the outcome and the unit experiencing the effect: within-unit, between-unit, and diagonal spillover effects. We then present the INTENTS framework: Intended Non-intended TargEted Non-Targeted Spillovers. The INTENTS framework considers the units and outcomes which may be affected by an intervention and the mechanisms by which spillover effects are generated. Conclusions The INTENTS framework provides a structured guide for researchers and policymakers when considering the potential effects that implementation strategies may generate, and the steps to take when designing and evaluating such interventions. Application of the INTENTS framework will enable spillover effects to be addressed appropriately in future evaluations and decision-making, ensuring that the full range of costs and benefits of interventions are correctly identified.

Abstract Background Systems approaches are currently being advocated and implemented to address complex challenges in Public Health. These approaches work by bringing multi-sectoral stakeholders together to develop a collective understanding of the system, and then to identify places where they can leverage change across the system. Systems approaches are unpredictable, where cause-and-effect cannot always be disentangled, and unintended consequences – positive and negative – frequently arise. Evaluating such approaches is difficult and new methods are warranted. Methods Ripple Effects Mapping (REM) is a qualitative method which can capture the wider impacts, and adaptive nature, of a systems approach. Using a case study example from the evaluation of a physical activity-orientated systems approach in Gloucestershire, we: a) introduce the adapted REM method; b) describe how REM was applied in the example; c) explain how REM outputs were analysed; d) provide examples of how REM outputs were used; and e) describe the strengths, limitations, and future uses of REM based on our reflections. Results Ripple Effects Mapping is a participatory method that requires the active input of programme stakeholders in data gathering workshops. It produces visual outputs (i.e., maps) of the programme activities and impacts, which are mapped along a timeline to understand the temporal dimension of systems change efforts. The REM outputs from our example were created over several iterations, with data collected every 3–4 months, to build a picture of activities and impacts that have continued or ceased. Workshops took place both in person and online. An inductive content analysis was undertaken to describe and quantify the patterns within the REM outputs. Detailed guidance related to the preparation, delivery, and analysis of REM are included in this paper. Conclusion REM may help to advance our understanding and evaluation of complex systems approaches, especially within the field of Public Health. We therefore invite other researchers, practitioners and policymakers to use REM and continuously evolve the method to enhance its application and practical utility.