Search

Background: Much has been written about the use of evidence in policy; however, there is still little known about whether and how research is engaged with and used in policy development or the impact of reported barriers and facilitators. This paper aims to (1) describe the characteristics of 131 policy documents, (2) describe the ways in which research was engaged with (e.g. was searched for, appraised or generated) and used (e.g. to clarify understanding, persuade others or inform a policy) in the development of these policy documents, and (3) identify the most commonly reported barriers and facilitators and describe their association with research engagement and use., Methods: Six health policy and program development agencies based in Sydney, Australia, contributed four recently finalised policy documents for consideration over six measurement periods. Structured, qualitative interviews were conducted with the policymakers most heavily involved in developing each of the 131 policy documents. Interviews covered whether and how research was engaged with and used in the development of the policy product and any barriers or facilitators related to this. Interviews were scored using the empirically validated SAGE tool and thematically analysed. Descriptive statistics were calculated for all key variables and comparisons made between agencies. Multiple regression analyses were used to estimate the impact of specific barriers and facilitators on research engagement and use., Results: Our data shows large variations between policy agencies in the types of policy documents produced and the characteristics of these documents. Nevertheless, research engagement and use was generally moderate across agencies. A number of barriers and facilitators to research use were identified. No barriers were significantly associated with any aspects of research engagement or use. Access to consultants and relationships with researchers were both associated with increased research engagement but not use. Thus, access to consultants and relationships with researchers may increase the extent and quality of the evidence considered in policy development., Conclusions: Our findings suggest that those wishing to develop interventions and programs designed to improve the use of evidence in policy agencies might usefully target increasing access to consultants and relationships with researchers in order to increase the extent and quality of the research considered, but that a greater consideration of context might be required to develop strategies to increase evidence use.

Background: This paper describes the trial of a novel intervention, Supporting Policy In health with evidence from Research: an Intervention Trial (SPIRIT). It examines (1) the feasibility of delivering this kind of programme in practice; (2) its acceptability to participants; (3) the impact of the programme on the capacity of policy agencies to engage with research; and (4) the engagement with and use of research by policy agencies., Methods: SPIRIT was a multifaceted, highly tailored, stepped-wedge, cluster-randomised, trial involving six health policy agencies in Sydney, Australia. Agencies were randomly allocated to one of three start dates to receive the 1-year intervention programme. SPIRIT included audit, feedback and goal setting; a leadership programme; staff training; the opportunity to test systems to facilitate research use in policies; and exchange with researchers. Outcome measures were collected at each agency every 6 months for 30 months., Results: Participation in SPIRIT was associated with significant increases in research use capacity at staff and agency levels. Staff reported increased confidence in research use skills, and agency leaders reported more extensive systems and structures in place to support research use. Self-report data suggested there was also an increase in tactical research use among agency staff. Given the relatively small numbers of participating agencies and the complexity of their contexts, findings suggest it is possible to effect change in the way policy agencies approach the use of research. This is supported by the responses on the other trial measures; while these were not statistically significant, on 18 of the 20 different measures used, the changes observed were consistent with the hypothesised intervention effect (that is, positive impacts)., Conclusions: As an early test of an innovative approach, SPIRIT has demonstrated that it is possible to increase research engagement and use in policy agencies. While more work is needed to establish the replicability and generalisability of these findings, this trial suggests that building staff skills and organisational structures may be effective in increasing evidence use.

Background: Cross-sectoral, multidisciplinary partnership research is considered one of the most effective means of facilitating research-informed policy and practice, particularly for addressing complex problems such as chronic disease. Successful research partnerships tend to be underpinned by a range of features that enable knowledge mobilisation (KMb), seeking to connect academic researchers with decision-makers and practitioners to improve the nature, quality and use of research. This paper contributes to the growing discourse on partnership approaches by illustrating how knowledge mobilisation strategies are operationalised within the Australian Prevention Partnership Centre (the Centre), a national collaboration of academics, policy-makers and practitioners established to develop systems approaches for the prevention of lifestyle-related chronic diseases., Methods: We undertook interviews with key academics, policy, and practice partners and funding representatives at the mid-point of the Centre's initial 5-year funding cycle. We aimed to explore how the Centre is functioning in practice, to develop a conceptual model of KMb within the Centre for use in further evaluation, and to identify ways of strengthening our approach to partnership research. Inductive and deductive thematic analysis was used to identify the key mechanisms underpinning the Centre's KMb approach., Results: Six key mechanisms appeared to facilitate KMb within our Centre, namely Engagement, Partnerships, Co-production, Capacity and Skills, Knowledge Integration, and Adaptive Learning and Improvement. We developed a conceptual model that articulated these mechanisms in relation to the structures and processes that support them, as well as the Centre's goals. Findings also informed adaptations designed to strengthen the Centre., Conclusions: Findings provide insights into the practical realities of operationalising KMb strategies within a research partnership. Overall, the centre is perceived to be progressing towards its KMb goals, but challenges include stakeholders from different settings understanding each other's contexts and working together effectively, and ensuring knowledge generated across different projects within the Centre is integrated into a more comprehensive understanding of chronic disease prevention policy and practice. Our conceptual model is now informing ongoing developmental evaluation activities within the Centre, where it is being tested and refined.

Background: Health policy-making can benefit from more effective use of research. In many policy settings there is scope to increase capacity for using research individually and organisationally, but little is known about what strategies work best in which circumstances. This review addresses the question: What causal mechanisms can best explain the observed outcomes of interventions that aim to increase policy-makers' capacity to use research in their work?, Methods: Articles were identified from three available reviews and two databases (PAIS and WoS; 1999-2016). Using a realist approach, articles were reviewed for information about contexts, outcomes (including process effects) and possible causal mechanisms. Strategy + Context + Mechanism = Outcomes (SCMO) configurations were developed, drawing on theory and findings from other studies to develop tentative hypotheses that might be applicable across a range of intervention sites., Results: We found 22 studies that spanned 18 countries. There were two dominant design strategies (needs-based tailoring and multi-component design) and 18 intervention strategies targeting four domains of capacity, namely access to research, skills improvement, systems improvement and interaction. Many potential mechanisms were identified as well as some enduring contextual characteristics that all interventions should consider. The evidence was variable, but the SCMO analysis suggested that tailored interactive workshops supported by goal-focused mentoring, and genuine collaboration, seem particularly promising. Systems supports and platforms for cross-sector collaboration are likely to play crucial roles. Gaps in the literature are discussed., Conclusion: This exploratory review tentatively posits causal mechanisms that might explain how intervention strategies work in different contexts to build capacity for using research in policy-making.

Background: Rapid reviews are increasingly used by policy agencies to access relevant research in short timeframes. Despite the growing number of programmes, little is known about how rapid reviews are used by health policy agencies. This study examined whether and how rapid reviews commissioned using a knowledge brokering programme were used by Australian policy-makers., Methods: This study used interview data to examine the use of 139 rapid reviews by health policy agencies that were commissioned between 2006 and 2015. Transcripts were coded to identify how rapid reviews were used, the type of policy processes in which they were used, what evidence of use was provided and what reasons were given when rapid reviews were not used. Fisher's exact test was used to assess variation between types of agencies., Results: Overall, 89% of commissioned rapid reviews were used by the commissioning agencies and 338 separate instances of use were identified, namely, on average, three uses per review. Policy-makers used reviews primarily to determine the details of a policy or programme, identify priorities for future action or investment, negotiate interjurisdictional decisions, evaluate alternative solutions for a policy problem, and communicate information to stakeholders. Some variation in use was observed across agencies. Reasons for non-use were related to changes in organisational structures, resources or key personnel in the commissioning agencies, or changes in the broader political environment., Conclusions: This study found that almost all rapid reviews had been used by the agencies who commissioned them, primarily in policy and programme development, agenda-setting, and to communicate information to stakeholders. Reviews were used mostly in instrumental and conceptual ways and there was little evidence of symbolic use. Variations in use were identified across agencies. The findings suggest that commissioned rapid reviews are an effective means of providing timely relevant research for use in policy processes and that review findings may be applied in a variety of ways.

Background: Few measures of research use in health policymaking are available, and the reliability of such measures has yet to be evaluated. A new measure called the Staff Assessment of Engagement with Evidence (SAGE) incorporates an interview that explores policymakers' research use within discrete policy documents and a scoring tool that quantifies the extent of policymakers' research use based on the interview transcript and analysis of the policy document itself. We aimed to conduct a preliminary investigation of the usability, sensitivity, and reliability of the scoring tool in measuring research use by policymakers., Methods: Nine experts in health policy research and two independent coders were recruited. Each expert used the scoring tool to rate a random selection of 20 interview transcripts, and each independent coder rated 60 transcripts. The distribution of scores among experts was examined, and then, interrater reliability was tested within and between the experts and independent coders. Average- and single-measure reliability coefficients were computed for each SAGE subscales., Results: Experts' scores ranged from the limited to extensive scoring bracket for all subscales. Experts as a group also exhibited at least a fair level of interrater agreement across all subscales. Single-measure reliability was at least fair except for three subscales: Relevance Appraisal, Conceptual Use, and Instrumental Use. Average- and single-measure reliability among independent coders was good to excellent for all subscales. Finally, reliability between experts and independent coders was fair to excellent for all subscales., Conclusions: Among experts, the scoring tool was comprehensible, usable, and sensitive to discriminate between documents with varying degrees of research use. Secondly, the scoring tool yielded scores with good reliability among the independent coders. There was greater variability among experts, although as a group, the tool was fairly reliable. The alignment between experts' and independent coders' ratings indicates that the independent coders were scoring in a manner comparable to health policy research experts. If the present findings are replicated in a larger sample, end users (e.g. policy agency staff) could potentially be trained to use SAGE to reliably score research use within their agencies, which would provide a cost-effective and time-efficient approach to utilising this measure in practice.

Background: An intervention's success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research., Methods: Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors., Results: Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention's overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study's assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) 'Getting good stuff' (identifying useful ideas, resources or connections); (5) Self-efficacy (believing 'we can do this!'); (6) Respect (feeling that SPIRIT understands and values one's work); (7) Confidence (believing in the study's integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory., Conclusion: This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants' perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies.

Background: Evidence-based decision-making is an important foundation for health policy and service planning decisions, yet there remain challenges in ensuring that the many forms of available evidence are considered when decisions are being made. Mobilising knowledge for policy and practice is an emergent process, and one that is highly relational, often messy and profoundly context dependent. Systems approaches, such as dynamic simulation modelling can be used to examine both complex health issues and the context in which they are embedded, and to develop decision support tools., Objective: This paper reports on the novel use of participatory simulation modelling as a knowledge mobilisation tool in Australian real-world policy settings. We describe how this approach combined systems science methodology and some of the core elements of knowledge mobilisation best practice. We describe the strategies adopted in three case studies to address both technical and socio-political issues, and compile the experiential lessons derived. Finally, we consider the implications of these knowledge mobilisation case studies and provide evidence for the feasibility of this approach in policy development settings., Conclusion: Participatory dynamic simulation modelling builds on contemporary knowledge mobilisation approaches for health stakeholders to collaborate and explore policy and health service scenarios for priority public health topics. The participatory methods place the decision-maker at the centre of the process and embed deliberative methods and co-production of knowledge. The simulation models function as health policy and programme dynamic decision support tools that integrate diverse forms of evidence, including research evidence, expert knowledge and localised contextual information. Further research is underway to determine the impact of these methods on health service decision-making.

Background: Housing is a key determinant of the poor health of Aboriginal Australians. Most Aboriginal people live in cities and large towns, yet research into housing conditions has largely focused on those living in remote areas. This paper measures the prevalence of housing problems amongst participants in a study of urban Aboriginal families in New South Wales, Australia, and examines the relationship between tenure type and exposure to housing problems., Methods: Cross-sectional survey data was provided by 600 caregivers of 1406 Aboriginal children aged 0-17 years participating in Phase One of the Study of Environment on Aboriginal Resilience and Child Health (SEARCH). Regression modelling of the associations between tenure type (own/mortgage, private rental or social housing) and housing problems was conducted, adjusting for sociodemographic factors., Results: The majority (60%) of SEARCH households lived in social housing, 21% rented privately and 19% either owned their home outright or were paying a mortgage ("owned"). Housing problems were common, particularly structural problems, damp and mildew, vermin, crowding and unaffordability. Physical dwelling problems were most prevalent for those living in social housing, who were more likely to report three or more physical dwelling problems than those in owned (PR 3.19, 95%CI 1.97, 5.73) or privately rented homes (PR 1.49, 1.11, 2.08). However, those in social housing were the least likely to report affordability problems. Those in private rental moved home most frequently; children in private rental were more than three times as likely to have lived in four or more homes since birth than those in owned homes (PR 3.19, 95%CI 1.97, 5.73). Those in social housing were almost half as likely as those in private rental to have lived in four or more homes since birth (PR 0.56, 95%CI 0.14, 0.77). Crowding did not vary significantly by tenure type., Conclusions: The high prevalence of housing problems amongst study participants suggests that urban Aboriginal housing requires further attention as part of efforts to reduce the social and health disadvantage experienced by Aboriginal Australians. Particular attention should be directed to the needs of those renting in the private and social housing sectors, who are experiencing the poorest dwelling conditions.

Background: Rapid reviews are increasingly being used to help policy makers access research in short time frames. A clear articulation of the review's purpose, questions, scope, methods and reporting format is thought to improve the quality and generalisability of review findings. The aim of the study is to explore the effectiveness of knowledge brokering in improving the perceived clarity of rapid review proposals from the perspective of potential reviewers. To conduct the study, we drew on the Evidence Check program, where policy makers draft a review proposal (a pre knowledge brokering proposal) and have a 1-hour session with a knowledge broker, who re-drafts the proposal based on the discussion (a post knowledge brokering proposal)., Methods: We asked 30 reviewers who had previously undertaken Evidence Check reviews to examine the quality of 60 pre and 60 post knowledge brokering proposals. Reviewers were blind to whether the review proposals they received were pre or post knowledge brokering. Using a six-point Likert scale, reviewers scored six questions examining clarity of information about the review's purpose, questions, scope, method and format and reviewers' confidence that they could meet policy makers' needs. Each reviewer was allocated two pre and two post knowledge brokering proposals, randomly ordered, from the 60 reviews, ensuring no reviewer received a pre and post knowledge brokering proposal from the same review., Results: The results showed that knowledge brokering significantly improved the scores for all six questions addressing the perceived clarity of the review proposal and confidence in meeting policy makers' needs; with average changes of 0.68 to 1.23 from pre to post across the six domains., Conclusions: This study found that knowledge brokering increased the perceived clarity of information provided in Evidence Check rapid review proposals and the confidence of reviewers that they could meet policy makers' needs. Further research is needed to identify how the knowledge brokering process achieves these improvements and to test the applicability of the findings in other rapid review programs.

Background: Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers' capacity to engage with and use research., Methods: We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales)., Results: Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal., Conclusions: Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Background: Poor housing is widely cited as an important determinant of the poor health status of Aboriginal Australians, as for indigenous peoples in other wealthy nations with histories of colonisation such as Canada, the United States of America and New Zealand. While the majority of Aboriginal Australians live in urban areas, most research into housing and its relationship with health has been conducted with those living in remote communities. This study explores the views of Aboriginal people living in Western Sydney about their housing circumstances and what relationships, if any, they perceive between housing and health., Methods: Four focus groups were conducted with clients and staff of an Aboriginal community-controlled health service in Western Sydney (n = 38). Inductive, thematic analysis was conducted using framework data management methods in NVivo10., Results: Five high-level themes were derived: the battle to access housing; secondary homelessness; overcrowding; poor dwelling conditions; and housing as a key determinant of health. Participants associated their challenging housing experiences with poor physical health and poor social and emotional wellbeing. Housing issues were said to affect people differently across the life course; participants expressed particular concern that poor housing was harming the health and developmental trajectories of many urban Aboriginal children., Conclusions: Housing was perceived as a pivotal determinant of health and wellbeing that either facilitates or hinders prospects for full and healthy lives. Many of the specific health concerns participants attributed to poor housing echo existing epidemiological research findings. These findings suggest that housing may be a key intervention point for improving the health of urban Aboriginal Australians.

Background: In this paper, we identify and respond to the fidelity assessment challenges posed by novel contextualised interventions (i.e. interventions that are informed by composite social and psychological theories and which incorporate standardised and flexible components in order to maximise effectiveness in complex settings). We (a) describe the difficulties of, and propose a method for, identifying the essential elements of a contextualised intervention; (b) provide a worked example of an approach for critiquing the validity of putative essential elements; and (c) demonstrate how essential elements can be refined during a trial without compromising the fidelity assessment. We used an exploratory test-and-refine process, drawing on empirical evidence from the process evaluation of Supporting Policy In health with Research: an Intervention Trial (SPIRIT). Mixed methods data was triangulated to identify, critique and revise how the intervention's essential elements should be articulated and scored., Results: Over 50 provisional elements were refined to a final list of 20 and the scoring rationalised. Six (often overlapping) challenges to the validity of the essential elements were identified. They were (1) redundant-the element was not essential; (2) poorly articulated-unclear, too specific or not specific enough; (3) infeasible-it was not possible to implement the essential element as intended; (4) ineffective-the element did not effectively deliver the change principles; (5) paradoxical-counteracting vital goals or change principles; or (6) absent or suboptimal-additional or more effective ways of operationalising the theory were identified. We also identified potentially valuable 'prohibited' elements that could be used to help reduce threats to validity., Conclusions: We devised a method for critiquing the construct validity of our intervention's essential elements and modifying how they were articulated and measured, while simultaneously using them as fidelity indicators. This process could be used or adapted for other contextualised interventions, taking evaluators closer to making theoretically and contextually sensitive decisions upon which to base fidelity assessments.

Background: Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations' research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations' capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership)., Method: We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe., Results: The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4)., Conclusion: We utilised qualitative and quantitative methods to develop a system to assess and score organisations' capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Greenhalgh and Fahy's paper about the 2014 Research Excellence Framework provides insights into the challenges of assessing research impact. Future research assessment exercises should consider how best to include measurement of indirect and non-linear impact and whether efforts in knowledge transfer and co-production should be explicitly recognised. Greenhalgh and Fahy's findings also demonstrate that the structure of the assessment exercise can privilege certain kinds of research and may therefore miss some research that has a high impact on policy and practice. There are a growing number of courses, tools, and funding models to assist researchers in making an impact, although as yet there is little evidence about whether these approaches work in practice. Please see related article: http://www.biomedcentral.com/1741-7015/13/232.

Background: To maximise the impact of public health research, research interventions found to be effective in improving health need to be scaled up and delivered on a population-wide basis. Theoretical frameworks and approaches are useful for describing and understanding how effective interventions are scaled up from small trials into broader policy and practice and can be used as a tool to facilitate effective scale-up. The purpose of this literature review was to synthesise evidence on scaling up public health interventions into population-wide policy and practice, with a focus on the defining and describing frameworks, processes and methods of scaling up public health initiatives., Methods: The review involved keyword searches of electronic databases including MEDLINE, CINAHL, PsycINFO, EBM Reviews and Google Scholar between August and December 2013. Keywords included 'scaling up' and 'scalability', while the search terms 'intervention research', 'translational research', 'research dissemination', 'health promotion' and 'public health' were used to focus the search on public health approaches. Studies included in the review were published in English from January 1990 to December 2013 and described processes, theories or frameworks associated with scaling up public health and health promotion interventions., Results: There is a growing body of literature describing frameworks for scaling health interventions, with the review identifying eight frameworks, the majority of which have an explicit focus on scaling up health action in low and middle income country contexts. Key success factors for scaling up included the importance of establishing monitoring and evaluation systems, costing and economic modelling of intervention approaches, active engagement of a range of implementers and the target community, tailoring the scaled-up approach to the local context, the use of participatory approaches, the systematic use of evidence, infrastructure to support implementation, strong leadership and champions, political will, well defined scale-up strategy and strong advocacy., Conclusions: Effective scaling up requires the systematic use of evidence, and it is essential that data from implementation monitoring is linked to decision making throughout the scaling up process. Conceptual frameworks can assist both policy makers and researchers to determine the type of research that is most useful at different stages of scaling up processes.

Background: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts., Methods: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use., Results: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation., Conclusions: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

Background: Effective use of research to inform policymaking can be strengthened by policymakers undertaking various research engagement actions (e.g., accessing, appraising, and applying research). Consequently, we developed a thorough measurement and scoring tool to assess whether and how policymakers undertook research engagement actions in the development of a policy document. This scoring tool breaks down each research engagement action into its key 'subactions' like a checklist. The primary aim was to develop the scoring tool further so that it assigned appropriate scores to each subaction based on its effectiveness for achieving evidence-informed policymaking. To establish the relative effectiveness of these subactions, we conducted a conjoint analysis, which was used to elicit the opinions and preferences of knowledge translation experts., Method: Fifty-four knowledge translation experts were recruited to undertake six choice surveys. Respondents were exposed to combinations of research engagement subactions called 'profiles', and rated on a 1-9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of each research engagement action. Generalised estimating equations were used to analyse respondents' choice data, where a utility coefficient was calculated for each subaction. A large utility coefficient indicates that a subaction was influential in guiding experts' ratings of extensive engagement with research., Results: The calculated utilities were used as the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of engaging with research: searching academic literature databases, obtaining systematic reviews and peer-reviewed research, appraising relevance by verifying its applicability to the policy context, appraising quality by evaluating the validity of the method and conclusions, engaging in thorough collaborations with researchers, and undertaking formal research projects to inform the policy in question., Conclusions: We have generated an empirically-derived and context-sensitive method of measuring and scoring the extent to which policymakers engaged with research to inform policy development. The scoring system can be used by organisations to quantify staff research engagement actions and thus provide them with insights into what types of training, systems, and tools might improve their staff's research use capacity.

Background: Research funding agencies continue to grapple with assessing research impact. Theoretical frameworks are useful tools for describing and understanding research impact. The purpose of this narrative literature review was to synthesize evidence that describes processes and conceptual models for assessing policy and practice impacts of public health research., Methods: The review involved keyword searches of electronic databases, including MEDLINE, CINAHL, PsycINFO, EBM Reviews, and Google Scholar in July/August 2013. Review search terms included 'research impact', 'policy and practice', 'intervention research', 'translational research', 'health promotion', and 'public health'. The review included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, and conceptual models for assessing research impact. The review was conducted in two phases: initially, abstracts were retrieved and assessed against the review criteria followed by the retrieval and assessment of full papers against review criteria., Results: Thirty one primary studies and one systematic review met the review criteria, with 88% of studies published since 2006. Studies comprised assessments of the impacts of a wide range of health-related research, including basic and biomedical research, clinical trials, health service research, as well as public health research. Six studies had an explicit focus on assessing impacts of health promotion or public health research and one had a specific focus on intervention research impact assessment. A total of 16 different impact assessment models were identified, with the 'payback model' the most frequently used conceptual framework. Typically, impacts were assessed across multiple dimensions using mixed methodologies, including publication and citation analysis, interviews with principal investigators, peer assessment, case studies, and document analysis. The vast majority of studies relied on principal investigator interviews and/or peer review to assess impacts, instead of interviewing policymakers and end-users of research., Conclusions: Research impact assessment is a new field of scientific endeavour and there are a growing number of conceptual frameworks applied to assess the impacts of research.

Background: There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003-2007) of intervention research funded by Australia's National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts., Methods: We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance., Results: Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews., Conclusions: While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

Background: Decisions to scale up population health interventions from small projects to wider state or national implementation is fundamental to maximising population-wide health improvements. The objectives of this study were to examine: i) how decisions to scale up interventions are currently made in practice; ii) the role that evidence plays in informing decisions to scale up interventions; and iii) the role policy makers, practitioners, and researchers play in this process., Methods: Interviews with an expert panel of senior Australian and international public health policy-makers (n = 7), practitioners (n = 7), and researchers (n = 7) were conducted in May 2013 with a participation rate of 84%., Results: Scaling up decisions were generally made through iterative processes and led by policy makers and/or practitioners, but ultimately approved by political leaders and/or senior executives of funding agencies. Research evidence formed a component of the overall set of information used in decision-making, but its contribution was limited by the paucity of relevant intervention effectiveness research, and data on costs and cost effectiveness. Policy makers, practitioners/service managers, and researchers had different, but complementary roles to play in the process of scaling up interventions., Conclusions: This analysis articulates the processes of how decisions to scale up interventions are made, the roles of evidence, and contribution of different professional groups. More intervention research that includes data on the effectiveness, reach, and costs of operating at scale and key service delivery issues (including acceptability and fit of interventions and delivery models) should be sought as this has the potential to substantially advance the relevance and ultimately usability of research evidence for scaling up population health action.

Background: Intervention research provides important information regarding feasible and effective interventions for health policy makers, but few empirical studies have explored the mechanisms by which these studies influence policy and practice. This study provides an exploratory case series analysis of the policy, practice and other related impacts of the 15 research projects funded through the New South Wales Health Promotion Demonstration Research Grants Scheme during the period 2000 to 2006, and explored the factors mediating impacts., Methods: Data collection included semi-structured interviews with the chief investigators (n = 17) and end-users (n = 29) of each of the 15 projects to explore if, how and under what circumstances the findings had been used, as well as bibliometric analysis and verification using documentary evidence. Data analysis involved thematic coding of interview data and triangulation with other data sources to produce case summaries of impacts for each project. Case summaries were then individually assessed against four impact criteria and discussed at a verification panel meeting where final group assessments of the impact of research projects were made and key influences of research impact identified., Results: Funded projects had variable impacts on policy and practice. Project findings were used for agenda setting (raising awareness of issues), identifying areas and target groups for interventions, informing new policies, and supporting and justifying existing policies and programs across sectors. Reported factors influencing the use of findings were: i) nature of the intervention; ii) leadership and champions; iii) research quality; iv) effective partnerships; v) dissemination strategies used; and, vi) contextual factors., Conclusions: The case series analysis provides new insights into how and under what circumstances intervention research is used to influence real world policy and practice. The findings highlight that intervention research projects can achieve the greatest policy and practice impacts if they address proximal needs of the policy context by engaging end-users from the inception of projects and utilizing existing policy networks and structures, and using a range of strategies to disseminate findings that go beyond traditional peer review publications.

Background: Clinical networks are increasingly being viewed as an important strategy for increasing evidence-based practice and improving models of care, but success is variable and characteristics of networks with high impact are uncertain. This study takes advantage of the variability in the functioning and outcomes of networks supported by the Australian New South Wales (NSW) Agency for Clinical Innovation's non-mandatory model of clinical networks to investigate the factors that contribute to the success of clinical networks., Methods/design: The objective of this retrospective study is to examine the association between external support, organisational and program factors, and indicators of success among 19 clinical networks over a three-year period (2006-2008). The outcomes (health impact, system impact, programs implemented, engagement, user perception, and financial leverage) and explanatory factors will be collected using a web-based survey, interviews, and record review. An independent expert panel will provide judgements about the impact or extent of each network's initiatives on health and system impacts. The ratings of the expert panel will be the outcome used in multivariable analyses. Following the rating of network success, a qualitative study will be conducted to provide a more in-depth examination of the most successful networks., Discussion: This is the first study to combine quantitative and qualitative methods to examine the factors that contribute to the success of clinical networks and, more generally, is the largest study of clinical networks undertaken. The adaptation of expert panel methods to rate the impacts of networks is the methodological innovation of this study. The proposed project will identify the conditions that should be established or encouraged by agencies developing clinical networks and will be of immediate use in forming strategies and programs to maximise the effectiveness of such networks.

Background: Better communication is often suggested as fundamental to increasing the use of research evidence in policy, but little is known about how researchers and policy makers work together or about barriers to exchange. This study explored the views and practice of policy makers and researchers regarding the use of evidence in policy, including: (i) current use of research to inform policy; (ii) dissemination of and access to research findings for policy; (iii) communication and exchange between researchers and policy makers; and (iv) incentives for increasing the use of research in policy., Methods: Separate but similar interview schedules were developed for policy makers and researchers. Senior policy makers from NSW Health and senior researchers from public health and health service research groups in NSW were invited to participate. Consenting participants were interviewed by an independent research company., Results: Thirty eight policy makers (79% response rate) and 41 researchers (82% response rate) completed interviews. Policy makers reported rarely using research to inform policy agendas or to evaluate the impact of policy; research was used more commonly to inform policy content. Most researchers reported that their research had informed local policy, mainly by increasing awareness of an issue. Policy makers reported difficulty in accessing useful research syntheses, and only a third of researchers reported developing targeted strategies to inform policy makers of their findings. Both policy makers and researchers wanted more exchange and saw this as important for increasing the use of research evidence in policy; however, both groups reported a high level of involvement by policy makers in research., Conclusion: Policy makers and researchers recognise the potential of research to contribute to policy and are making significant attempts to integrate research into the policy process. These findings suggest four strategies to assist in increasing the use of research in policy: making research findings more accessible to policy makers; increasing opportunities for interaction between policy makers and researchers; addressing structural barriers such as research receptivity in policy agencies and a lack of incentives for academics to link with policy; and increasing the relevance of research to policy.

Background: There is considerable potential for health research to contribute to improved health services, programs, and outcomes; the policies of health research funding agencies are critical to achieving health gains from research. The need for research to better address health disparities in Indigenous people has been widely recognised. This paper: (i) describes the policy changes made by the National Health and Medical Research Council (NHMRC) from 1997 to 2002 to improve funding of Aboriginal health research (ii) examines catalysts for the policy changes (iii) describes the extent to which policy changes were followed by new models of research and (iv) outlines issues for Indigenous health policy in the future., Methods: This study had two parts: (i) semi-structured interviews were conducted over a four -month period with seven individuals who played a leading role in the policy changes at NHMRC during the period 1997-2002, to describe policy changes and to examine the catalysts for the changes; (ii) a case study was undertaken to evaluate projects by recipients of NHMRC People Support awards and NHMRC Capacity Building Grants in Population Health Research to examine the types of research being undertaken five years after the policy changes were implemented. The proposals of these researchers were assessed in terms of whether they reported intending to: evaluate interventions; engage Indigenous community members and organisations; and build research capacity among Indigenous people., Results: Seven policy changes over a period of five years were identified, including those to: establish an ethical approach to working with Indigenous people; increase the influence of Indigenous people within NHMRC; encourage priority research directed at improving Indigenous health; and recognise Aboriginal and Torres Strait Islander health research as a priority area including a commitment to an expenditure target of 5% of annual funds. Seven catalysts for this change were identified. These included: a perceived lack of effective response to the health needs of Indigenous people; a changed perception of the role of NHMRC in encouraging research to maximise health gains; and leadership within the organisation.The case study analysis demonstrated that 45% of all People Support recipients intend to engage Indigenous community members and organisations in consultation, 26% included an evaluation of an intervention and two (6.5%) were granted to an individual from an Indigenous background. Six of seven Population Health Capacity Building Grants that were awarded to study Indigenous health between 2004 and 2006 included an intervention component; these grants supported 34 researchers from Indigenous backgrounds., Conclusion: NHMRC made significant policy changes from 1997 to 2002 to better support Indigenous health as a result of external pressure and internal commitment.The policy changes have made some progress in supporting better research models particularly in improving engagement with Indigenous communities. However, there remains a need for further reform to optimise research outcomes for Indigenous people from research.

Background: Accreditation has become ubiquitous across the international health care landscape. Award of full accreditation status in health care is viewed, as it is in other sectors, as a valid indicator of high quality organisational performance. However, few studies have empirically demonstrated this assertion. The value of accreditation, therefore, remains uncertain, and this persists as a central legitimacy problem for accreditation providers, policymakers and researchers. The question arises as to how best to research the validity, impact and value of accreditation processes in health care. Most health care organisations participate in some sort of accreditation process and thus it is not possible to study its merits using a randomised controlled strategy. Further, tools and processes for accreditation and organisational performance are multifaceted., Methods/design: To understand the relationship between them a multi-method research approach is required which incorporates both quantitative and qualitative data. The generic nature of accreditation standard development and inspection within different sectors enhances the extent to which the findings of in-depth study of accreditation process in one industry can be generalised to other industries. This paper presents a research design which comprises a prospective, multi-method, multi-level, multi-disciplinary approach to assess the validity, impact and value of accreditation., Discussion: The accreditation program which assesses over 1,000 health services in Australia is used as an exemplar for testing this design. The paper proposes this design as a framework suitable for application to future international research into accreditation. Our aim is to stimulate debate on the role of accreditation and how to research it.