Search

Pawson and Tilley’s acknowledgment of programs embedded in multiple social systems has gained little traction in realist synthesis and evaluation practice. A practice focusing on fairly closed systems—explaining how programs work and do not work—has emerged. This article negotiates the boundaries of knowledge pertinent to have in program design and evaluation from a realist perspective. It highlights critical realism as another possible response to systems thinking in evaluation. Moving one level up a program, it theorizes about social structures, mechanisms, and causes operating in a complex system within which an education-to-work program is nested. Three implications of the approach are highlighted: it foregrounds the relational nature of social, psychological, and programmatic structures and mechanisms; enables policymakers to develop a broader understanding of structures needed to support a program; and enables program architects to ascertain how a planned program might assimilate and adapt to social structures and mechanisms already established in a context.

Realist research and evaluation methods are increasingly being used in health and social care research settings, including integrated care endeavours. The realist view of how the world is, and how we can explain it, seeks to look 'beneath the hood' and to ask the questions how and why. Health and social policy and service settings, within which integrated care strives to make a difference, are complex 'open' systems that can only partially be understood using empirical methods alone. Abductive and retroductive modes of reasoning, as used by realist researchers, are commonly used by physicians, detectives and lawyers to postulate hidden explanations based on available experienced evidence. The approach is, therefore, not foreign to many readers of this chapter, and realist scientific methodology gives validity to this common but scientifically neglected mode of analysis.

Introduction: Healthy Homes and Neighbourhoods (HHAN), an integrated care programme in the Sydney Local Health District (SLHD), seeks to address the needs of disadvantaged families through care coordination, as one of its components. This research aims to determine for whom, when and why the care coordination component of HHAN works, and establish the reported outcomes for clients, service-providers and partner organisations.Methods: Critical realist methodology was utilised to undertake a qualitative evaluation of the impact of care coordination. Purposive sampling was used to select a total of 37 participants for interview, including consumers, service-providers and key stakeholders. Thematic analysis was undertaken to derive the major modes of intervention of HHAN, and data representing these elements was coded and summarised under contexts, mechanisms and outcomes.Results: Analysis indicates that care coordination has a positive impact on clients’ sense of independence, self-awareness and outlook on life. Trust and favourable interpersonal relations were identified as major underlying mechanisms for a successful client-provider working relationship. The identified modes of intervention facilitating positive consumer outcomes included accessibility, flexibility and service navigation. Persistent siloes in health and systemic resistance to collaboration was seen to hinder effective care delivery.Conclusions: This study suggests that a care coordination model may be effective in engaging disadvantaged families in healthcare, assist them in navigating the health system and can lead to beneficial health and social outcomes. Successful implementation of care coordination requires flexible programme design and experienced and skilful clinicians to fulfil the care coordinator role. There is a need to appreciate the negative impact that the complex and siloed health system can have on disadvantaged families.

Introduction: The Healthy Homes and Neighbourhoods (HHAN) Integrated Care Program seeks to enhance vulnerable family engagement with health and social services through a care coordination model. Besides servicing families in Sydney, HHAN has also established place-based initiatives (PBIs) in areas of disadvantage such as Redfern. The Redfern PBI co-locates HHAN with housing, drug and alcohol services, and financial and legal services. This integration aims to facilitate service access and multi-agency support for vulnerable families in Redfern. Hence, this study aims to evaluate for whom, when and why HHAN’s PBI at Redfern works, or not, and what are its outcomes. Methods: The project utilises critical realist methodology to undertake a qualitative evaluation of the impact of the PBI on clients, services and the community. Purposive sampling was used to identify 21 participants including HHAN clients, HHAN staff and stakeholders (HHAN partners). In-depth, semi-structured interviews were audio-recorded, transcribed, coded and analysed using a context (C), intervention (I), mechanism (M) outcome (O) (CIMO) approach to abductive analysis. Results: Five key CIMO configurations of the Redfern PBI emerged – whole-of-family involvement, flexibility, trust, building connections and co-location. Whilst each theory had specific outcomes, overall client outcomes included improved access to services, better outlook, empowerment and engagement with services. Service outcomes included increased collaboration and foundation for integration between HHAN and other services. Negative outcomes included lack of full integration, the risk associated with integration and difficulty evaluating impact. Conclusion: This study successfully refined the program theory for subsequent use in later implementation of critical realist evaluation studies.

Modern science communication has emerged in the twentieth century as a field of study, a body of practice and a profession—and it is a practice with deep historical roots. We have seen the birth of interactive science centres, the first university actions in teaching and conducting research, and a sharp growth in employment of science communicators. This collection charts the emergence of modern science communication across the world. This is the first volume to map investment around the globe in science centres, university courses and research, publications and conferences as well as tell the national stories of science communication. How did it all begin? How has development varied from one country to another? What motivated governments, institutions and people to see science communication as an answer to questions of the social place of science? Communicating Science describes the pathways followed by 39 different countries. All continents and many cultures are represented. For some countries, this is the first time that their science communication story has been told.

Background Healthy Homes and Neighbourhoods (HHAN) Integrated Care Initiative was established to improve the care of families with complex health and social needs who reside in Sydney Local Health District. HHAN seeks to provide long-term multi-disciplinary care coordination as well as enhance capacity building and promote integrated care. The critical realist study reported here is part of the longitudinal development and evaluation of complex integrated health and social care interventions in Sydney, Australia. Methods We describe the qualitative component of a critical realist pilot case study aimed at exploring, explaining and refining emerging HHAN programme theories in relation to care coordination. Qualitative interviews were undertaken with HHAN clients (n = 12), staff and other stakeholders (n = 21). Interviews and coding used a context (C), mechanism (M) and outcome (O) framework. Inductive, deductive, retroductive and abductive modes of reasoning were used with the CMO heuristic tool to inform the developing programme theory. Results The mechanisms underpinning effective engagement of clients by care coordinators included: building trust, leveraging other family, social and organisational relationships, meeting clients on their own terms, demonstrating staff effectiveness as quickly as possible, and client empowerment. Mechanisms for enhancing care integration included knowledge transfer activities and shared learning among collaborators, structural and cultural changes, enhancing mutual respect, co-location of multidisciplinary and/or interagency staff and cultivating faith in positive change among staff. Conclusions Use of a critical realism case study approach served to elucidate the varied influences of contexts and mechanisms on programme outcomes, to highlight what works for whom and in what context. Findings supported the initial programme theory that engagement and trust building with clients, alongside enhanced collaboration and integration of services, improved outcomes for vulnerable families with complex needs. Further research is needed to explore the cost-effectiveness of integrated care initiatives, in view of the long term nature of service provision and the risk of staff burnout.

Low employment rates of individuals with intellectual disabilities in Australia stem from apprehensions about their abilities to engage in work, and employers’ low confidence in hiring them in paid positions. Drawing on principles of evaluative reasoning, this paper reports on findings from a mixed-methods formative evaluation. It was conducted to understand the need for a two-year Education-to-Work training program, the quality of its content, and the views that 8 parents, 5 facilitators, and 4 students had about changes attributable to the program after its first year. Program performance was assessed against the quality standards of excellent, good, satisfactory, marginal, and needs improvement. Findings from the evaluation highlighted the potential usefulness of post-secondary education (PSE) transitioning programs to secure longer term employment, for adults with Down syndrome. Specific recommendations are made to improve the quality of the program, and abstractions gleaned from the study are highlighted.

This article re-examines the contents of Singapore’s Thinking Schools Learning Nation (TSLN) and Teach Less Learn More (TLLM) educational initiatives, introduced and implemented to promote change and to prepare Singaporeans for a twenty-first century knowledge-based economy. Adopting a critical realist perspective that enables investigations into complex social systems, the paper highlights the concepts, change process and possible outcomes of change proposed by realist social theory. An explanatory critique responding to the question, ‘What social structural changes were implemented by the TSLN and TLLM initiatives, and why?’ is developed, tracing the programmes of change in TSLN and TLLM. Findings reported in 2013, by a local large-scale research project, has made claims about the ineffectiveness of the initiatives in bringing about desired changes in classroom instructional practices. The critique questions—given Singapore’s recent and consistent successful performances in international benchmarking tests—whether it is only in the classroom that educational change that matters, counts. It suggests that despite making strong statements about the limited effectiveness of the TSLN and TLLM initiatives, many programs introduced and adopted by primary, secondary and post-secondary institutions, especially under TLLM, were left unexamined by the research project. The explanatory critique theorises that two kinds of changes have taken place—the reorientation of pedagogical practices in post-secondary institutions and extensions of what already exists in the primary and secondary sections. The paper concludes by highlighting some implications the explanatory critique have for research into educational change in general, and for educational change in Singapore.

Introduction: In March 2014, the New South Wales (NSW) Government (Australia) announced the NSW Integrated Care Strategy. The NSW Government’s Integrated Care Strategy funding enabled the establishment of an integrated care initiative called Healthy Homes and Neighbourhoods (HHAN). The Initiative was designed as a population-based, family-centered, care-coordination network that functioned across agencies to assist vulnerable families to navigate the health and social-care system, to keep themselves and their children safe, and in doing so, promote social cohesiveness. The evaluation framework is described. Methods: Since integrated care initiatives are an example of complex interventions in health and social care, we underpinned this programme of development, research and evaluation by the critical realist metatheory of science and the NSW Health Monitoring and Evaluation Framework, using the UK Medical Research Council (MRC) Framework for evaluating complex interventions.1 We adapted the iterative approach of development, feasibility/piloting, evaluation and implementation of the MRC framework to include critical realist, participatory, continuous and theory driven elements. Discussion: Both the 2008 UK Medical Research Council framework for evaluating complex interventions, and our previously reported critical realist methodology, provided the tools to enable an iterative approach to development, design, testing and continuous evaluation. The philosophical paradigms underpinning each approach are different. Both the NSW Health and MRC frameworks were developed from in the “logical positivist” inductive and deductive reasoning traditions with a strong focus on activities and outcomes but little emphasis on process. That weakness was been recognised and the MRC subsequently endorsed advice on process evaluation for complex interventions which accommodated the realist approach incorporated here. Conclusion: Integrated care initiatives are examples of complex interventions in health and social care. The interventions are multi-layered and operationalization is strongly influenced by both historical and current context. This aspect alone makes it imperative that the methodology used is post-positive and takes ontologically stratified context into account. The NSW Health Monitoring and Evaluation Framework did not make provision for assessment of context or mechanism of effect. We describe here a multilevel approach including a continuous improvement approach through constant comparison and triangulation of mixed method findings. Utilising the MRC framework for evaluating complex interventions within a critical realist methodology, enabled us to study both mechanisms of effect and context. Lessons learned: Evaluation of integrated care interventions will require a pluralist approach to methodology. It is possible to design evaluation approaches that allow study of context, mechanism, reach and efficacy in a complex layered design as used for integrated care initiatives. Limitations: The approach utilised was contrained by the requirements of the policy and funding organisation. We did not fully explore the application of Theory of Change and complexity science within the MRC framework. Suggestions for future research: We recommend that further theoretical and applied research be undertaken to advance the methods and tools available for evaluation of complex multi-layered integrated care initiatives. Reference: 1. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655

Introduction : In July 2015 Sydney Local Health District (SLHD) implemented an integrated care initiative for vulnerable families in the inner West region of Sydney, Australia. That initiative was designed as a cross-agency care coordination network that would ensure that vulnerable families: had their complex health and social needs met; kept themselves and their children safe; and were connected to society. We will describe the development of the integrated care design that drew on earlier realist causal and program theoretical work. Theory/Methods : Realist causal and program theory were used to inform the collaborative design of initiatives for vulnerable families. The collaborative design process included: identification of desirable and undesirable outcomes and contextual factors, consultation forums, interagency planning, and development of a service proposal. Results : The program theory mechanisms identified included: family-provider trust, willingness to share power, co-operation, Information, and building self-help skills. The Design Elements included: Wrap around services, place-based initiatives, care coordination, sustained nurse home visiting, primary care support, family group conferencing, targeted parenting, social media, outcome monitoring, workforce development and realist program evaluation. Discussion : The Design Elements included: identification of vulnerable families; care coordination; evidence-informed intervention(s); general Practice engagement and support; family health improvement; placed-based neighbourhood initiatives; interagency system change and planning; monitoring of individual and family outcomes; and evaluation. Conclusions : The design study described advances our earlier empirical and programme design studies toward the implementation of a full whole-of-government integrated health and social care initiative. That initiative was designed as a cross-agency care coordination network that would ensure that vulnerable families: had their complex health and social needs met; kept themselves and their children safe; and were connected to society. In so doing we aim to break intergenerational cycles of poverty, violence and crime, poor education and employment opportunities, psychopathology, and poor lifestyle and health behaviours, through strengthening family resiliency, improving access to services, and addressing the social determinants of health and well-being. Lessons learned : In undertaking this study we identified the importance of our earlier hierarchal program analysis for identifying the elements for the full design. The application of theory added rigour to the design of integrated care initiatives. The design benefited from the inclusion of elements of population health, primary care, consumer engagement, community engagement and social determinants of health. Limitations : The analysis and design process did not include a full critical realist analysis of pre-existing context. Consequently existing structural, cultural, relationship and agency barriers and enablers were not fully analysed as part of the design process. Suggestions for future research : Further methodological research to develop: 1) tools that can be rapidly applied during the design process to identify pre-existing contextual barriers and enablers; and 2) approaches to build the knowledge of context into the final design.

Introduction: In March 2014, the New South Wales (NSW) Government (Australia) announced the NSW Integrated Care Strategy. In response, a family-centred, population-based, integrated care initiative for vulnerable families and their children in Sydney, Australia was developed. The initiative was called Healthy Homes and Neighbourhoods. A realist translational social epidemiology programme of research and collaborative design is at the foundation of its evaluation. Theory and Method: The UK Medical Research Council (MRC) Framework for evaluating complex health interventions was adapted. This has four components, namely 1) development, 2) feasibility/piloting, 3) evaluation and 4) implementation. We adapted the Framework to include: critical realist, theory driven, and continuous improvement approaches. The modified Framework underpins this research and evaluation protocol for Healthy Homes and Neighbourhoods. Discussion: The NSW Health Monitoring and Evaluation Framework did not make provisions for assessment of the programme layers of context, or the effect of programme mechanism at each level. We therefore developed a multilevel approach that uses mixed-method research to examine not only outcomes, but also what is working for whom and why. Published version

Background: We will describe here a translational social epidemiology protocol for confirming a critical realist “Theory of Neighbourhood Context, Stress, Depression, and the Developmental Origins of Health and Disease (DOHaD)”. The approach will include the concretising and contextualising of the above causal theory into programme theories for child and adolescent interventions that aim to break intergenerational cycles of disadvantage and poor life outcomes. In undertaking this work we seek to advance realist translational methodology within the discipline of applied perinatal and paediatric social epidemiology. Theory and Methods: The research settings are in metropolitan Sydney. The design will be a longitudinal, multi-level, mixed method realist evaluation of applied programme interventions that seek to break the intergeneration cycle of social disadvantage and poor child health and developmental outcomes. The programme of research will consist of three components: 1) Operationalisation of the theory and designing of programme initiatives for implementation; 2) Evaluation of the translated programme and implementation theory using Theory of Change and critical realist evaluation; and 3) Theory Testing of realist hypotheses using both intensive and extensive critical realist research methods including realist structural modelling. Discussion: The proposed programme of research will assist in translating empirical explanatory theory building to theory driven interventions. The research will be situated in socially disadvantaged regions of Sydney where the local child and family inter-agencies will collaborate to design and implement new initiatives that address significant disparities in childhood development and adolescent outcomes attributed to neighbourhood circumstances, family stress and intergenerational cycles of disadvantage and poor mental health. Published version

Background: Realist philosophy and methodology is increasingly being explicitly used in the field of integrated care to research the extant context, structure and mechanisms at play, and to design and evaluation interventions. Philosophic realism is the view that entities exist independently of our perception or our theories about them. While the realist philosophy underpins much of modern health and social science, it is only recently that this philosophical approach has been popularised within main stream health and social science literature. Realist research methodology is increasingly being described in research areas that are relevant to the study of integrated care, namely, organisational management, information science, social epidemiology, economics, and health services evaluation. Importantly realist approaches are increasingly being used in mixed method research designs and to elucidate the processes at play in experimental and quasi-experimental studies. Realist researchers seek to explain the underlying “cause” or mechanisms that generate observed phenomenon. The realist understanding of how the world is ontology includes the notion of a hidden or “real” domain where mechanisms generate forces that result in the phenomena which we observe. Realists also view the world as consisting of strata or layers of reality which may interact with other layers to produce new mechanism. This approach is proving useful for studying and developing theory about complex health and social care systems, and then designing and evaluating possible interventions. Over the last number of years a number of realist studies have been presented at the International Foundation of Integrated Care conferences including: realist studies of context and mechanisms; realist design of integrated care interventions; realist synthesis of literature; and realist evaluations of integrated-care interventions. The methodologies, and the dialectical debates, are complex, and therefore, deserving of special consideration within the Integrated Care community. This preliminary workshop will support evaluators and researchers to use realist methods in integrated care and other complex evaluations. Because realist evaluation was initially developed using smaller-scale programs, methods need to be modified for large scale programs, while remaining consistent with underlying methodological principles. The workshop will discuss the value of and the dilemmas involved in using realist methods for large and complex programs, and demonstrate strategies to address the dilemmas. Aims and Objectives: 1- Present and discuss the basic tenants of realist research, design and evaluation as it applies to the large scale and complex programs such as integrated care 2- Discuss the scope and purpose of a Realist Research, Design and Evaluation for Integrated Care Special Interest Group RIC-SIG 3- Modify the scope and purpose of a Realist Research, Design and Evaluation for Integrated Care - SIG Format: Group Discussion Target Audience: Integrated care practitioners and researchers who are familiar with realist research including the current focus on critical realism and realist evaluation Learning and Takeaway: The workshop will define the scope and purpose of a SIG for Realist Research, Design and Evaluation for Integrated Care

This article focuses on the design of a critical realist review that deployed Bhaskar’s resolution, redescribing, retroduction, eliminating, identifying, and correcting schema and Pawson and Tilley’s Context–Mechanism–Outcome configuration underpinned by realist social theory. Methodologically, the review examined the relationship between Singapore’s educational restructuring efforts and the research program theory adopted to measure and provide feedback on its outcomes. Specifically, the review investigated how the standards for authentic achievement and pedagogy from America (and a modified version from Australia) influenced educational research conduct in Singapore. The mechanism of interest was the practice of using the standards and accompanying assumptions from the American experience as a generalizable mechanism for producing evidence to measure and inform Singapore’s educational restructuring policy and educational practice. The article proposes that the critical realist review design provides one example of analyzing complexity and accounting for the historical, highlighting the limits of generalizing program theories across borders.

This article explains how the meta-theoretical framework of critical realism addresses methodological issues of concern to social psychologists and social scientists. The article outlines key tenets of critical realism – its notion of the stratified nature of reality and generative mechanisms as powers in natural and social objects that cause things; its notion of knowledge of reality as stratified rather than only empirical; its acceptance of epistemological but not judgmental relativism; and its monist ontology. The article then introduces realist social theory to provide a framework for understanding the society–person connection. It explains how issues relating to culture, context, and society raised in the indigenous psychology English literature might be addressed from a critical realist perspective. Some implications arising from adopting a critical realist perspective in research practice are outlined and social psychologists and social scientists are encouraged to explore the potential of critical realism as a meta-theoretical framework and new paradigm.

This article aims to contribute to the methodological discussion initiated by Pawson and Manzano-Santaella (2012). It proposes a possible elaboration of the ‘trio of explanatory components’ (Pawson and Tilley, 1997: 77) used as a strategy in realist evaluation – the Context-Mechanism-Outcome configuration (CMOc). This temporarily diverts attention away from social programs to focus on the pre-existing context of action and related mechanisms that may be operating within it. It draws on the conceptual and theoretical guidance offered in Roy Bhaskar’s Transformational Model of Social Action (TMSA) and elaborated later by Margaret Archer in Realist Social Theory: The Morphogenetic Approach. The article concludes by articulating how the CMOc elaboration complements the work on realist evaluation.

Background: Realist research methodology is increasingly being used for the design and evaluation of integrated care initiatives. Much of current realist research and evaluation is qualitative in nature and yet many realist methodologists, including Pawson, Tilley, Sayer and Olsen, have argued for the use of mixed methods. There remains much debate regarding the role of quantitative methods, including statistics, within a critical realist philosophical framework. For many realist researchers the use of quantitative methods is both a philosophical and methodological stumbling block. Yet quantitative methods can contribute to realist theory building,, design and theory testing through their ability to use observable events to infer the existence of underlying mechanisms and to detect changes in both context and outcomes. This workshop will examine debates about the use of quantitative methods in realist research and evaluation, either as part of a mixed-method design or on their own. Examples of their use for both theory development and theory testing will be provided. The workshop will provide practical demonstration of the use of quantitative methods for both small and large scale studies. Knowledge, techniques and practices to be addressed include: The philosophical debates concerning the use of quantitative methods for realist research and evaluation Quantitative methods used for explanatory theory building research Construction of quantitative tools from qualitative data Quantitative methods used for evaluation and theory testing Application of quantitative methods within realist mixed method study designs Practical application of quantitative analysis to infer mechanistic action. By the conclusion of the workshop, participants should be able to: Understand the philosophical rejection of quantitative methods for both natural and social scientific realms Determine how to interpret quantitative methods and incorporate them into realist research and evaluation while maintaining a distinction between method and methodology Describe quantitative methods for inclusion in realist mixed method research and/or evaluation study designs Undertake practical realist analysis of quantitative study data using simple statistical methods. Aims and Objectives: 1) To examine debates about the use of quantitative methods in realist research and evaluation, either as part of a mixed-method design or on their own. 2) To demonstrate the use of realist quantitative methods for both integrated care design and evaluation. Format: 90 minutes, one speaker and facilitator, Group work at tables using problem-based learning. Discussion, debate and learning resources. Target Audience: Integrated care design and evaluation practitioners and researchers. Qualitative and quantitative researchers. Mixed methodologists. Learning and Takeaway: Increased understanding of how quantitative methods can be appropriately used with realist methodology. Introduction to the use of advanced quantitative methods for design and evaluation of integrated care initiatives including latent class analysis, latent variable pathways, and discontiuity quasiexperimental designs.

Introduction: This research protocol has been developed as part of an integrated care initiative for vulnerable families in Sydney, Australia. The aim is to develop “whole-of-government” interventions that break intergenerational cycles of disadvantage and poor life outcomes. A realist methodology will be described for the translation of empirically ‘data derived’ causal middle-range theories of social mechanisms, into health and social programme theories, whose theoretical propositions can be operationalized and studied in concrete situations using collective theory driven approaches. Theory/Methods: The research settings are in metropolitan Sydney, Australia. The design will be a longitudinal, multi-level, mixed method realist evaluation of applied programme interventions that seek to break the intergenerational cycle of social disadvantage and poor child health and developmental outcomes. The programme of research will consist of three components: 1) Operationalization of the theory and designing of programme initiatives for implementation; 2) Evaluation of the translated programme and implementation theory using Theory of Change and Realistic Evaluation; and 3) Theory Testing of realist hypotheses using both intensive and extensive critical realist research methods including realist structural modelling. Results: No applicable Discussion: The proposed programme of research will assist in translating empirical explanatory theory building to theory driven interventions. The research will be situated in socially disadvantaged regions of Sydney where the local child and family interagencies will collaborate to design and implement new initiatives that address significant disparities in early childhood development and adolescent outcomes attributed to neighbourhood circumstances, family stress and intergenerational cycles of disadvantage and poor mental health. Conclusions: Central to this protocol is the development of programme theory. Much of the theory driven and realist evaluation literature begins with existing interventions. The first task in those situations is to identify the underlying programme theory. In preparing this programme of work we were faced with the translation of causal theory to programme and implementation theory. We have proposed as a first step in this protocol the formal translation of the middle-range ‘causal’ theory into a middle-range ‘programme theory’ followed by an intervention design process based on Theory of Change approaches. Lessons learned: In preparing the protocol we have identified extensive debate concern the nature of implementation theory and programme theory. There is also on-going critical analysis of the nature of context (c), mechanisms (M) and outcomes (o) and how these might be defined and measured. Limitations: The application of realist methodologies to population health interventions is relatively new and there is little extant literature on the translational process from causal theory to programme theory. The study protocol is thus breaking new ground and it is anticipated that the approaches described will require modification. Suggestions for future research: The protocol development identified a need for further clarification of the nature of causal mechanism and programme mechanisms among the integrated care and evaluation research communities.