Search

Introduction: Place-based public health evaluations are increasingly making use of natural experiments. This scoping review aimed to provide an overview of the design and use of natural experiment evaluations (NEEs), and an assessment of the plausibility of the as-if randomization assumption.Methods: A systematic search of three bibliographic databases (Pubmed, Web of Science and Ovid-Medline) was conducted in January 2020 to capture publications that reported a natural experiment of a place-based public health intervention or outcome. For each, study design elements were extracted. An additional evaluation of as-if randomization was conducted by 12 of this paper's authors who evaluated the same set of 20 randomly selected studies and assessed ‘as-if ' randomization for each.Results: 366 NEE studies of place-based public health interventions were identified. The most commonly used NEE approach was a Difference-in-Differences study design (25%), followed by before-after studies (23%) and regression analysis studies. 42% of NEEs had likely or probable as-if randomization of exposure (the intervention), while for 25% this was implausible. An inter-rater agreement exercise indicated poor reliability of as-if randomization assignment. Only about half of NEEs reported some form of sensitivity or falsification analysis to support inferences.Conclusion: NEEs are conducted using many different designs and statistical methods and encompass various definitions of a natural experiment, while it is questionable whether all evaluations reported as natural experiments should be considered as such. The likelihood of as-if randomization should be specifically reported, and primary analyses should be supported by sensitivity analyses and/or falsification tests. Transparent reporting of NEE designs and evaluation methods will contribute to the optimum use of place-based NEEs.

Supplementary Material 1

In recent decades, the use of conditionality backed by benefit sanctions for those claiming unemployment and related benefits has become widespread in the social security systems of high-income countries. Critics argue that sanctions may be ineffective in bringing people back to employment or indeed harmful in a range of ways. Existing reviews largely assess the labour market impacts of sanctions but our understanding of the wider impacts is more limited. We report results from a scoping review of the international quantitative research evidence on both labour market and wider impacts of benefit sanctions. Following systematic search and screening, we extract data for 94 studies reporting on 253 outcome measures. We provide a narrative summary, paying attention to the ability of the studies to support causal inference. Despite variation in the evidence base and study designs, we found that labour market studies, covering two thirds of our sample, consistently reported positive impacts for employment but negative impacts for job quality and stability in the longer term, along with increased transitions to non-employment or economic inactivity. Although largely relying on non-experimental designs, wider-outcome studies reported significant associations with increased material hardship and health problems. There was also some evidence that sanctions were associated with increased child maltreatment and poorer child well-being. Lastly, the review highlights the generally poor quality of the evidence base in this area, with few studies employing research methods designed to identify the causal impact of sanctions, especially in relation to wider impacts.

PRISMA checklist. (DOC 63 kb)

Bibliographic databases searched, search terms and example search. (DOCX 16 kb)

Background Lone parents and their children experience higher than average levels of adverse health and social outcomes, much of which are explained by high rates of poverty. Many high income countries have attempted to address high poverty rates by introducing employment requirements for lone parents in receipt of welfare benefits. However, there is evidence that employment may not reduce poverty or improve the health of lone parents and their children. Methods We conducted a systematic review of qualitative studies reporting lone parents’ accounts of participation in welfare to work (WtW), to identify explanations and possible mechanisms for the impacts of WtW on health and wellbeing. Twenty one bibliographic databases were searched. Two reviewers independently screened references and assessed study quality. Studies from any high income country that met the criteria of focussing on lone parents, mandatory WtW interventions, and health or wellbeing were included. Thematic synthesis was used to investigate analytic themes between studies. Results Screening of the 4703 identified papers and quality assessment resulted in the inclusion of 16 qualitative studies of WtW in five high income countries, USA, Canada, UK, Australia, and New Zealand, covering a variety of welfare regimes. Our synthesis found that WtW requirements often conflicted with child care responsibilities. Available employment was often poorly paid and precarious. Adverse health impacts, such as increased stress, fatigue, and depression were commonly reported, though employment and appropriate training was linked to increased self-worth for some. WtW appeared to influence health through the pathways of conflict and control, analytical themes which emerged during synthesis. WtW reduced control over the nature of employment and care of children. Access to social support allowed some lone parents to manage the conflict associated with employment, and to increase control over their circumstances, with potentially beneficial health impacts. Conclusion WtW can result in increased conflict and reduced control, which may lead to negative impacts on mental health. Availability of social support may mediate the negative health impacts of WtW. Electronic supplementary material The online version of this article (doi:10.1186/s12889-016-2880-9) contains supplementary material, which is available to authorized users.