Search

Automated content analysis methods treat ‘‘text as data'' and can therefore analyze efficiently large qualitative databases. Yet, despite their potential, these methods are rarely used to supplement qualitative analysis in small-N designs. We address this gap by replicating the qualitative findings of a case study of a social policy reform using automated content analysis. To characterize the ideology of this reform, we reanalyze the same interview data with Wordscores, using academic publications as reference texts. As expected, the reform's ideology is center/center-right, a result that we validate using content, convergent and discriminant strategies. The validation evidence suggests not only that the ideological positioning of the policy reform is credible, but also that Wordscores' scope of application is greater than expected

The concept of explanation has attracted considerable attention in the social sciences and, in particular, in political science. However, scholars are not always familiar with what explaining political phenomena means, let alone with what it entails for developing sound causal arguments. This essay introduces Craig Parsons’ fourfold typology of explanation (institutional, ideational, structural, and psychological) before assessing its value for the causal analysis of political behaviour and processes. As argued, despite its limitations, his typology clearly maps the explanations in political science while helping scholars to combine them more rigorously, when needed. This is why Parsons typology has the potential to move political scientists to the ‘next level’, as far as explanation is concerned.

Background. Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. Objectives. This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines 1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; 2) the convergence between the scores of coders and those of key respondents (i.e., authors); and 3) the convergence between the authors’ scores on the PEMI and the Evaluation Involvement Scale (EIS). Sample. A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Measures. Stakeholder participation was measured on nominal and ordinal scales. Cohen’s kappa, the intraclass correlation coefficient and Spearman’s rho were used to assess reliability and convergence. Results. Reliability results ranged from fair to excellent. Convergence between coders’ and authors’ scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Conclusions. Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.

Plusieurs provinces canadiennes ont mis en œuvre des réformes substantielles de leur régime d’aide sociale dans les années 1990. Or, nos connaissances à propos des idées qui sous-tendent ces réformes sont étonnamment limitées. En particulier, peu d’études empiriques ont directement examiné les idées des acteurs de politique. Cet article présente une étude de cas rétrospective d’une initiative saskatchewanaise majeure, soit la réforme Building Independence (BI), et examine sa congruence avec trois paradigmes d’aide sociale. Les données proviennent d’entrevues avec des acteurs de politique. Cette étude conclut que les idées sous-jacentes à BI sont étroitement alignées avec le paradigme de l’activation, mais présente certaines similitudes avec les paradigmes du droit social (entitlement) et du workfare. Ce résultat est ensuite discuté à la lumière de trois enjeux : 1) les autochtones, 2) la perspective des prestataires d’aide sociale, et 3) l’idéologie de la Troisième voie. Des pistes de recherche sur les idées et le changement de politique viennent conclure cet article., Many provinces have enacted substantial reforms of their social assistance regime in the 1990s. However, we know surprisingly little about the ideas that underlie welfare reform in Canada. In particular, few empirical studies have directly examined the ideas of policy actors. This article presents a retrospective case study of a major policy initiative, namely Saskatchewan’s Building Independence (BI), and examines its alignment with three paradigms of social assistance. Data come primarily from interviews with policy actors. This study concludes that the policy ideas informing BI align closely with the activation paradigm, but also share some similarities with the entitlement and workfare paradigms. The significance of this finding is then discussed in light of three issues: 1) Aboriginal people, 2) the perspective of social assistance clients, and 3) Third Way ideology. To conclude this article, a research agenda on ideas and policy change is proposed.

La qualité des évaluations représente un enjeu fondamental au sein du mouvement de la gestion et des politiques fondées sur des données probantes. Le Secrétariat du Conseil du Trésor du Canada (SCT) a ainsi réalisé en 2004 un examen de la qualité des évaluations fédérales d’une envergure considérable. Au regard de sa pertinence, de sa méthodologie, et de la crédibilité de ses conclusions, cette méta-évaluation est-elle réussie? Cet article offre une réponse à cette question en présentant une évaluation de l’examen du SCT. Malgré des lacunes importantes au niveau de la théorie et des critères de qualité, du devis de recherche, du processus de codage, et de l’analyse des données, cette méta-évaluation présente des conclusions qui sont dans l’ensemble pertinentes et d’un niveau de crédibilité acceptable. Les enseignements que l’on peut tirer de cet examen sont proposés à titre de recommandations à l’intention des évaluateurs et des responsables des services d’évaluation désirant réaliser un examen similaire mais qui ne souffrirait pas des mêmes lacunes., Evaluation quality is a fundamental issue within the evidence- based management and policy movement. The Treasury Board Secretariat of Canada (TBS) conducted a wide-ranging review of the quality of federal evaluations in 2004. In terms of its relevance and methodology and the credibility of its conclusions, is the metaevaluation a success? This article answers the question by presenting an evaluation of the TBS quality review. Despite serious shortcomings with respect to the quality theory and criteria, design, coding process, and data analysis, the meta-evaluation conclusions are relevant and credible overall. Lessons learned from this quality review are proposed as recommendations for evaluators and officials responsible for evaluation units who wish to conduct a similar review that does not suffer from the same shortcomings.

No abstract

The concept of policy paradigm, which was developed by Peter Hall (1993), has given rise to an abundance of literature on ideas and policy dynamics. Despite its theoretical value, however, the original concept was insufficiently specified and has been used in problematic ways by many policy scholars, thereby undermining the validity of their descriptive and causal inferences. The main argument defended in this article is that scholars studying policy paradigms should devote as much attention to the ideas of policy makers as to the policies they adopt, while distinguishing between the two constructs as much as possible. This article reviews these issues and put forward seven guidelines that aim to better circumscribe the concept of policy paradigms and improve research on ideas and policy dynamics.

Systematic review and synthesis methods have gained wide acceptance within the social sciences and, as a result, many postgraduate students now consider using them for their thesis or dissertation research. However, students are rarely aware of all the concrete implications that their decision entails. This reflective narrative reports the experience of a political science student who began to conduct a systematic review as part of his PhD dissertation but who did not complete it. The aim of this article is to identify challenges and lessons learned from this experience and to formulate recommendations for postgraduate students who wish to make an informed choice with respect to the use of these methods.

La gestion horizontale pose des enjeux significatifs, en particulier en ce qui a trait au suivi et à l’évaluation de la performance des programmes publics. Cette étude de cas porte sur le Supplément à la prime au travail (SPT), une mesure sociofiscale de type interministériel, qui n’a fait l’objet d’aucune évaluation de sa performance depuis son introduction en 2008. À partir d’entretiens qualitatifs avec des acteurs publics, nous tentons d’expliquer pourquoi le SPT a pris la forme d’une mesure sociofiscale gérée par trois organisations publiques (ministère des Finances du Québec, Revenu Québec et le ministère du Travail, de l’Emploi et de la Solidarité sociale). Nous analysons ensuite les implications de cette horizontalité pour le suivi et l’évaluation de la performance de la mesure. Nous concluons à un décalage entre les responsabilités officielles et officieuses pour l’évaluation du SPT, l’intérêt d’une telle évaluation pour les organisations impliquées, et leur capacité à la réaliser., Horizontal management raises significant challenges, particularly when assessing the performance of public programs. This case study focuses on Quebec’s Supplement to the Work Premium (SWP), a social and fiscal cross‐ministerial measure, which was never subjected to any performance assessment since its introduction in 2008. We report on findings from interviews with public servants and attempt to explain how the SWP became a social and fiscal measure managed by three public organizations (ministère des Finances du Québec, Revenu Québec and the ministère du Travail, de l’Emploi et de la Solidarité sociale). Then, we analyze the implementation and performance evaluation of the measure. We argue there is a disjuncture between official and informal responsibilities when assessing the SWP and between the interests of the organizations involved in the assessment.

Program take-up is a necessary condition for program effectiveness. Yet, non-take-up is a significant challenge for many social programs, including Québec’s Supplement to the Work Premium (SWP), a refundable tax credit targeted toward long-term welfare clients. Based on interviews with 21 public actors and 46 program participants and nonparticipants, this study explains how low program awareness, the low value of the benefit and the significant administrative burden borne by potential and actual participants contribute to the non-take-up of the SWP in this sample. Moreover, four policy implications and recommendations that can inform the design, implementation, and evaluation of social programs, are derived from this study.

Program take-up is a necessary condition for program effectiveness. Yet, non-take-up is a significant challenge for many social programs, including Québec’s Supplement to the Work Premium (SWP), a refundable tax credit targeted toward long-term welfare clients. Based on interviews with 21 public actors and 46 program participants and nonparticipants, this study explains how low program awareness, the low value of the benefit and the significant administrative burden borne by potential and actual participants contribute to the non-take-up of the SWP in this sample. Moreover, four policy implications and recommendations that can inform the design, implementation, and evaluation of social programs, are derived from this study.

Dans un contexte où les politiques d’aide financière de dernier recours visent de plus en plus la réintégration au marché du travail, cette étude qualitative traite de l’expérience des prestataires d’assistance sociale (AS) de longue durée au Québec ainsi que de leur rapport au travail. Les analyses d’entretiens semi-dirigés auprès de 46 prestataires et ex-prestataires ayant effectué une transition vers le travail mettent en lumière les nombreux obstacles à une éventuelle transition vers le travail. Les prestataires accordent généralement une grande valeur au travail fondée sur ses divers avantages psychologiques et financiers. Ces avantages prennent part au processus décisionnel des prestataires par rapport à l’AS et au marché du travail. Le sentiment de sécurité que procure l’AS, combiné aux obstacles administratifs qu’imposent une sortie et une éventuelle réintégration du dispositif d’AS, en cas de nécessité, viennent cependant se placer comme éléments dissuasifs à une sortie de l’AS., Grounded in a context where last-resort financial assistance policies are increasingly aimed at facilitating reintegration into the labour market, this qualitative study examines the experiences of long-term social assistance (SA) recipients in Quebec, as well as their relationship to work. Analyses of semi-structured interviews with 46 recipients and “ex-recipients” who have made a transition back to work highlight the many obstacles to reintegrating into the workforce. Recipients generally place a high value on work, based on its various psychological and financial benefits. These benefits influence the decision-making process of recipients with respect to both SA and the labour market. However, the sense of security provided by SA, combined with the administrative obstacles imposed when seeking to leave the SA system or to re-enter it when necessary, act as deterrents to getting off SA.

L’ouvrage collectif dirigé par Daigneault et Pétry représente l’un des rares efforts systématiques visant à explorer, en français, les méthodes d’analyse de contenu automatisées de textes principalement politiques. Les chapitres couvrent un large éventail de thèmes et de techniques d’analyse de contenu – allant de la compilation des fréquences de mots jusqu’au positionnement idéologique des acteurs à l’aide de Wordfish et Wordscores, en passant par l’analyse de sentiments. À une époque où les textes électroniques sont devenus facilement disponibles, ce volume inspirera de nombreux chercheurs travaillant sur des techniques d’analyse de contenu, à partir d’une gamme de plus en plus large de plateformes et de langues.