Search

Automated content analysis methods treat ‘‘text as data'' and can therefore analyze efficiently large qualitative databases. Yet, despite their potential, these methods are rarely used to supplement qualitative analysis in small-N designs. We address this gap by replicating the qualitative findings of a case study of a social policy reform using automated content analysis. To characterize the ideology of this reform, we reanalyze the same interview data with Wordscores, using academic publications as reference texts. As expected, the reform's ideology is center/center-right, a result that we validate using content, convergent and discriminant strategies. The validation evidence suggests not only that the ideological positioning of the policy reform is credible, but also that Wordscores' scope of application is greater than expected

The concept of explanation has attracted considerable attention in the social sciences and, in particular, in political science. However, scholars are not always familiar with what explaining political phenomena means, let alone with what it entails for developing sound causal arguments. This essay introduces Craig Parsons’ fourfold typology of explanation (institutional, ideational, structural, and psychological) before assessing its value for the causal analysis of political behaviour and processes. As argued, despite its limitations, his typology clearly maps the explanations in political science while helping scholars to combine them more rigorously, when needed. This is why Parsons typology has the potential to move political scientists to the ‘next level’, as far as explanation is concerned.

Background. Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. Objectives. This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines 1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; 2) the convergence between the scores of coders and those of key respondents (i.e., authors); and 3) the convergence between the authors’ scores on the PEMI and the Evaluation Involvement Scale (EIS). Sample. A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Measures. Stakeholder participation was measured on nominal and ordinal scales. Cohen’s kappa, the intraclass correlation coefficient and Spearman’s rho were used to assess reliability and convergence. Results. Reliability results ranged from fair to excellent. Convergence between coders’ and authors’ scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Conclusions. Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.

Plusieurs provinces canadiennes ont mis en œuvre des réformes substantielles de leur régime d’aide sociale dans les années 1990. Or, nos connaissances à propos des idées qui sous-tendent ces réformes sont étonnamment limitées. En particulier, peu d’études empiriques ont directement examiné les idées des acteurs de politique. Cet article présente une étude de cas rétrospective d’une initiative saskatchewanaise majeure, soit la réforme Building Independence (BI), et examine sa congruence avec trois paradigmes d’aide sociale. Les données proviennent d’entrevues avec des acteurs de politique. Cette étude conclut que les idées sous-jacentes à BI sont étroitement alignées avec le paradigme de l’activation, mais présente certaines similitudes avec les paradigmes du droit social (entitlement) et du workfare. Ce résultat est ensuite discuté à la lumière de trois enjeux : 1) les autochtones, 2) la perspective des prestataires d’aide sociale, et 3) l’idéologie de la Troisième voie. Des pistes de recherche sur les idées et le changement de politique viennent conclure cet article., Many provinces have enacted substantial reforms of their social assistance regime in the 1990s. However, we know surprisingly little about the ideas that underlie welfare reform in Canada. In particular, few empirical studies have directly examined the ideas of policy actors. This article presents a retrospective case study of a major policy initiative, namely Saskatchewan’s Building Independence (BI), and examines its alignment with three paradigms of social assistance. Data come primarily from interviews with policy actors. This study concludes that the policy ideas informing BI align closely with the activation paradigm, but also share some similarities with the entitlement and workfare paradigms. The significance of this finding is then discussed in light of three issues: 1) Aboriginal people, 2) the perspective of social assistance clients, and 3) Third Way ideology. To conclude this article, a research agenda on ideas and policy change is proposed.

La qualité des évaluations représente un enjeu fondamental au sein du mouvement de la gestion et des politiques fondées sur des données probantes. Le Secrétariat du Conseil du Trésor du Canada (SCT) a ainsi réalisé en 2004 un examen de la qualité des évaluations fédérales d’une envergure considérable. Au regard de sa pertinence, de sa méthodologie, et de la crédibilité de ses conclusions, cette méta-évaluation est-elle réussie? Cet article offre une réponse à cette question en présentant une évaluation de l’examen du SCT. Malgré des lacunes importantes au niveau de la théorie et des critères de qualité, du devis de recherche, du processus de codage, et de l’analyse des données, cette méta-évaluation présente des conclusions qui sont dans l’ensemble pertinentes et d’un niveau de crédibilité acceptable. Les enseignements que l’on peut tirer de cet examen sont proposés à titre de recommandations à l’intention des évaluateurs et des responsables des services d’évaluation désirant réaliser un examen similaire mais qui ne souffrirait pas des mêmes lacunes., Evaluation quality is a fundamental issue within the evidence- based management and policy movement. The Treasury Board Secretariat of Canada (TBS) conducted a wide-ranging review of the quality of federal evaluations in 2004. In terms of its relevance and methodology and the credibility of its conclusions, is the metaevaluation a success? This article answers the question by presenting an evaluation of the TBS quality review. Despite serious shortcomings with respect to the quality theory and criteria, design, coding process, and data analysis, the meta-evaluation conclusions are relevant and credible overall. Lessons learned from this quality review are proposed as recommendations for evaluators and officials responsible for evaluation units who wish to conduct a similar review that does not suffer from the same shortcomings.

No abstract

The concept of policy paradigm, which was developed by Peter Hall (1993), has given rise to an abundance of literature on ideas and policy dynamics. Despite its theoretical value, however, the original concept was insufficiently specified and has been used in problematic ways by many policy scholars, thereby undermining the validity of their descriptive and causal inferences. The main argument defended in this article is that scholars studying policy paradigms should devote as much attention to the ideas of policy makers as to the policies they adopt, while distinguishing between the two constructs as much as possible. This article reviews these issues and put forward seven guidelines that aim to better circumscribe the concept of policy paradigms and improve research on ideas and policy dynamics.

Systematic review and synthesis methods have gained wide acceptance within the social sciences and, as a result, many postgraduate students now consider using them for their thesis or dissertation research. However, students are rarely aware of all the concrete implications that their decision entails. This reflective narrative reports the experience of a political science student who began to conduct a systematic review as part of his PhD dissertation but who did not complete it. The aim of this article is to identify challenges and lessons learned from this experience and to formulate recommendations for postgraduate students who wish to make an informed choice with respect to the use of these methods.