Search

Each year a great many evaluations are conducted of international development efforts around the world. These development evaluations study projects, programs, country-wide portfolios, policy reform efforts, and other topics of interest to funders, governments, program managers, and other involved stakeholders. Although some of these evaluations are well funded, and while a rare evaluation might possibly even be overfunded, the majority of individual development evaluations are probably underfunded. This underfunding can produce several implications for a development evaluation: "logistical" implications such as the number of evaluators that can be involved and the amount of travel possible, "methodological" implications such as the types of study designs possible and the types and amount of data that can be gathered, and even "political" implications such as who can be involved in the evaluation, at what stages, and in what ways. Many of these implications are well known, especially among experienced development evaluators, but the authors wonder if there might also be one additional set of implications of this underfunding--implications which they believe have not been sufficiently discussed to date. Specifically, the authors wonder if there might also be "ethical" implications of underfunding development evaluations. This article focuses attention on what the authors are calling the microlevel (project and program level) ethical implications of the underfunding of individual development evaluations. (Contains 1 table.)

The purpose of this article is to extend the discussion of issues currently being debated on the need for more rigorous program evaluation in educational and other sectors of research, to the field of international development evaluation, reviewing the different approaches which can be adopted to rigorous evaluation methodology and their applicability in a development setting. The authors seek common ground on ways to strengthen the methodological rigor and quality of development impact evaluations, while adapting the methodology to the technical, administrative, political and sociocultural contexts within which these evaluations are developed, implemented and used. The authors conclude that given the current limitations of randomized control trials (RCTs) that it is necessary to identify other means of undertaking impact studies that address the time and budget constraints under which evaluators are frequently forced to operate. The report presents a range of designs with a range of costs and rigor. The authors suggest that where the most rigorous designs are not possible then a good theory-based approach will lend plausibility to the findings. (Contains 1 table.)

The paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the "Shoestring Evaluation" approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced. Although the approach was designed to assist evaluators working in developing countries, the principles are equally applicable for conducting evaluations under similar constraints in industrial nations. Case studies are included to illustrate the different Shoestring approaches, drawing from the authors' experience in Eastern Europe, Africa, Asia, and Latin America.

Describes differences between program evaluation in the United States and international program evaluation. Also describes a number of promising developments and methodological approaches and suggests areas in which a closer exchange of experience between U.S. evaluation practitioners and their colleagues from developing countries could be beneficial. (SLD)

The political context of evaluation is compared for the United States and developing countries, and a framework is proposed for discussing the political context of evaluation. Significant differences are the role that international donor agencies play in developing countries and the emphasis on stakeholder evaluation in the United States. (SLD)