Search

We consider the conditional randomization test as a way to account for covariate imbalance in randomized experiments. The test accounts for covariate imbalance by comparing the observed test statistic to the null distribution of the test statistic conditional on the observed covariate imbalance. We prove that the conditional randomization test has the correct significance level and introduce original notation to describe covariate balance more formally. Through simulation, we verify that conditional randomization tests behave like more traditional forms of covariate adjustmet but have the added benefit of having the correct conditional significance level. Finally, we apply the approach to a randomized product marketing experiment where covariate information was collected after randomization.

We persuaded entities conducting two legal aid programs designed to provide evidence regarding a civil right to counsel to allow us to randomize which potential clients would receive offers of traditional attorney-client relationships from legal aid provider staff attorneys and which would receive only limited (“unbundled”) assistance. In both pilot programs, potential clients were occupants facing eviction from their housing units, and in neither pilot program did the legal aid provider have capacity sufficient to offer full representation to all occupants who sought it. In this Article, we report the results of one of the two randomized trials, which we label the “District Court Study” after the type of court in which it took place. In this District Court Study, most occupants who became part of the study population received limited assistance in how-to sessions, which included instruction on the summary eviction process as well as help in filling out answer and discovery request forms. After receiving this “unbundled” assistance, members of a randomly selected treated group were offered a traditional attorney-client relationship from a legal aid provider staff attorney; members of the remaining randomly selected control group received no such offer. We compared outcomes for the treated group versus the control group on a variety of dimensions, focusing primarily on possession of the unit, financial consequences of the litigation, and measures of court burden. At least for the clientele involved in this District Court Study – a clientele recruited and chosen by the legal aid provider’s proactive, timely, specific, and selective outreach and intake system – an offer of full representation mattered. Approximately two-thirds of occupants in the treated group, versus about one-third of occupants in the control group, retained possession of their units at the end of litigation. Using a conservative proxy for financial consequences, and based on a subset of cases in which financial issues were at the forefront, treated-group occupants received payments or rent waivers worth on average a net of 9.4 months of rent per case, versus 1.9 months of rent per case in the control group. Both results were statistically significant. Meanwhile, although treated cases did take longer to reach judgment, the offer of representation caused no increase in court burden as measured by other, more salient metrics, such as the number of party motions or the quantity of judicial rulings. We discuss possible reasons for the magnitude of the differences between the outcomes experienced by the treated and control groups. For example, following previous work, we discuss the possible importance of the legal aid provider’s process for client recruitment and selection. Here, the provider invested substantial resources into a system designed to recruit and identify clients for whom unbundled legal assistance would be inadequate, suggesting that identifying such cases can be done but that doing so may be expensive. We conclude by discussing future directions for a movement, growing in momentum, toward an evidence-based approach for access to, and administration of, justice.

Randomization of treatment assignment in experiments generates treatment groups with approximately balanced baseline covariates. However, in observational studies, where treatment assignment is not random, patients in the active treatment and control groups often differ on crucial covariates that are related to outcomes. These covariate imbalances can lead to biased treatment effect estimates. The propensity score is the probability that a patient with particular baseline characteristics is assigned to active treatment rather than control. Though propensity scores are unknown in observational studies, by matching or subclassifying patients on estimated propensity scores, we can design observational studies that parallel randomized experiments, with approximate balance on observed covariates. Observational study designs based on estimated propensity scores can generate approximately unbiased treatment effect estimates. Critically, propensity score designs should be created without access to outcomes, mirroring the separation of study design and outcome analysis in randomized experiments. This paper describes the potential outcomes framework for causal inference and best practices for designing observational studies with propensity scores. We discuss the use of propensity scores in two studies assessing the effectiveness and risks of antifibrinolytic drugs during cardiac surgery., Statistics