Search

Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data on study students' performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization. We propose and demonstrate a method that leverages auxiliary non-experimental "remnant" data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in educational technologies to make more informed decisions when approaching analyses. [This paper will be published in the "Educational Technology Research and Development."]

Background: Implementation quality is a significant factor in determining whether a program has its intended impact (Durlak and DuPre, 2008; List et al., 2021). Therefore, when studying a program's efficacy, researchers must ensure that the program is implemented as intended to estimate its impact. To evaluate whether a program is being implemented properly in the classroom during a study, researchers generally conduct classroom observations or rely on teacher self-reports (Dart et al., 2012; Schoenwald et. al., 2011). However, live observations are time-consuming and costly because they require multiple observations for each teacher to reliably estimate fidelity metrics (Cross et al., 2015; Gresham, et. al, 2017). Although self-reposts require fewer resources, they are less reliable (Gresham et al., 2017; Lee et al., 2008). In this study, we explore how data automatically collected by online learning platforms can be leveraged to evaluate implementations and how these data can help us understand who has access to high-fidelity environments. Purpose/Objective/Research Question: (1) Which variables should be used as metrics of students' dosage and engagement across conditions? (2) How much of the variance in student-level dosage and engagement metrics are associated with the students' implementation contexts? (3) Can we classify high and low-fidelity implementation contexts using the dosage and engagement metrics? (4) Do students of different demographics receive different levels of fidelity? Setting: The analyses for the current study used data from a randomized control trial described in Section 3. This study employed student-level randomization into the following programs: two gamified conditions (OLP1 and OLP2), OLP3 of traditional problems with immediate feedback, and OLP3 without immediate feedback (active control). Population: Initially, 52 seventh-grade mathematics teachers from 11 schools were recruited to participate in the study from a large, suburban district in the Southeastern United States during the summer of 2020. The district provided demographic data for the 3,434 participating students. Of the analytic sample, 48.88% of the students were identified as White, 27.14% as Asian, 14.04% as Hispanic, 4.86% as Black, 0.69% as Native American, 0.04% as Pacific Islander, and 2.98% as two or more races. The district did not provide data for 1.37% of the students. About half the students were identified by the district as female (48.14%) and male (51.81%), with less than 1% of the population not reporting gender. The district did not provide any data on non-binary students. About one-tenth (9.10%) of students had an Individualized Education Plan. Programs: During the school year, teachers were expected to provide students with 13 half-hour minute sessions to complete 30 assignments within their respective OLP. Each OLP used a different method of teaching algebraic knowledge. The first, sixth, twelfth, and thirteenth sessions were 40-45 minute assessments of their algebra knowledge and attitudes toward math. Sessions 2-5 and 7-11 were activities in their assigned technology. In both the assessments and the activity sessions, students' usage and progress for each program were captured by their respective technologies. As students solved problems in the systems, the platforms also logged information about students' actions and performance. Design: To create metrics of student dosage and engagement (RQ1), we first calculated variables from the log-file data of each program, conducted exploratory data analysis, and standardized fidelity metrics within programs. Variables identified as suitable measures of dosage and engagement across programs in RQ1 are used to produce fidelity Setting: The analyses for the current study used data from a randomized control trial described in Section 3. This study employed student-level randomization into the following programs: two gamified conditions (OLP1 and OLP2), OLP3 of traditional problems with immediate feedback, and OLP3 without immediate feedback (active control). Population: Initially, 52 seventh-grade mathematics teachers from 11 schools were recruited to participate in the study from a large, suburban district in the Southeastern United States during the summer of 2020. The district provided demographic data for the 3,434 participating students. Of the analytic sample, 48.88% of the students were identified as White, 27.14% as Asian, 14.04% as Hispanic, 4.86% as Black, 0.69% as Native American, 0.04% as Pacific Islander, and 2.98% as two or more races. The district did not provide data for 1.37% of the students. About half the students were identified by the district as female (48.14%) and male (51.81%), with less than 1% of the population not reporting gender. The district did not provide any data on non-binary students. About one-tenth (9.10%) of students had an Individualized Education Plan. Programs: During the school year, teachers were expected to provide students with 13 half-hour minute sessions to complete 30 assignments within their respective OLP. Each OLP used a different method of teaching algebraic knowledge. The first, sixth, twelfth, and thirteenth sessions were 40-45 minute assessments of their algebra knowledge and attitudes toward math. Sessions 2-5 and 7-11 were activities in their assigned technology. In both the assessments and the activity sessions, students' usage and progress for each program were captured by their respective technologies. As students solved problems in the systems, the platforms also logged information about students' actions and performance. Design: To create metrics of student dosage and engagement (RQ1), we first calculated variables from the log-file data of each program, conducted exploratory data analysis, and standardized fidelity metrics within programs. Variables identified as suitable measures of dosage and engagement across programs in RQ1 are used to produce fidelity metrics which are then used in subsequent analysis to address RQ2 and RQ3. Next, to evaluate how much of the dosage and engagement metrics variance is associated with components of the implementation context (RQ2), we estimated a series of multilevel models. Each model included random intercepts for school, teacher, and class regressed on the fidelity metrics identified and standardized for RQ1, and computed ICCs to evaluate the importance of the students' contexts in predicting their dosage and engagement. Next, we conducted a latent profile analysis to identify different fidelity profiles. We aggregated the fidelity metrics identified in RQ1 at the context level (school, teacher, or class) with the highest ICCs from the analysis for RQ2. Finally, to understand whether their students had different opportunities to be in high-fidelity contexts, we estimated logistic regression models regressing whether or not the students were in a high-fidelity context on their demographic data. Data Collection: Students' log-file data were collected by their respective programs. The researchers did not have access to the OLP2 database; however, before and after students solved problems in OLP2, they logged into OLP3 and answered a series of self-reported questions about their starting and ending progress in the program. All data from this study are available through the Open Science Foundation (https://osf.io/r3nf2/). Variables included in the analyses of fidelity are presented in Table 1. Results: Three metrics emerged as best suited to evaluate the students' dosage and engagement in their respective programs: the percentage of assignments students started, the total number of problems they attempted during the study, and their average problems completed per assignment (Table 2). Teachers had the highest intraclass correlations on each fidelity metric, suggesting that in this study, the teachers were the most important contextual factor in determining implementation fidelity (Table 3). Next, we identified two latent profiles of teachers: high and low fidelity (Figure 1). Finally, we found that students who were not White or Asian and those with Individualized Education Plans (IEPs) were significantly less likely to be taught by high-fidelity teachers compared with their peers (Table 4), yet these effects went away after accounting for students pretest algebraic knowledge scores. These findings reveal bias in access to high-fidelity implementations, where lower-performing students are less likely to have access, and these students are more likely to be students of color. These findings highlight the importance of implementation fidelity monitoring, and future work should how to ensure equitable access to high-quality programs in high-fidelity environments.

Background: Randomized controlled trials (RCTs) give unbiased estimates of average effects. However, positive effects for the majority of students may mask harmful effects for smaller subgroups, and RCTs often have too small a sample to estimate these subgroup effects. In many RCTs, covariate and outcome data are drawn from a larger database. For instance, educational efficacy studies may target state standardized tests scores while adjusting for student demographics and prior achievement, all of which are drawn from a state longitudinal data system. In these situations, typically only a subset of students in the database participate in the RCT. We refer to RCT non-participants in the database as the "remnant" from the RCT. The remnant often includes a larger sample size than the RCT--especially when estimating subgroup effects--but is not randomized, so including may cause bias. This talk will describe and illustrate a method incorporating the remnant into the analysis of RCTs that gives unbiased estimates and conservative standard errors and has the potential to improve the precision of effect estimates from an RCT, sometimes dramatically. Research Questions: The talk will use data from RCTs that were run within an online tutoring system to answer three research questions: (1) To what extent might remnant data improve the precision of effect estimates from RCTs? (2) Can incorporating the remnant improve estimation of subgroup effects? (3) If the remnant is known to be drawn from a different population than the participants in A/B tests, can it still be useful? Data: E-Trials is a platform that allows researchers to design educational experiments that will then be run within the ASSISTments online tutor. Students working on "skill-builder" modules are randomized between two or more conditions, such as how subject matter is portrayed, available hints, and feedback to students. We analyzed data from 277 contrasts between pairs of treatment arms drawn from 68 multi-armed RCTs, including 113,963 students. We also collected rich clickstream-level log data from RCT participants as well as from a remnant of 193,218 ASSISTments users who did not participate in any of the RCTs we analyzed. Assignment completion was the outcome of interest across RCTs. Methodology: The methodology (Gagnon-Bartsch et al. 2023) builds on prior literature including Sales, et al (2018ab), Wu and Gagnon-Bartsch (2018), and Aronow and Middleton (2013). It involves three steps: (1) using only remnant data, train a model $f^R(\cdot):\mathbb{R}^p\rightarrow\mathbb{R}$ predicting outcomes Y as a function of a pdimensional covariate matrix X. This model can be of any form, including machine learning algorithms, and may be fit and tested multiple times, as long as only remnant data is used. It need not be correct, unbiased, or consistent in any sense, but should ideally yield accurate out-of-sample predictions. (2) Use the fitted model $\hat{f}(\cdot)$, along with covariate data from RCT participants X^{RCT} to predict RCT outcomes as $\hat{y}_C=\hat{f}(X^{RCT})$. (3) Use $\hat{y}_C$, as a covariate in a design-based covariate-adjusted effect estimator (e.g. Rosenbaum 2002, Wager, et al. 2016) perhaps alongside other covariates. In our analysis we use the LOOP estimator of Wu and Gagnon-Bartsch (2018), and refer to the estimator including only $\hat{y}_C$ as a covariate as "ReLOOP" (i.e. Remnant-based LOOP) and the estimator also including other covariates as "ReLOOP+." To carry out the method, we fit an ensemble model (Fig. 1) including three deep neural networks (e.g. Goodfellow, et al. 2016), each trained on a different set of covariates: data on each of the previous 20 assignments each students started, actions students took on each of the prior 60 days, and aggregated student-level data. These three models were then ensembled via a fourth feed-forward neural network. The same fitted model was used for all subgroup comparisons. To test the performance of our methods when the remnant is demographically distinct from the RCT, we used the "gender guesser" python script https://pypi.org/project/gender-guesser/ using students' first names, categorizing students as Male, Female, or Unknown. We fit a model to the subset of the remnant categorized as "Male"--since gender guesser is imperfect and trained using Eurasian names, we assume this group is mostly male and disproportionally white or Asian--and estimated effects for RCT participants from the other categories. Results: All of the estimators we consider are exactly unbiased, so our results focus on estimated sampling-variance of ReLOOP, ReLOOP+, the difference-in-means estimator (Neyman 2023) or the LOOP estimator without remnant data. Specifically, we consider sampling-variance ratios: since sampling-variance scales as 1/n, these can be thought of as "sample size multipliers." RQ1: Figure 2 shows boxplots of sampling variance ratios comparing ReLOOP or ReLOOP+ to difference-in-means (Labeled "T-Test") or LOOP. Incorporating $\hat{y}_C$ into the analysis never substantially hurt precision, but in many cases improved precision by as much as 30% or more. RQ2: Figures 3 shows boxplots of sampling-variance ratios for subgroup effects, and Figure 4 plots these ratios as a function of sample size within the subgroup. On average, ReLOOP+ improved precision for subgroup estimation for all sample sizes. For small subgroup samples, the improvement could be dramatic--equivalent to more than doubling the sample size in some cases--but in some other cases it hurt precision noticeably. RQ3: Figure 5 shows results from the experiment using the "male" remnant to estimate effects in the non-"male" subset of the RCTs. Surprisingly, ReLOOP improved precision roughly equally among "male" and non-"male" subgroups of the RCT, and ReLOOP+ improved precision more in the non-"male" subgroup, despite the unrepresentativeness of the remnant. Conclusion: Machine learning methods have an impressive potential, but can also reproduce biases present in their training samples (e.g. Bolukbasi et al. 2016). Randomized Controlled Trials give unbiased estimates (for the RCT participants) but these may be imprecise and mask treatment effect heterogeneity. The methods we propose, ReLOOP and ReLOOP+ use machine learning models fit to auxiliary data that are unbiased, like RCTs, but can also be more precise--especially for subgroup estimates, even when the remnant is itself biased. They can be valuable tools for using all available data to evaluate programs for both the majority of students as well as for vulnerable minorities.

Background: Persisting after experiencing difficulty allows students to work in the upper ends of their zones of proximal development, where most learning occurs (Ventura et al., 2013; Vygotsky & Cole, 1978). Digital educational games promote productive persistence by allowing students to repeat the same or similar problems until they reach mastery (Owen et al., 2019). One way gamification may influence students' behavior is by encouraging them to persist by re-attempting problems when they have not achieved optimal performance. Gamification has shown mixed effects on student persistence (Gaston & Cooper, 2017; O'Rourke et al., 2016; Chase et al., 2021; Malkiewich et al., 2016). Therefore, the mechanisms that connect different gamification systems to student persistence and learning must be understood to ensure that learning games are developed effectively to support these outcomes. Several studies have found that performance-based rewards -- one type of gamification system -- serve as feedback that may support persistence and learning (Gaston & Cooper, 2017; O'Rourke et al., 2016). Yet, there is a growing appreciation that the effects of behavioral interventions are not homogeneous (Bryan et al., 2021). In this study, we use a regression discontinuity design and log data from an online gamified algebra platform to estimate the heterogeneous effects of a performance-based feedback system on students' persistence behavior. Program: We explore whether different levels of performance-based feedback cause students to persist when struggling in a math game-based online learning program. This program is effective at improving students' understanding of algebra (Chan et al., 2022; Woodrow-Decker et al., 2023) using perceptual learning (Goldstone et al., 2017) and embodied cognition (Abrahamson et al., 2020). Students are asked to transform a starting algebraic expression into a mathematically-equivalent goal state (Figure 1). Each manipulation of the equation counts as a step. Each problem has an optimal solution (i.e. a minimum number of steps to complete the problem). Students receive feedback based on their proximity to the optimal solution (Figure 2). The clovers act as performance contingent reward: three clovers for optimal performance, two clovers if the student was within two steps, and one clover if they were over two steps of optimal. Students are encouraged to replay the problem if their original solution is suboptimal. Research Questions: 1. Do students respond to differences in performance-based rewards by persisting after encountering a challenging problem? 2. Does the effect of the performance-based reward vary based on student characteristics, such as gender, race, participation in English to Speakers of Other Languages (ESOL) classes, and prior knowledge? Setting: The current study utilizes secondary data from a larger efficacy study, which took place during the 2021-22 school year (Decker-Woodrow, In-press). A total of 52 seventh-grade mathematics teachers and their students from 11 middle schools were recruited from a large, suburban district in the Southeastern United States. Although the efficacy study included four conditions, we focus on the abovementioned gamified condition. Participants: The demographics of students randomly assigned to the gamified condition are presented in Table 1. The analysis included 1009 students completing 26,941 attempts of 245 different problems. Design: Since the performance rewards were determined using the number of steps required to complete a problem relative to cut points, the data represent a regression discontinuity design, where the treatment (Z) is receiving a higher performance-based reward (2 clovers vs. 1 clover), and the (discrete) running variable (R) is the students' numbers of steps over optimal. More specifically, we utilize the method established by Sales & Hansen (2020) of limitless regression discontinuity (LRD), which allows for the estimation of an average treatment effect for all subjects in a window around the cutoff, and which is amenable to discrete running variables. LRD is based on the assumption of "residual ignorability;" that the results for control potential outcomes are independent of treatment assignment. Therefore the difference in the average partial residuals generated using the predicted potential outcome assuming no treatment between treatment and control is an estimate of the causal effect. We implement this method using multi-level logistics regressions run using the glmer package in R (Bates et al., 2015), regressing whether the student replayed a problem on their steps over optimal and the cutpoint indicator. The coefficient associated with the cut point is an estimate of the Local Treatment Effect (LATE), and the partial residuals are used to calculate the Limitless Average Treatment Effect (ATE). We included random intercepts for the problems and students. We included random effects of receiving a higher reward to examine the variability of the effects across students. Results: Table 2 presents the regression discontinuity models, and Table 3 presents the LATE and limitless ATE in terms of the probability of replaying the problem. Interactions between the treatment and race/ethnicity or prior algebraic knowledge were non-significant and, therefore, not presented in the table. Overall the treatment of the higher reward decreases the likelihood that students will replay the problem. Notably, the effect is less for male students' experience than for female students and less for ESOL students than those not enrolled in ESOL. Finally, as the effect varied greatly by the student (tau=0.61), we estimated the ATE for each student within their attempted problems (Figure 3). Conclusions: These findings suggest that rewards can have a small but significant effect on students' persistence behaviors in a gamified learning program, but the impacts of the rewards vary greatly. Overall receiving the higher reward demotivated students to replay the problem compared with receiving the lower reward. This suggests that, on average, students are motivated to replay problems by rewards that provide feedback indicating game-like failure. Yet, despite the average effect, students respond differently based on this type of feedback. The smaller effect for ESOL students may be attributed to difficulties understanding the user experience, as the game was presented exclusively in English. The significant difference in feedback effects between males and females is unexpected and should be explored further. Finally, the unexplained heterogeneity of effects (e.g., the variance random effects) suggests the necessity of future research on who is influenced by the gamified elements of this program.

Background: The data revolution in education has led to more data collection, more randomized controlled trials (RCTs), and more data collection within RCTs. Often following IES recommendations, researchers studying program effectiveness gather data on how the intervention was implemented. Educational implementation data can be complex, including longitudinal measurements of several indicators of different types. When implementation fidelity is measured along a single variable, instrumental variables techniques [Angrist et al., 1996] may apply, estimating average treatment effects for subjects who implement the program to a certain extent. When implementation can take several forms, principal stratification (PS) [e.g. Frangakis and Rubin, 2002, Page, 2012, Feller et al., 2016] can play that role. However, when measures of implementation are longitudinal, highly multivariate and/or complex, traditional PS methods can fail [Sales et al., 2019]. This talk will describe and illustrate an extension of the PS framework, called "Fully-Latent Principal Stratification," (FLPS) that incorporates latent-variable measurement modeling into PS. Purpose: It is widely acknowledged that variance in program implementation can lead to variance in treatment effects, but it is unclear what statistical models are capable of capturing that variance. We will describe a framework for modeling treatment effects that vary with varying implementation and illustrate it in a secondary analysis of a randomized controlled trial (RCT) comparing two online mathematics tutors, one that offered students hints and immediate error feedback, and one that only offered feedback after a delay. The goal will be to learn if the effect of offering immediate feedback tends to be higher for students who receive more feedback. Data: 1,141 7th-grade students were blocked within classrooms, given an online pretest consisting of ten math questions, and individually-randomized between the two conditions. We excluded students who were missing either their pretest or their outcome measurements, leaving n=804 students; all other variables were imputed using a random forest imputation algorithm. Students from both treatment groups worked on the same set of problems within the tutor, but since our goal was to measure hints and feedback, we only modeled log data from the Immediate group, and from problems for hints were available and responses were marked either correct or incorrect. There were 212 different problems organized into nine "problem sets." Many problems had several parts, each of which had its own hints and was marked correct or incorrect separately; hence, we modeled each problem part--298 in total--on its own. The outcome of interest was student performance on a state standardized mathematics test. Methodology: In a randomized trial, let i represent the treatment e ect for student i, i = 1; : : : ; n. In the classic PS framework, say implementation is measured with a variable Mi, only available in the treatment group. Let MTi represent the value of M that student i would exhibit if assigned to the treatment condition. The goal of PS is to estimate E[jMT = m], the average treatment e ect for subjects who would implement as M = m, for some value of m. In this context, MT is observed as M in the treatment group, but unobserved in the control group; it must be imputed based on modeling assumptions and/or baseline covariates X. If implementation measurements are multivariate, longitudinal, or complex they may not be representable as a univariate measurement M without measurement error. Instead, represent them as a vector for each subjectMi and assume a measurement model p(mjT), relating implementation measurements to a latent variable T, representing students' latent propensity to implement in a certain way if assigned to treatment. In our example, we consider one- and two-parameter logistic models (1PL and 2PL) as well as a graded response model (GRM) for student feedback. Given a models for T as a function of X and Y as a function of treatment assignment, T, and X, we may use Bayesian techniques to estimate a fully-latent principal e ect E[jT], the e ect of randomization to the treatment condition for subjects with etaT propensity to implement the program. In our illustration, we let mij be either an indicator of whether student i answered question j correctly without receiving feedback or, for the GRM, an ordered categorical variable for whether student i answered question j correctly on the first try, received either a hint or an error message, but was not told the answer, or whether a student received the answer as part of their feedback. Results: Figure 1 shows the correlations between estimated student effects T for all three measurement models (1PL, 2PL, and GRM) and the percentage of problems on which students received feedback. Estimates from all methods were highly correlated, with the highest correlation between latent variables in the measurement models and slightly lower correlation with the proportion correct on the first try (i.e. no feedback). Table 1 shows the the coefficients on T, treatment assignment, and their interaction in the outcome model, for models run with each of the three measurement models we considered. When feedback was modeled via the proportion of questions students answered correctly on the first try ("Classical"), we estimated the treatment effect to decrease with feedback receipt. When we modeled feedback receipt with measurement models, the trend was in the opposite direction. However, in all four cases trends of opposite signs are also consistent with the data--i.e. the estimates were not statistically significant. Conclusions: There is a serious need for statistical models relating program implementation to effects, to help answer questions about when programs are effective, and what styles of implementation led to the greatest effects. However, the rich, informative implementation data gathered automatically in computerized educational application present both an opportunity and a challenge: they can provide deep and nuanced pictures of varying implementation, but also require more complicated models in an already difficulty-to-model context. FLPS can serve as a framework in which to develop and interpret such models. The wide range of models from psychometrics can be marshalled to summarize complex multidimensional implementation data into low-dimensional, interpretable latent variables; then, the framework of principal stratification, along with Bayesian modeling, can show us the relationship between implementation and effectiveness.

Education researchers frequently have to choose between statistical models for their data, and in many cases the candidate models or parameters can be listed in a sequence, m=1,...,M, from less preferable choices to more. For instance, in choosing a bandwidth for regression discontinuity designs, researchers would favor the largest possible bandwidth; in setting criteria for propensity matching, researchers would prefer the set of criteria that includes the largest number of subjects. One option for model specification in these scenarios relies on p-values for goodness-of-fit or specification tests, say p[subscript 1,]....,p[subscript M]. Given a threshold [alpha], often [alpha]=0.05, a researcher could choose the most preferable model that passes the test, "max"{m: p[subscript m]>[alpha]} or the largest model before the first rejection "max"{m*: p[subscript m]>[alpha] [turned A] m[less than or equal to]m*}. However, it is typically unclear how to choose [alpha], or whether the framework of null-hypothesis testing is appropriate for model selection, in which the goal is, essentially, to accept the null hypothesis. This talk will introduce an alternative model selector based on p-values p[subscript 1,]....,p[subscript M] that uses p-values for estimation, rather than for testing, and that does not require researchers to choose a threshold. The method, which originated in the change-point literature (Mallik et al. 2011), proceeds from the observation that when the null hypothesis is true, p-values are distributed as "Uniform"(0,1), with a mean of E[p]=0.5, and when the alternative hypothesis is true, p[right arrow]0 as sample sizes increase. This suggests a least-squares estimate of the optimal model, m[superscript cp]= argmin[subscript m*] [sigma subscript m(less than or equal to)m*](p[subscript m]-1/2)[superscript 2]+[sigma subscript m>m*](pm)[superscript 2], that is, the model choice such that p-values for acceptable models, those with m[less than or equal to]m*, are clustered around 1/2, while those for unacceptable models are small. This change-point selector m[superscript cp] uses the properties of p-values from goodness-of-fit tests to "estimate" the optimal model, rather than to choose which null hypotheses to reject and which to (implicitly) accept. Because m[superscript cp] is based on the full array of p-values, one errant p-value will not drive the entire choice. This talk will evaluate the performance of model selector m[superscript cp], relative to more conventional selection rules, in a simulation study. The simulation study shows that m[superscript cp] represents a compromise between more liberal and more conservative conventional model selectors, and that its performance, while not always the best, is the most consistent across different circumstances of the model selectors included in the study. The talk will also illustrate m[superscript cp] in bandwidth selection for the regression discontinuity design introduced in Lindo, et al. (2010) estimating the effect of academic probation status on college students' subsequent grade point averages (see the attached figure). At each possible bandwidth, we conduct covariate placebo tests, by estimating effects of academic probation on baseline student covariates, and use m[superscript cp], along with threshold-based selectors to choose an optimal bandwidth. We also compare the results to the bandwidth selector proposed by Imbens and Kalyanaraman (2011).

In a randomized controlled trial (RCT), some subjects assigned to the treatment condition may not fully comply. Often there is interest in the effect of the treatment within the "principal stratum" of subjects who would comply if assigned to treatment. However, it is unknown which control subjects would have complied if treated and which wouldn't. (This talk will be limited to the case of "one-sided" non-compliance, in which compliance is binary, so that principal stratum membership is observed in one treatment group and unobserved in the other.) One approach to identifying potential compliers in the control group uses "principal scores," probabilities of compliance, conditional on covariates, or e(x[subscript 1])=Pr(S[subscript pi]=1|x[subscript 1]), where S[subscript pi]=1 if subject i complies in the treatment condition and S[subscript pi]=0 otherwise. To use principal scores, the analyst will typically either fit a parametric finite mixture model--for instance, if outcomes are continuous, assuming that p(Y[subscript Ci]|e(x[subscript 1]))= e(x[subscript 1])[phi]{(x-[mu subscript 1])/[sigma subscript 1]}+(1e(x[subscript 1]))[phi]{(x-[mu subscript 2])/[sigma subscript 2]}, where p(Y[subscript Ci]|e(x[subscript 1])) is the density of i's control potential outcome conditional on i's principal score, [phi] is the standard normal density, and [mu subscript 1], [mu subscript 2], [sigma subscript 1], and [sigma subscript 2] are means and standard deviations to be estimated, or assume "principal ignorability," (c.f. Feller et al. [2017] and Ding and Lu [2017]) that potential outcomes are independent of principal stratum conditional on covariates, Y[subscript C][up tack]S[subscript T]|x. The mixture model approach is problematic if the distributional assumptions do not hold, and may be biased even when the model is correctly specified (Feller, et al. 2016). The approach based on principal ignorability is problematic if researchers do not believe the principal ignorability assumption, or if primary interest is in the difference between effects in different principal stratum. In this talk I will introduce a semiparametric M-estimation approach based an an alternative assumption of no interactions between principal strata and covariates in an outcome regression--specifically, given a model for Y[subscript C] conditional on covariates x, with parameter vector [beta], f(x,[beta]), E[Y[subscript C]-f(x,[beta])|S[subscript T], x]=E[Y[subscript C]-f(x,[beta])|S[subscript T]]. The estimator also assumes that estimated principal scores are unbiased. Given these assumptions, I will present a set of stacked estimating equations to estimate principal effects in the one-sided non-compliance case. Unlike typical mixture modeling, this new approach makes no distributional assumptions about potential outcomes. I will present a simulation study showing that confidence intervals from the M-estimation approach achieve their nominal levels in several circumstances in which confidence intervals from (even well-specified) mixture models do not, and that the M-estimator is typically less biased than the mixture-modeling estimator, though the M-estimator tends to have higher standard errors. The root-mean-squared-error of the M-estimator improves absolutely and relative to the mixture modeling estimator as sample size increases and the relationship between x and S[subscript T] becomes stronger. I will illustrate the M-estimator in a study of feedback methods within an intelligent tutoring system. Some students requesting hints within the software were randomly assigned to receive hints in video format, and others were randomly assigned to text hints. Computer log data shows that some students assigned to video did not watch the entire video. I will estimate the effect of assignment to video hints separately for the principal stratum of students who, if assigned to video hints, would watch the entire video, and the principal stratum of students who would not watch the entire video.