Search

More than ever in the past, researchers have access to broad, educationally relevant text data from sources such as literature databases (e.g., ERIC), an open-ended response from online courses/surveys, online discussion forums, digital essays, and social media. These advances in data availability can dramatically increase the possibilities for discovering new patterns in the data and testing new theories through processing texts with emerging analytic techniques. In our study, we extended the application of Topic Modeling (TM) to data collected from focus groups within the context of a larger study. Specifically, we compared the results of emergent qualitative coding and TM. We found a high level of agreement between TM and emergent qualitative coding, suggesting TM is a viable method for coding focus group data when augmenting and validating manual qualitative coding. We also found that TM was ineffective in capturing more nuanced information than the qualitative coding was able to identify. This can be explained by two factors: (1) the word level tokenization we used in the study; and (2) variations in the terminology teachers used to identify the different technologies. Recommendations include additional data cleaning steps researchers should take and specifications within the topic modeling code when using topic modeling to analyze focus group data.

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods—lasso regression and Bayesian Additive Regression Trees (BART)—using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modeling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites. [ABSTRACT FROM AUTHOR]

In this quasiexperimental study, we compared the effects of new developmental statistics-based courses to traditional algebra-based developmental courses. We used extant data from eight institutions and a matched sample of 2,041 students (Treatment, N = 748; Comparison, N = 1,293). All students enrolled in a treatment or comparison course in the 2017-18 school year and did not intend to declare a STEM major. After adjusting for covariates, we found evidence that treatment courses positively influenced whether students passed developmental mathematics (ß = .089, p < .001). Seventy-seven percent of treatment students passed developmental mathematics compared to 69% of comparison students who passed. Treatment students were also more likely to enroll in a subsequent credit-bearing mathematics course than comparison students (ß = .139, p < .001). Forty-nine percent of treatment students enrolled in the creditbearing course compared to 34% of comparison students who did. Once enrolled in a creditbearing course, treatment students successfully passed at a similar rate as comparison students (p = .601). Taking the treatment or comparison course did not significantly predict whether students remained continuously enrolled and/or graduated during the study (p = .711). We concluded that the treatment courses facilitated students' ability to pass developmental mathematics and enroll in credit-bearing mathematics. [ABSTRACT FROM AUTHOR]