Do Learners Know What's Good for Them? Crowdsourcing Subjective Ratings of OERs to Predict Learning Gains

We explored how learners' "subjective ratings" of open educational resources (OERs) in terms of how much they find them "helpful" can predict the actual "learning gains" associated with those resources as measured with pre- and post-tests. To this end, we developed a probabilistic model called GRAM (Gaussian Rating Aggregation Model) that combines subjective ratings from multiple learners into an aggregate quality score of each resource. Based on an experiment we conducted on Mechanical Turk (n = 304 participants with m = 17 math tutorial videos as resources), we found that aggregated subjective ratings are highly (and stat. sig.) predictive of the resources' average learning gains, with Pearson correlation of 0.78. Moreover, when predicting average learning gains of "new" learners, subjective scores were still predictive (Pearson correlation of 0.49) and attained higher prediction accuracy than a model that directly uses pre- and post-test data to estimate learning gains for each resource. These results have potential implications for large-scale learning platforms (e.g., MOOCs, Khan Academy) that assign resources (tutorials, explanations, hints, etc.) to learners based on the expected learning gains. [For the full proceedings, see ED599096.]