Search

Predictive analytics methods in education are seeing widespread use and are producing increasingly accurate predictions of students' outcomes. With the increased use of predictive analytics comes increasing concern about fairness for specific subgroups of the population. One approach that has been proposed to increase fairness is using demographic variables directly in models, as predictors. In this paper we explore issues of fairness in the use of demographic variables as predictors of long-term student outcomes, studying the arguments for and against this practice in the contexts where this literature has been published. We analyze arguments for the inclusion of demographic variables, specifically claims that this approach improves model performance and charges that excluding such variables amounts to a form of 'color-blind' racism. We also consider arguments against including demographic variables as predictors, including reduced actionability of predictions, risk of reinforcing bias, and limits of categorization. We then discuss how contextual factors of predictive models should influence case-specific decisions for the inclusion or exclusion of demographic variables and discuss the role of proxy variables. We conclude that, on balance, there are greater benefits to fairness if demographic variables are used to validate fairness rather than as predictors within models.

This study builds on prior research by leveraging natural language processing (NLP), click-stream analyses, and survey data to predict students' mathematics success and math identity (namely, self-concept, interest, and value of mathematics). Specifically, we combine NLP tools designed to measure lexical sophistication, text cohesion, and sentiment with analyses of student click-stream data within an online mathematics tutoring system. We combine these data sources to predict elementary students' success within the system as well as components of their math identity as measured though a standardized survey. Data from 147 students was examined longitudinally over a year of study. The results indicated links between math success and non-cognitive measures of math identity. Additionally, the results indicate that math identity was strongly predicted by click-stream variables and the production of more lexically sophisticated and cohesive language. In addition, significant variance in math identity was explained by affective and cognitive variables. The results indicate that NLP and click-stream data can combine to provide insights into non-cognitive constructs such as math identity.

Affect dynamics, the investigation of how student affect transitions from one state to another, is a popular area of research in adaptive learning environments. Recently, the commonly used transition metric "L" has come under critical examination when applied to data that exclude self-transitions (i.e., transitions where a student remains in the same affective state on consecutive observations); in this situation, recent work has shown that "L" potentially overestimates the significance of certain transitions. In order to deal with the unintended side effects of removing self-transitions, a solution was proposed that shifts the chance value of "L" from zero to a positive value that varies based on the number of affective states in the study. Although this treatment compensates for the aforementioned issues, it could lead to misinterpretation of the results due to the counterintuitive nonzero chance value. Motivated by these issues, in this work we study a modified version of the "L" statistic, which we refer to as "L*" , with an aim to shift the chance value back to zero when self-transitions are removed. We begin by studying the mathematical and statistical properties of "L*," and we also compare these properties to those of the "L" statistic. After analyzing the theoretical attributes of "L*," we then evaluate its performance when applied to data. In our first evaluation, we compute "L*" values on simulated sequences of affective states, where the base rates of the states vary from being uniform to highly non-uniform. The results provide evidence that the "L*" values at chance are not sensitive to unequal base rates, as the computed values from our experiments are centered closely around zero. Our second evaluation applies "L*" to actual student data generated from students working in a digital learning environment; in this case, the use of "L*" seemingly gives a more coherent picture in comparison to the values returned by the "L" statistic. Finally, along with these analyses, we also outline a recommended procedure for applying "L*" to sequences of affective states.

Educational technology (EdTech) designers need to ensure population validity as they attempt to meet the individual needs of all students. EdTech researchers often have access to larger and more diverse samples of student data to test replication across broad demographic contexts as compared to either the small-scale experiments or the larger convenience samples often seen in experimental psychology studies of learning. However, the source of typical EdTech data (i.e., online learning systems) and concerns related to student privacy often limit the opportunity to collect demographic variables from individual students--the sample is diverse, but the researcher does not know how that diversity is realized in individual learners. In order to ensure equitable student outcomes, the EdTech community should make greater efforts to develop new methods for addressing this shortcoming. Recent work has sought to address this issue by investigating publicly-available, school-level differences in demographics, which can be useful when individual-level variation may be difficult or impossible to acquire data for. In this study, we use this approach to better understand the role of social factors in students' self-regulated learning and motivation-related behaviors, behaviors whose effectiveness appears to be highly variable between groups. We demonstrate that school-level demographics can be significantly associated with the relationships between students' help-seeking behavior, motivation, and outcomes (math performance and math self-concept). We do so in the context of reasoning mind, an intelligent tutoring system for elementary mathematics. By studying the conditions under which these relationships vary across different demographic contexts, we challenge implicit assumptions of generalizability and provide an evidence-based commentary on future research practices in the EdTech community surrounding how we consider diversity in our field's investigations.

Phishing is an increasing threat that causes billions in losses and damage to productivity, trade secrets, and reputations each year. This work explores how security gamification techniques can improve phishing reporting. We contextualized the cognitive evaluation theory (CET) as a kernel theory and constructed a prototype phishing reporting system. With three experiments in a simulated work setting, we tested gamification elements of validation, attribution, incentives, and public presentation for improvements in experiential (e.g., motivation) and instrumental outcomes (e.g., hits and false positives) in phishing reporting. Our findings suggest public attribution with rewards and punishments best balance the competing necessities of accuracy with widespread reporting. Furthermore, our results demonstrate the unique benefits of security gamification to phishing reporting over and above other phishing mitigation techniques (e.g., training and warnings). However, we also noted that unintended consequences in false alarms might arise from shifts in motivation resulting from public display of incentives. These findings suggest that carefully calibrated external incentives (rather than intrinsic rewards) are most likely to improve the ancillary task of phishing reporting. [ABSTRACT FROM AUTHOR]

Various areas of educational research are interested in the transitions between different states--or events--in sequential data, with the goal of understanding the significance of these transitions; one notable example is affect dynamics, which aims to identify important transitions between affective states. Unfortunately, several works have uncovered issues with the metrics and procedures commonly used to analyze these transitions. As such, our goal in this work is to address these issues by outlining an alternative procedure that is based on the use of marginal models. We begin by looking at the specific mechanisms responsible for a recently discovered statistical bias with several metrics used in sequential data analysis. After giving a theoretical explanation for the issue, we show that the marginal model procedure appears to adjust for this bias. Next, a related problem is that the common practice of removing transitions to repeated states has been shown to have unintended side-effects--to account for this issue, we develop a method for extending the marginal model procedure to this specific type of analysis. Finally, in a recent study evaluating the problem of multiple comparisons and sequential data analysis, the Benjamini-Hochberg (BH) procedure, a commonly used approach to control for false discoveries, did not perform as expected. By applying a technique from the biostatistics and epidemiology literature, we show that the performance of the BH procedure, when used with the marginal model method, can be brought back to its expected level. In all of our analyses, we evaluate the proposed method by both running simulations and using actual student data. The results indicate that the marginal model procedure seemingly compensates for the problems observed with other transition metrics, thus resulting in more accurate estimates of the importance of transitions between states. [ABSTRACT FROM AUTHOR]

Affect dynamics, the study of how affect develops and manifests over time, has become a popular area of research in affective computing for learning. In this article, we first provide a detailed analysis of prior affect dynamics studies, elaborating both their findings and the contextual and methodological differences between these studies. We then address methodological concerns that have not been previously addressed in the literature, discussing how various edge cases should be treated. Next, we present mathematical evidence that several past studies applied the transition metric (L) incorrectly - leading to invalid conclusions of statistical significance - and provide a corrected method. Using this corrected analysis method, we reanalyze ten past affect datasets collected in diverse contexts and synthesize the results, determining that the findings do not match the most popular theoretical model of affect dynamics. Instead, our results highlight the need to focus on cultural factors in future affect dynamics research. [ABSTRACT FROM AUTHOR]

Predictive analytics methods in education are seeing widespread use and are producing increasingly accurate predictions of students' outcomes. With the increased use of predictive analytics comes increasing concern about fairness for specific subgroups of the population. One approach that has been proposed to increase fairness is using demographic variables directly in models, as predictors. In this paper we explore issues of fairness in the use of demographic variables as predictors of long-term student outcomes, studying the arguments for and against this practice in the contexts where this literature has been published. We analyze arguments for the inclusion of demographic variables, specifically claims that this approach improves model performance and charges that excluding such variables amounts to a form of 'color-blind' racism. We also consider arguments against including demographic variables as predictors, including reduced actionability of predictions, risk of reinforcing bias, and limits of categorization. We then discuss how contextual factors of predictive models should influence case-specific decisions for the inclusion or exclusion of demographic variables and discuss the role of proxy variables. We conclude that, on balance, there are greater benefits to fairness if demographic variables are used to validate fairness rather than as predictors within models. [ABSTRACT FROM AUTHOR]