Search

Background: Digital‐first assessments leverage the affordances of technology in all elements of the assessment process: from design and development to score reporting and evaluation to create test taker‐centric assessments. Objectives: The goal of this paper is to describe the engineering, machine learning, and psychometric processes and technologies of a test security framework (part of a larger ecosystem; Burstein et al., 2021) that can be used to create systems that protect the integrity of test scores. Methods: We use the Duolingo English Test to exemplify the processes and technologies that are presented. This includes methods for actively detecting and deterring malicious behaviour (e.g., a custom desktop app). It also includes methods for passively detecting and deterring malicious behaviour (e.g., a large item bank created through automatic generation methods). We describe the certification process that each test administration undergoes, which includes both automated and human review. Additionally, we describe our quality assurance dashboard which leverages psychometric data mining techniques to monitor test quality and inform decisions about item pool maintenance. Results and Conclusions: As assessment developers transition to online delivery and to a design approach that places the test taker at the centre, it becomes increasingly important to take advantage of the tools and methodological advances in different fields (e.g., engineering, machine learning, psychometrics). These tools and methods are essential to maintaining the security of assessments so that the score reliability is sustained and the interpretations and uses of test scores remain valid. Lay Description: What is known about this topic?: As more and more testing programmes transition to test taker‐centric administrations, effective measures to prevent cheating and protect content are critical to ensure the validity and integrity of scores.Two of the most common forms of cheating in online testing are (a) having someone other than the person who has registered take the test, and (b) stealing content and providing it to others to assist them in achieving a higher score. What does this paper add?: In designing a test taker‐centric digital‐first assessment, a test security framework must inform decisions from end‐to‐end (i.e., registration, onboarding, communications regarding test taker behaviours, test preparation and practice, test administration, and post‐administrative activities including scoring).Security is safeguarded through active and passive design methods; active methods include having test takers attest that they will follow the rules governing testing and informing them that they will be videoed during testing; passive methods include a computer adaptive design that limits item exposure and test overlap rates, development of a large item pool using automated item generation, and applying artificial intelligence to review test administration videos to flag unauthorized behaviours for human review. Implications: With more educational and assessment programmes transitioning to online digital models, the paper presents a comprehensive review of security issues and identifies an integrated approach for preventing cheating and other unauthorized behaviours. [ABSTRACT FROM AUTHOR]

To date, extensive work has been devoted to incorporating computational thinking in K‐12 education. Recognizing students' computational thinking stages in game‐based learning environments is essential to capture unproductive learning and provide appropriate scaffolding. However, few reliable and valid computational thinking measures have been developed, especially in games, where computational knowledge acquisition and computational skill construction are implicit. This study introduced an innovative approach to explore students' implicit computational thinking through various explicit factors in game‐based learning, with a specific focus on Zoombinis, a logical puzzle‐based game designed to enhance students' computational thinking skills. Our results showed that factors such as duration, accuracy, number of actions and puzzle difficulty were significantly related to students' computational thinking stages, while gender and grade level were not. Besides, findings indicated gameplay performance has the potential to reveal students' computational thinking stages and skills. Effective performance (shorter duration, fewer actions and higher accuracy) indicated practical problem‐solving strategies and systematic computational thinking stages (eg, Algorithm Design). This work helps simplify the process of implicit computational thinking assessment in games by observing the explicit factors and gameplay performance. These insights will serve to enhance the application of gamification in K‐12 computational thinking education, offering a more efficient method to understanding and fostering students' computational thinking skills. Practitioner notesWhat is already known about this topic Game‐based learning is a pedagogical framework for developing computational thinking in K‐12 education.Computational thinking assessment in games faces difficulties because students' knowledge acquisition and skill construction are implicit.Qualitative methods have widely been used to measure students' computational thinking skills in game‐based learning environments.What this paper adds Categorize students' computational thinking experiences into distinct stages and analyse recurrent patterns employed at each stage through sequential analysis. This approach serves as inspiration for advancing the assessment of stage‐based implicit learning with machine learning methods.Gameplay performance and puzzle difficulty significantly relate to students' computational thinking skills. Researchers and instructors can assess students' implicit computational thinking by observing their real‐time gameplay actions.High‐performing students can develop practical problem‐solving strategies and exhibit systematic computational thinking stages, while low‐performing students may need appropriate interventions to enhance their computational thinking practices.Implications for practice and/or policy Introduce a practical method with the potential for generalization across various game‐based learning to better understand learning processes by analysing significant correlations between certain gameplay variables and implicit learning stages.Allow unproductive learning detection and timely intervention by modelling the reflection of gameplay variables in students' implicit learning processes, helping improve knowledge mastery and skill construction in games.Further investigations on the causal relationship between gameplay performance and implicit learning skills, with careful consideration of more performance factors, are expected. [ABSTRACT FROM AUTHOR]

In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction. Existing models designed to produce interpretable traces of their decision-making process typically require these traces to be supervised at training time. In this paper, we present a novel neural modular approach that performs compositional reasoning by automatically inducing a desired subtask decomposition without relying on strong supervision. Our model allows linking different reasoning tasks through shared modules that handle common routines across tasks. Experiments show that the model is more interpretable to human evaluators compared to other state-of-the-art models: users can better understand the model's underlying reasoning procedure and predict when it will succeed or fail based on observing its intermediate outputs. [ABSTRACT FROM AUTHOR]

Due to advancements in data collection, storage, and processing techniques, machine learning has become a thriving and dominant paradigm. However, one of its main shortcomings is that the classical machine learning paradigm acts in isolation without utilizing the knowledge gained through learning from related tasks in the past. To circumvent this, the concept of Lifelong Machine Learning (LML) has been proposed, with the goal of mimicking how humans learn and acquire cognition. Human learning research has revealed that the brain connects previously learned information while learning new information from a single or small number of examples. Similarly, an LML system continually learns by storing and applying acquired information. Starting with an analysis of how the human brain learns, this paper shows that the LML framework shares a functional structure with the brain when it comes to solving new problems using previously learned information. It also provides a description of the LML framework, emphasizing its similarities to human brain learning. It also provides citation graph generation and scientometric analysis algorithms for the LML literatures, including information about the datasets and evaluation metrics that have been used in the empirical evaluation of LML systems. Finally, it presents outstanding issues and possible future research directions in the field of LML. [ABSTRACT FROM AUTHOR]

Massively open online courses (MOOCs) offer learners opportunities for self‐improvement and knowledge development. Linking cognitive processes and learning outcomes is helpful in supporting students learning with the help of MOOCs. Based on the cognitive presence of the Community of Inquiry framework, this study quantified the effect of cognitive presence on students' learning performance by analysing more than 400,000 posts of 13 MOOCs. First, the study built a highly predictive classification model using a machine learning algorithm to automatically identify the phases of cognitive presence in MOOC forum posts. Subsequently, multilevel modelling was used to analyse the influence of learners' cognitive presence on their learning performance. The results showed that different phases of cognitive presence influenced students' learning differently. The findings help us understand the phases and depth of students' cognitive presence and can be used as the basis for MOOCs to automatically provide appropriate cognitive feedback and support to students. Practitioner notesWhat is already known about this topic Massively open online courses (MOOCs) offer learners opportunities for self‐improvement and knowledge development.Linking cognitive processes and learning outcomes is helpful in supporting student learning in MOOCs.Little is known so far about the role of cognitive presence in influencing students' learning performance.What this paper adds This study built a highly predictive classification model using a machine learning algorithm to automatically identify the phases of cognitive presence in MOOC forum posts.Multilevel modelling was used to quantify the influence of learners' cognitive presence on their learning performance.The results showed that different phases of cognitive presence influenced students' learning differently.Implications for practice and/or policy This study built an effective, high‐performance machine learning model with easy feature extraction, which could be used to automatically identify students' cognitive processes in MOOC forums and beyond.The combination of cognitive presence and learning performance helps us understand the phases and depth of students' cognitive presence.The findings can be used as the basis for MOOCs to automatically provide appropriate cognitive feedback and support to students. [ABSTRACT FROM AUTHOR]

With the widespread use of learning analytics (LA), ethical concerns about fairness have been raised. Research shows that LA models may be biased against students of certain demographic subgroups. Although fairness has gained significant attention in the broader machine learning (ML) community in the last decade, it is only recently that attention has been paid to fairness in LA. Furthermore, the decision on which unfairness mitigation algorithm or metric to use in a particular context remains largely unknown. On this premise, we performed a comparative evaluation of some selected unfairness mitigation algorithms regarded in the fair ML community to have shown promising results. Using a 3‐year program dropout data from an Australian university, we comparatively evaluated how the unfairness mitigation algorithms contribute to ethical LA by testing for some hypotheses across fairness and performance metrics. Interestingly, our results show how data bias does not always necessarily result in predictive bias. Perhaps not surprisingly, our test for fairness‐utility tradeoff shows how ensuring fairness does not always lead to drop in utility. Indeed, our results show that ensuring fairness might lead to enhanced utility under specific circumstances. Our findings may to some extent, guide fairness algorithm and metric selection for a given context. Practitioner notesWhat is already known about this topic LA is increasingly being used to leverage actionable insights about students and drive student success.LA models have been found to make discriminatory decisions against certain student demographic subgroups—therefore, raising ethical concerns.Fairness in education is nascent. Only a few works have examined fairness in LA and consequently followed up with ensuring fair LA models.What this paper adds A juxtaposition of unfairness mitigation algorithms across the entire LA pipeline showing how they compare and how each of them contributes to fair LA.Ensuring ethical LA does not always lead to a dip in performance. Sometimes, it actually improves performance as well.Fairness in LA has only focused on some form of outcome equality, however equality of outcome may be possible only when the playing field is levelled.Implications for practice and/or policy Based on desired notion of fairness and which segment of the LA pipeline is accessible, a fairness‐minded decision maker may be able to decide which algorithm to use in order to achieve their ethical goals.LA practitioners can carefully aim for more ethical LA models without trading significant utility by selecting algorithms that find the right balance between the two objectives.Fairness enhancing technologies should be cautiously used as guides—not final decision makers. Human domain experts must be kept in the loop to handle the dynamics of transcending fair LA beyond equality to equitable LA. [ABSTRACT FROM AUTHOR]

The experimental study of learning and plasticity has always been driven by an implicit question: how can physiological changes be adaptive and improve performance? For example, in Hebbian plasticity only synapses from presynaptic neurons that were active are changed, avoiding useless changes. Similarly, in dopamine‐gated learning synapse changes depend on reward or lack thereof and do not change when everything is predictable. Within machine learning we can make the question of which changes are adaptive concrete: performance improves when changes correlate with the gradient of an objective function quantifying performance. This result is general for any system that improves through small changes. As such, physiology has always implicitly been seeking mechanisms that allow the brain to approximate gradients. Coming from this perspective we review the existing literature on plasticity‐related mechanisms, and we show how these mechanisms relate to gradient estimation. We argue that gradients are a unifying idea to explain the many facets of neuronal plasticity. [ABSTRACT FROM AUTHOR]

This article proposed an autoencoder‐decoder architecture with convolutional long‐short‐term memory (ConvLSTM) cell for the purpose of learning topology optimization iterations. The overall topology optimization process is treated as time‐series data, with each iteration as a single step. The first few steps are fed into the encoder to generate encoder embedding, which is fed into the decoder. The decoder uses the encoder embedding as input and generates the result at each future step until the end of iteration. To train the proposed neural network, a large dataset is generated by a conventional topology optimization method, that is, solid isotropic material with penalization for intermediate densities, with randomly picked boundary conditions, load conditions, and volume constraints. Unlike other deep learning models introduced before, the proposed method can learn each topology optimization step iteratively and present the full optimization path. Furthermore, the proposed method can be extended to give solutions to unseen boundary and load conditions with a significant reduction in computation cost in a little sacrifice on the performance of the optimum design. [ABSTRACT FROM AUTHOR]

The insights that I The Alignment Problem i offers into how AI/ML systems are being designed to tackle these challenging questions may prove useful in thinking about how to improve human risk analyses. An architecture reminiscent of the dual-process "thinking, fast and slow" in people has proved successful in creating AI/ML systems such as DeepMind's AlphaGo Zero that taught itself to play world champion-level Go in 3 days without any examples of human games or instructions, guidance, or advice from human experts. Henry Kissinger et al.'s 2021 book I The Age of AI and Our Human Future i discusses at a less technical level implications of current and emerging AI/ML technologies (including the revolutionary GPT-3 language model) and their implications for AI governance and the future co-evolution of AI and humanity. Modern artificial intelligence (AI) and machine learning (ML) wrestle with many of the same challenges in guiding the decisions of robots and autonomous agents and teams of such agents operating under uncertainty or under novel conditions. [Extracted from the article]

Path planning can guarantee that agents reach their goals without colliding with obstacles and other agents in an optimal way and it is a very important component in the research of crowd simulation. In this article, we propose a novel path planning approach for multiple agents which combines soft actor critic (SAC) algorithm and curriculum learning to solve the problems of single policy, slow convergence of the policy in an unknown environment with sparse rewards. The path planning task is set as lessons from easy to difficult, and the neural network of the SAC algorithm is arranged to learn in sequence, and finally the neural network can be fully competent for the path planning task. We also stack the state information to address the problems caused by limited observation for policy learning, and design a comprehensive reward function to make agents reach their goals successfully and avoid collisions with static obstacles and other agents. The experimental results demonstrate that our approach can plan smooth and natural paths for multiple agents, and furthermore, our model has a certain generalization ability and a better adaptability to the changes in a dynamic environment. [ABSTRACT FROM AUTHOR]

The fields of machine learning (ML) and cognitive science have developed complementary approaches to computationally modeling human behavior. ML's primary concern is maximizing prediction accuracy; cognitive science's primary concern is explaining the underlying mechanisms. Cross‐talk between these disciplines is limited, likely because the tasks and goals usually differ. The domain of e‐learning and knowledge acquisition constitutes a fruitful intersection for the two fields' methodologies to be integrated because accurately tracking learning and forgetting over time and predicting future performance based on learning histories are central to developing effective, personalized learning tools. Here, we show how a state‐of‐the‐art ML model can be enhanced by incorporating insights from a cognitive model of human memory. This was done by exploiting the predictive performance equation's (PPE) narrow but highly specialized domain knowledge with regard to the temporal dynamics of learning and forgetting. Specifically, the PPE was used to engineer timing‐related input features for a gradient‐boosted decision trees (GBDT) model. The resulting PPE‐enhanced GBDT outperformed the default GBDT, especially under conditions in which limited data were available for training. Results suggest that integrating cognitive and ML models could be particularly productive if the available data are too high‐dimensional to be explained by a cognitive model but not sufficiently large to effectively train a modern ML algorithm. Here, the cognitive model's insights pertaining to only one aspect of the data were enough to jump‐start the ML model's ability to make predictions—a finding that holds promise for future explorations. Cognitive models and machine learning techniques were combined to improve predictions in naturalistic vocabulary learning data. Key predictive features for the ML model were distilled from the cognitive model to better capture the underlying temporal dynamics. [ABSTRACT FROM AUTHOR]