Digital‐first assessments: A security framework.

Background: Digital‐first assessments leverage the affordances of technology in all elements of the assessment process: from design and development to score reporting and evaluation to create test taker‐centric assessments. Objectives: The goal of this paper is to describe the engineering, machine learning, and psychometric processes and technologies of a test security framework (part of a larger ecosystem; Burstein et al., 2021) that can be used to create systems that protect the integrity of test scores. Methods: We use the Duolingo English Test to exemplify the processes and technologies that are presented. This includes methods for actively detecting and deterring malicious behaviour (e.g., a custom desktop app). It also includes methods for passively detecting and deterring malicious behaviour (e.g., a large item bank created through automatic generation methods). We describe the certification process that each test administration undergoes, which includes both automated and human review. Additionally, we describe our quality assurance dashboard which leverages psychometric data mining techniques to monitor test quality and inform decisions about item pool maintenance. Results and Conclusions: As assessment developers transition to online delivery and to a design approach that places the test taker at the centre, it becomes increasingly important to take advantage of the tools and methodological advances in different fields (e.g., engineering, machine learning, psychometrics). These tools and methods are essential to maintaining the security of assessments so that the score reliability is sustained and the interpretations and uses of test scores remain valid. Lay Description: What is known about this topic?: As more and more testing programmes transition to test taker‐centric administrations, effective measures to prevent cheating and protect content are critical to ensure the validity and integrity of scores.Two of the most common forms of cheating in online testing are (a) having someone other than the person who has registered take the test, and (b) stealing content and providing it to others to assist them in achieving a higher score. What does this paper add?: In designing a test taker‐centric digital‐first assessment, a test security framework must inform decisions from end‐to‐end (i.e., registration, onboarding, communications regarding test taker behaviours, test preparation and practice, test administration, and post‐administrative activities including scoring).Security is safeguarded through active and passive design methods; active methods include having test takers attest that they will follow the rules governing testing and informing them that they will be videoed during testing; passive methods include a computer adaptive design that limits item exposure and test overlap rates, development of a large item pool using automated item generation, and applying artificial intelligence to review test administration videos to flag unauthorized behaviours for human review. Implications: With more educational and assessment programmes transitioning to online digital models, the paper presents a comprehensive review of security issues and identifies an integrated approach for preventing cheating and other unauthorized behaviours. [ABSTRACT FROM AUTHOR]