The design, development, and validation of a multimedia-based performance assessment for credentialing confined space guards

In this article, we present a study on the design, development, and validation of a multimedia-based performance assessment (MBPA) for measuring the skills of confined space guards in Dutch vocational education. An MBPA is a computer-based assessment that incorporates multimedia to simulate tasks. It is designed to measure performance-based skills. A confined space guard (CSG) supervises operations that are carried out in a confined space (e.g., a tank or silo). In the Netherlands, individuals who want to become certified CSGs must participate in a one-day training program, and pass both a multiple-choice knowledge test and a performance-based assessment. In the first part of this article, we focus on the design and development of the MBPA, using a specific framework for design and development. In the second part of the article, we present a validation study. We use the argument-based approach to validation to validate the MBPA (Kane in Educational measurement. American Council on Education and Praeger Publishers, Westport, 2006 and J Educ Meas 50(1):1–73, 2013). More specifically, the extended argument-based approach to validation is used (Wools et al. in CADMO 18(1):63–82, 2010 and Stud Educ Eval 48:10–18, 2016). The approach suggests using multiple sources of validity evidence to build a comprehensive validity case for the proposed interpretation of assessment scores (Kane 2006, 2013) and to evaluate the strength of the validity case (Wools et al. 2010, 2016). We demonstrate that MBPA scores can be used for their intended purpose; students’ performance in the MBPA can be used as the basis for making a CSG certification decision.