Back to Search
Start Over
Web-Based and Mixed-Mode Cognitive Large-Scale Assessments in Higher Education: An Evaluation of Selection Bias, Measurement Bias, and Prediction Bias
- Source :
- Behavior Research Methods
- Publication Year :
- 2020
-
Abstract
- Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paper-based, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on study-related success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education. Electronic supplementary material The online version of this article (10.3758/s13428-020-01480-7) contains supplementary material, which is available to authorized users.
- Subjects :
- Computer science
media_common.quotation_subject
Experimental and Cognitive Psychology
Selection effect
Article
Alternative assessment
Cognition
Bias
0504 sociology
Arts and Humanities (miscellaneous)
Statistics
Web-based testing
Developmental and Educational Psychology
ddc:330
Humans
Web application
Measurement invariance
Higher education
Mode effect
Selection Bias
General Psychology
media_common
Response rate (survey)
Selection bias
Internet
business.industry
Data Collection
Computerized testing
05 social sciences
050401 social sciences methods
050301 education
Test (assessment)
Psychology (miscellaneous)
business
0503 education
Subjects
Details
- Language :
- English
- Database :
- OpenAIRE
- Journal :
- Behavior Research Methods
- Accession number :
- edsair.doi.dedup.....f8f9718df3661eb15e12d425613b00ab
- Full Text :
- https://doi.org/10.13140/rg.2.2.32233.31847