Back to Search
Start Over
A Meta-Analysis of Machine Learning-Based Science Assessments: Factors Impacting Machine-Human Score Agreements
- Source :
- Journal of Science Education and Technology. 30:361-379
- Publication Year :
- 2020
- Publisher :
- Springer Science and Business Media LLC, 2020.
-
Abstract
- Machine learning (ML) has been increasingly employed in science assessment to facilitate automatic scoring efforts, although with varying degrees of success (i.e., magnitudes of machine-human score agreements [MHAs]). Little work has empirically examined the factors that impact MHA disparities in this growing field, thus constraining the improvement of machine scoring capacity and its wide applications in science education. We performed a meta-analysis of 110 studies of MHAs in order to identify the factors most strongly contributing to scoring success (i.e., high Cohen's kappa [κ]). We empirically examined six factors proposed as contributors to MHA magnitudes: algorithm, subject domain, assessment format, construct, school level, and machine supervision type. Our analyses of 110 MHAs revealed substantial heterogeneity in $$\kappa{ (mean =} \, \text{.64; range = .09-.97}$$ , taking weights into consideration). Using three-level random-effects modeling, MHA score heterogeneity was explained by the variability both within publications (i.e., the assessment task level: 82.6%) and between publications (i.e., the individual study level: 16.7%). Our results also suggest that all six factors have significant moderator effects on scoring success magnitudes. Among these, algorithm and subject domain had significantly larger effects than the other factors, suggesting that technical features and assessment external features might be primary targets for improving MHAs and ML-based science assessments.
- Subjects :
- business.industry
05 social sciences
General Engineering
Educational technology
050301 education
Moderation
Machine learning
computer.software_genre
01 natural sciences
Science education
Education
Inter-rater reliability
Cohen's kappa
Meta-analysis
0103 physical sciences
Artificial intelligence
010306 general physics
Construct (philosophy)
business
0503 education
computer
Kappa
Subjects
Details
- ISSN :
- 15731839 and 10590145
- Volume :
- 30
- Database :
- OpenAIRE
- Journal :
- Journal of Science Education and Technology
- Accession number :
- edsair.doi...........b11a5d1744663f38729307360e8faa4e