Are Experts Able To Predict Learner Problems during Usability Evaluations?

The usability of Instructional Multimedia (IMM) applications is vital for their success and for the satisfaction of their users, as the confusion resulting from using poorly designed programs can be particularly detrimental to learning performance. A number of approaches for expert-based evaluation of IMM have been proposed during the past few years. However, there is little evidence in the literature regarding how effective they are, especially in identifying real learner problems. This paper reports an empirical study that assesses whether experts can predict the problems experienced by students. The evidence suggests that expert evaluators, although successful in predicting usability problems, still have difficulties identifying certain types of learner problems, such as comprehension and learning support. The paper concludes that expert evaluations do not eliminate the need for tests with actual learners. Ways of improving their effectiveness are suggested. (Author/AEF)