Extending the Rule Space Model to a Semantically-Rich Domain: Diagnostic Assessment in Architecture.

This paper presents a technique for applying the Rule Space Model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically rich domain. Responses of 122 architects to 22 architecture test items developed to assess a range of architectural knowledge were analyzed using Rule Space. Verbal protocol analysis guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item (labeled "understand"), forming goals and performing actions based on those goals ("solve"), and determining whether goals have been attempted and satisfied ("check"). Item attributes derived from these processes formed the basis for diagnosis. Successful diagnostic classifications were obtained for approximately 65 percent, 90 percent, and 40 percent of examinees based, respectively, on attributes associated with the "understand,""solve," and "check" processes of the problem-solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels. Nine tables and three figures present study data. (Contains 23 references.) (Author/SLD)