Search

This paper describes a multi-institutional study that used a repeated single-criterion card sort to investigate graduating computer science students' knowledge of programming concepts. The study seeks to improve computer science instruction by gaining insight into how graduating students retain and assimilate introductory programming knowledge into their broader understanding of the discipline. A total of 291 card sorts was elicited from 65 undergraduate students in their final year of study at eight colleges and universities throughout the USA. To fully exploit the rich qualitative and quantitative aspects of the card sort data, an integrative analysis process was used that combined content analysis with two measures, normalized minimum spanning tree and edit distance, both developed specifically to analyze card sort data.

Card sorts can be used to study the way human subjects acquire and organize conceptual knowledge. Analyses of card sorts often involve subjective examination of criteria or category names or using clustering techniques, neither of which lend themselves well to direct statistical analysis. This paper defines NMST, a quantitative measure of knowledge discrimination based on repeated, single-criterion card sorts that is independent of criteria or category names and that is particularly amenable to statistical analysis. The NMST measure is illustrated by applying it to a particular data set collected in a large multinational card sort study of subjects, where the knowledge area comes from a first-year programming course. Applied to this data set, the NMST measure is shown to distinguish, with statistical significance, between a set of subjects with only an introduction to programming and a set of subjects who have completed the equivalent of a bachelor's degree or higher in a computing-related discipline.

This paper presents a case study of the use of a repeated single-criterion card sort with an unusually large, diverse participant group. The study, whose goal was to elicit novice programmers' knowledge of programming concepts, involved over 20 researchers from four continents and 276 participants drawn from 20 different institutions. In this paper we present the design of the study and the unexpected result that there were few discernible systematic differences in the population. The study was one of the activities of the National Science Foundation funded Bootstrapping Research in Computer Science Education project (2003).

Card sorts are a knowledge elicitation technique in which participants are given a collection of items and are asked to partition them into groups based on their own criteria. Information about the participant's knowledge structure is inferred from the groups formed and the names used to describe the groups through various methods ranging from simple quantitative statistical measures (e.g. co-occurrence frequencies) to complex qualitative methods (e.g. content analysis on the group names). This paper introduces a new technique for analyzing card sort data that uses quantitative measures to discover rich qualitative results. This method is based upon a distance metric between sorts that allows one to measure the similarity of groupings and then look for clusters of closely related sorts across individuals. By using software for computing these clusters, it is possible to identify common concepts across individuals, despite the use of different terminology.

Among the knowledge elicitation techniques card sorting is notable for its simplicity of use, its focus on subjects' terminology (rather than that of external experts) and its ability to elicit semi-tacit knowledge. Card sorting involves categorizing a set of pictures, objects or labelled cards into distinct groups using a single criterion. This paper focuses on the challenges associated with analyzing the data that result from card sorts, especially when large data sets are generated. Traditional semantic analysis methods that require direct researcher interpretation of elicited linguistic terms are distinguished from methods that are purely syntactic, and hence can be automated. Each paper within this special issue is summarized and its contribution to card sorting in general, and data analysis in particular, is highlighted. The set of novel computational techniques presented in several of the papers in this issue is examined. The paper concludes by noting that even large-scale data sets can be meaningfully analysed by combining well-known interpretative methods with the new computational approaches presented within this special issue.