Search

Developing good psychological measures benefits from the input of content experts. For many constructs or domains, however, who constitutes an ‘expert’ might be ill-defined. Novices—such as students, customers, or co-workers—may possess the same knowledge as experts. Moreover, as convenience samples, novices are more readily available and less costly. Card sorting is a technique frequently used in human factors to elicit expert knowledge. This study compared novice and expert performance on a card sort task under two conditions, an open sort and a closed sort. Because the closed sort offered a category structure for sorting, it was predicted and found that novices in the closed sort tended to match expert sorting results more closely than in the open sort. The structure also made novice solutions better approximate the expert open sort. This suggests that novices can be useful in the follow-up stage of measure development, but not in the initial stage.

Teaching philosophy statements clarify why we do what we do in the classroom, and the process of drafting a philosophy offers an opportunity for developmental reflection. Personal teaching philosophies can be grounded in the shared foundation of historical educational philosophies. The authors offer here for facilitators a reflective card-sort exercise that helps surface the philosophical roots of personal teaching philosophies and helps teachers create or renew a teaching philosophy statement. They explain the exercise activities and typical outcomes based on their experiences facilitating the exercise. The authors conclude with insights and considerations for facilitators.

Medicaid home and community-based services (HCBS) care plans should be person-centered, yet there is little research on how to ascertain this information in practice. The purpose of this study was to investigate the feasibility of a home care aide (HCA)–led health interview with clients during usual HCBS. We provided interview training, and HCAs ( n = 21) conducted five interviews with one client each using a card sort methodology to elicit client care preferences. HCAs audio-recorded interviews and photographed card sorts for analysis. We used a mixed-methods approach of semistructured interviews and focus groups with clients and HCAs to evaluate the health interviewing experience and client surveys of Your Health Orientation, Willingness to Communicate, and PROMIS (Patient-Reported Outcomes Measurement Information System) global health and HCA surveys of the Active Empathetic Listening Scale. We used t tests to investigate changes in survey outcomes pre and post interviews. Results show HCAs can conduct health interviews, and doing so contributes new knowledge on client preferences for care. Clients desire HCAs who provide empathy, compassion, and motivation, and HCAs felt interviewing clients helped them to better understand their care recipient’s needs.

Objective: To explore cognitive strategies clinicians apply while performing a medication reconciliation task, handling incomplete and conflicting information. Background: Medication reconciliation is a method clinicians apply to find and resolve inconsistencies in patients’ medications and medical conditions lists. The cognitive strategies clinicians use during reconciliation are unclear. Controlled lab experiments can explore how clinicians make sense of uncertain, missing, or conflicting information and therefore support the development of a human performance model. We hypothesize that clinicians apply varied cognitive strategies to handle this task and that profession and experience affect these strategies. Method: 130 clinicians participated in a tablet-based experiment conducted in a large American teaching hospital. They were asked to simulate medication reconciliation using a card sorting task (CaST) to organize medication and medical condition lists of a specific clinical case. Later on, they were presented with new information and were asked to add it to their arrangements. We quantitatively and qualitatively analyzed the ways clinicians arranged patient information. Results: Four distinct cognitive strategies were identified (“Conditions first”: n = 76 clinicians, “Medications first”: n = 7, “Crossover”: n = 17, and “Alternating”: n = 10). The strategy clinicians applied was affected by their experience ( p = .02) but not by their profession. At the appearance of new information, clinicians moved medication cards more frequently (75.2 movements vs. 49.6 movements, p < .001), suggesting that they match medications to medical conditions. Conclusion: Clinicians apply various cognitive strategies while reconciling medications and medical conditions. Application: Clinical information systems should support multiple cognitive strategies, allowing flexibility in organizing information.

Background: Clinical practice guidelines (CPGs) in medicine are recommendations supported by systematic review of evidence to facilitate optimal patient outcomes. Primary care practices are expected to implement more than 200 CPGs, overwhelming many practices. This qualitative study elucidated the perspectives and priorities of family medicine physicians and office managers in grouping CPGs to facilitate implementation. Methods: A qualitative study was performed using individual, semistructured interviews. During the interviews the participants completed an open card-sort exercise grouping 20 CPGs. Purposive sampling was used to identify family medicine physicians and office managers practicing in medically underserved zip codes listed in the local medical society directory. Seven physicians and 6 office managers were interviewed. The interviews were transcribed and analyzed using thematic analysis and compared with the card-sort results. Results: Thematic content analysis identified priorities and perspectives of office managers and physicians when grouping multiple CPGs for implementation: delegation, personalization, triggers, and change management. The card sort exercise revealed grouping by standardized preventive care visit, standardized rooming and discharge processes, and chronic illness. Chronic illness-based groupings and personalization of guidelines were recognized as presenting barriers to delegation of CPGs to the care team. Development of standardized preventive exams, standard rooming and discharge processes and chronic disease management were identified as promoting CPG adherence through team-based care. Standardized workflows provided opportunities for task delegation through predicable roles. Medicalization of CPG implementation relied heavily on the physician alone to remember to adhere to CPGs and inhibited task sharing by not giving office staff clear disease-based protocols to follow. Conclusions: This study identified priorities and perspectives of office managers and physicians when grouping multiple CPGs for concomitant implementation: delegation, personalization, triggers, and change management. Successful implementation was perceived to be associated with standardized preventive exams, standard rooming and discharge processes, and chronic disease management.

The study of visual content using qualitative methods currently involves much scholarly work in the field of digital humanities. Documentary films are considered among those objects defined as visual content and are therefore not the focus of most attention within Film Studies. One of the best-known researchers in this area, Bill Nichols, proposed a theoretical classification of six documentary voices that is widely recognized in academic circles. However, the classification is based on induction and intuition which, until now, have not been verified in accordance with the evidence. This study proposes a mixed qualitative and quantitative methodology that allows the authors to surpass Nichols’ model and generate a new classification based on empirical criteria. To that end, they study the opinions of users of and experts on Nichols’ theory through the card sorting technique borrowed from cognitive science that is based on multivariate statistics. In this way, they generate a hierarchy of formal cinematographic elements grouped into four documentary voices and Nicholas’ classification proposal is partly replaced as two of the voices in his model disappear but the four others are confirmed. In addition, this new model allows the authors to determine which formal elements are assigned to each one of the confirmed documentary modes and with what relative weights, thereby opening up the possibility of generating a predictive model. The article therefore demonstrates that cluster analysis and other techniques borrowed from multivariate statistics can be used to empirically and quantitatively verify theories in film studies as well as in other areas such as poetics. This suggests that it is possible to verify empirically and use quantitative models for the study of visual content and complement other methods based on induction and intuition.

Card sorting has been used in qualitative studies in various fields to better understand how individuals organize information and make choices based on it. As part of a mixed-methods study of why Black engineering students initially chose their major and why they subsequently decided to persist in or switch out of it, we developed a card-sort activity and used it in 79 semi-structured student interviews. Besides generating data relevant to the mission of our study, the activity shifted the students' focus to their own experiences and away from the interviewers, who were not matched with them on race or age and with whom there was a power differential (interviewers with doctorates talking with undergraduates). The article contains a brief overview of the broader study, a description of steps taken to promote authentic storytelling by the interviewees, an outline of the development and application of the card-sort activity, representative findings, and recommendations for other researchers contemplating using a similar technique. We believe our experience in using the card-sort technique and our subsequent mixed-method analysis of the resulting data will benefit qualitative researchers seeking authentic stories.

Training hour reductions for resident physicians have resulted in fewer opportunities for novices to manage critically ill patients. Our goals were (a) to understand differences in how novices and experts notice and interpret clinical cues using sepsis as an exemplar and (b) to develop simulations that replicate clinical cues to facilitate acquisition of expertise. Researchers conducted 14 critical decision method (CDM) interviews with four novices (interns), four senior trainees (senior residents), and six faculty (expert) physicians. We interviewed across a spectrum of experience to better assess for experience-based differences in sepsis recognition. Investigators analyzed transcribed interviews using a card sort technique. Experts described more hypothesis testing and violated expectations than novices. Expert–novice differences in sepsis recognition informed the design and future piloting of training scenarios that require novices to seek, interpret, and act on relevant cues.

During the content validation phase of the development of a hybrid heuristic/survey-based usability assessment technique, the authors of this paper conducted a card sort with Subject Matter Experts (SMEs) in Human Factors Psychology. The information provided would inform the initial usability dimensions (e.g., Help, Learnability, Graphic Design & Aesthetics) used in the resulting technique. While Hierarchical Cluster Analysis (HCA) is the standard approach to data analysis with card sorts, several factors (e.g., size of data set, application of system, nature of construct measured) led the research team to consider alternatives. As a result, data derived from this card sort were analyzed in two ways: 1) using Confirmatory Factor Analysis (CFA) and, 2) Using (HCA). The use of the two methods provided an opportunity for researchers to compare their quality and utility in informing dimensions used during the construct validation phase. While HCA was more detailed and provided a better visual description as to how individual items were related to others and broad dimensions, the results were far from parsimonious, making it difficult to determine the meaning of broad dimensions and exactly where those dimensions deviated from others. In contrast, CFA resulted in clear and distinct dimensions with individual items loading on no more than two dimensions (e.g., one primary and one secondary). With that said, the CFA resulted in the rejection of more items as many did not load on the extant dimensions. The general conclusions of this psychometric methodological comparison, for the validation of a Usability evaluation method, is that while HCA is an excellent method for understanding the intercorrelations and intricacies of a construct (i.e., Usability), it may not be the best suited approach for developing a usability method intended to provide systematic feedback to system developers. If the goal is the development of a usability assessment technique, researchers and practitioners alike can use CFA to analyze their card sort data and inform the construct validation phase of their measure.

Emerging adulthood defines transition to employment, higher education, and domestic life. This study describes the development of an assessment of self-reported participation in a range of age-appropriate activities. Item selection was established from literature review, feedback from youth and professionals, the former Adolescent Activity Card Sort (AACS), and the original Activity Card Sort (ACS). Iterative item selection occurred with three separate samples of emerging adults and six professionals. Test–retest reliability was evaluated. The Adolescent and Young Adult Activity Card Sort (AYA-ACS) consists of chores (11 items), leisure (13), social (10), health and fitness (9), work (10), education (8), and parenting (9). Test–retest reliability showed significant moderate to substantial Kappa agreement (.48-.85) for all domains except parenting (κ = .15). This preliminary study describes the development of the AYA-ACS to be used with individuals who encounter challenges when transitioning to young adulthood.

The U.S. Army is focused on improving leader decision making, yet little is known as to how military officers develop optimal decision making. Two key components of optimal decision making are reinforcement learning and cognitive flexibility. Thirty-four military officers completed military versions of two standard cognitive assessments, the Iowa Gambling Test and the Wisconsin Card Sorting Task, while their eye movements were tracked. Results indicated that the military versions of these tasks successfully provided objective assessments of reinforcement learning and cognitive flexibility. Preliminary analyses of eye movements provide insights into the subjects’ decision strategies. Training implications of the results are discussed.

Behavioral measures of risky decision making are frequently used by researchers and clinicians; however, most of these measures are strongly associated with personality characteristics and state mood. The present study sought to examine personality, mood, and executive function predictors of performance on a newer measure of decision making, the Columbia Card Task (CCT). Participants were 489 undergraduate students who completed either the hot or cold version of the CCT as well as measures of state mood, impulsive sensation seeking, behavioral inhibition and activation systems, and executive functions (Wisconsin Card Sort Task; Digit Span). Results indicated that performance on the CCT-cold was predicted by Wisconsin Card Sort Task errors, and Digit Span predicted the CCT-hot. In addition, significant correlations were found between the CCT information use variables and the predictor variables. Implications for the utility of the CCT as a clinical instrument and its relationship with other measures of decision making are discussed.

The purpose of this paper is to provide some Australian data on the use of the Career Interest Card Sort. Results from 196 adult clients who participated in a vocational evaluation are analysed. The assessment provided a hierarchy of career interest results that reflected demographic features such as gender and education. The vocational interest results were consistent with the pre-accident occupational background of clients. The findings provided support for a direct approach to vocational interest assessment that may find some application in practice.

The serious games community is moving toward research focusing on direct comparisons between learning outcomes of serious games and those of more traditional training methods. Such comparisons are difficult, however, due to the lack of a consistent taxonomy of game attributes for serious games. Without a clear understanding of what truly constitutes a game, scientific inquiry will continue to reveal inconsistent findings, making it hard to provide practitioners with guidance as to the most important attribute(s) for desired training outcomes. This article presents a game attribute taxonomy derived from a comprehensive literature review and subsequent card sorts performed by subject matter experts (SMEs). The categories of serious game attributes that emerged represent the shared mental models of game SMEs and serve to provide a comprehensive collection of game attributes. In order to guide future serious games research, the existing literature base is organized around the framework of this taxonomy.

This study engaged parents to develop concise, informative, and comprehensible text messages for an immunization reminder system using Human Factors techniques. Fifty parents completed a structured interview including demographics, technology questions, willingness to receive texts from their child’s doctor, and health literacy. Each participant was assigned to one user-centered design test: card sort, needs analysis, or comprehension. The majority of respondents were female (90%), White non-Hispanic (62%), and averaged 29 years ( SD = 5). Nearly all (96%) had “adequate” health literacy. The card sort, an activity in which end users organize information into categories, identified six pieces of critical information. These were compiled into eight example texts, which were ranked in the needs assessment. The top two were assessed for comprehension, with 100% of participants able to understand the content and describe intention to act. Using user-centered design methods, the authors developed concise, informative text messages that parents indicated would prompt them to schedule their child’s immunization appointment.

Cognitive flexibility, the ability to abandon an active cognitive strategy in favor of another, may predict successful performance in tasks that require divided attention, but measuring cognitive flexibility is challenging. Here, two studies assessed cognitive flexibility using Grant and Berg’s (1948) Wisconsin Card Sorting Task (WCST) and an easy-to-administer puzzle task under development. Three variables of flexibility in the WCST were hypothesized to predict puzzle performance. In Study 1, undergraduate students (n = 88) from California State University, Northridge completed both the WCST and the puzzle. Results indicated that only the variable ‘trials to complete first category’ reliably correlated with puzzle performance, therefore a revised puzzle was created. In Study 2, undergraduate students (n = 40) from the same university repeated the experiment, resulting in a stronger relationship between ‘trials to complete first category’ and the modified puzzle. The results suggest that cognitive flexibi...

Adults with autism spectrum disorders (ASD) have shown deficits in switching between rules governing their behaviour, as have high-functioning children with ASD. However, there are few studies of flexibility in lower-functioning children with ASD. The current study investigated this phenomenon with a group of low-functioning children with ASD compared to a mental-age-matched comparison group. The ASD group learned an initial discrimination task as quickly as the matched comparison group, but when the rule governing the discrimination was shifted, the comparison group learned the task with fewer errors, and made the discrimination more quickly than the groups with ASD. These findings suggest that low-functioning children with ASD do display the predicted deficits in extra-dimensional shift.

Hewes’s socioegocentric model is applied to discussion data from an information-sharing task. The primary question is whether discussion data display psychological (individual-level) or interactional (group-level) associations among contributions. Two studies are described. The first used a card sort task to group the 42 items from the choose-the-best-candidate task. The second study used the groupings from the first task to code information-based contributions to discussion. Results indicated that participants reliably group the items into valence-based categories and that there were valence-based associations between items contributed to discussion. However, the associations were at the individual level of analysis—there were no group-level effects for information sharing. However, other features of the discussions did have group-level effects, and we argue for a better understanding of information sharing in terms of the comments that frame information-based contributions to interaction.

An occupational therapy card sort was developed as part of an enquiry-based learning initiative to enable students to evaluate their skill development. The online card sort comprises modified playing cards that students deal to assess their skill development. The card items reflect standards of proficiency for occupational therapists. The card sort was evaluated to determine whether it was effective in measuring changes in skills over time and to see whether students found it useful to identify their skill levels and plan their learning needs. Data were collected from second-year occupational therapy students, both by using questionnaires and by collecting completed card sorts from 35 students at two intervals in their studies for comparison. The card sort was found to be sensitive to changes in students' perceived skill levels over time and to detecting differences between categories of skills. The students found the card sort effective as a means to promote the active learning skills necessary for enquiry-based learning. The card sort was identified as a useful tool for planning professional development needs and for engaging in continuing professional development.

This article argues that accessing and taking note of the voice of pupils is critical in working towards removing barriers to effective learning. Pupil voice was accessed using an innovative form of group interview incorporating an Ishikawa or fishbone tool and card sorting exercise. The data were collected from 132 pupils, from Years 7 to 9 (aged 11 to 14) in three UK secondary schools. The techniques proved to be successful ways to help pupils to articulate their views. There was an immediacy and authenticity in the process, which generated 13 ‘classroom conditions’ that pupils felt were significant in securing effective learning. The 13 conditions potentially provide a grounded framework to engage teachers and student teachers in analysis, reflection and discussion regarding effective teaching skills and strategies.

Existing reviews of card sorting applications usually focus on the end user's perspective of the sorting experience. In this paper, we describe objective (task success and time on task) and subjective (task difficulty, ease of interpretation of results output, satisfaction scores, and preference rankings) results from a usability evaluation of three card sorting applications from a researcher's perspective. Results from this analysis show that the ease of setting up a card sort exercise and the ability to quickly generate meaningful results are of vital importance to researchers. Design recommendations for computerized card sort applications are discussed.

Teaching philosophy statements clarify why we do what we do in the classroom, and the process of drafting a philosophy offers an opportunity for developmental reflection. Personal teaching philosophies can be grounded in the shared foundation of historical educational philosophies. The authors offer here for facilitators a reflective card-sort exercise that helps surface the philosophical roots of personal teaching philosophies and helps teachers create or renew a teaching philosophy statement. They explain the exercise activities and typical outcomes based on their experiences facilitating the exercise. The authors conclude with insights and considerations for facilitators.

Under incidental instructions, thirty-eight 2-year-olds were trained on a six-element deterministic sequence of spatial locations. Following training, subjects were informed of the presence of a sequence and asked to either reproduce or suppress the learned material. Children's production of the trained sequence was modulated by these instructions. When asked to suppress the trained sequence, the children were able to increase generation of paths that were not from the training sequence. Their performance was thus dependent on active suppression of knowledge, rather than on a random generation strategy. This degree of control in 2-year-olds stands in stark contrast to 3-year-olds∗ failure to control explicitly instructed rule-based knowledge (as measured by the dimensional-change card-sort task). We suggest that the incidental nature of a learning episode enables the acquisition of a more procedural form of knowledge with which this age group has more experience prior to the onset of fluent language.

This research investigates the design and evaluation of interfaces for the presentation of search results using categorization. Card sorting study was conducted to design categorized search results interfaces. Eye-tracking technology was utilized to evaluate the designed category interfaces, comparing them with a conventional list interface. Two factors of category interfaces, the approach used to order categories and the number of abstracts initially displayed within each category, are investigated. Forty participants performed two types of tasks, navigation and information search, using a PC integrated with a non-intrusive Tobii 1750 binocular eye tracking system. Performance measures such as time for search and percentage of tasks completed did not reveal significant differences among the category and list interfaces. Process measures such as overall number of fixations, mean fixation durations, and relative change from the baseline pupil diameter also did not show significant differences among the category and list interfaces. Although no significant differences are found, there is a consistent positive trend in the performance, process, and subjective measures that supports design of categorical search results interfaces.

Software and product designers use card sorting to understand item groups and relationships. In the usability community, a common method of formal statistical analysis for open card sort data is hierarchical cluster analysis, which results in a tree of the items sorted into distinct, nested clusters. Hierarchical cluster analysis is appropriate for highly structured settings, like software menus. However, many situations call for softer clusters, such as designing websites where multiple pages link to the same target page. Factor analysis summarizes the categories created in card sorts and generates clusters that can overlap. This paper explains how to prepare card sort data for statistical analysis, describes the results of factor analysis and how to interpret them, and discusses when hierarchical cluster analysis and factor analysis are appropriate.

Several studies have been published listing sources of practice knowledge used by nurses. However, the authors located no studies that asked clinicians to describe comprehensively and categorize the kinds of knowledge needed to practice or in which the researchers attempted to understand how clinicians privilege various knowledge sources. In this article, the authors report findings from two large ethnographic case studies in which sources of practice knowledge was a subsidiary theme. They draw on data from individual and card sort interviews, as well as participant observations, to identify nurses’ sources of practice knowledge. Their findings demonstrate that nurses categorize their sources of practice knowledge into four broad groupings: social interactions, experiential knowledge, documents, and a priori knowledge. The insights gained add new understanding about sources of knowledge used by nurses and challenge the disproportionate weight that proponents of the evidence-based movement ascribe to research knowledge.

The authors successfully used the card sort data collection technique with 17 female family caregivers in a large ethnographic study of nonsupport. In this article, they describe the practical issues they addressed. Initially, they developed strategies to construct meaningful statements that reflected key themes and were manageable in an interview. Later, to address challenges for participants, they developed approaches to assist women in moving beyond their own experience, dealing with test anxiety, and anticipating an emotional response. To facilitate effective data collection, they made detailed arrangements in advance, collected “talk aloud” data that captured women’s decisions, and maintained accurate records. The women felt validated in their caregiving roles, as the card statements reflected their experience and rich data was elicited. The card sort exercise contributed information about variability in the meaning of similar interactions for different women and a beginning understanding of the criteria women used to make decisions.

Previous research has shown that individuals process stimuli differently based on the modality in which the stimuli are presented. In an effort to further understand how stimuli are processed and organized, the current study utilized card sorting in one of three media modalities (text only, picture only, and text and picture combined) and in one of two instructional styles (free sort or structured sort, via function) to assess the degree of sharedness between the resulting mental models. It was believed that structure sorts would produces greater levels of sharedness and that the semantic nature of the structured sort would yield the greatest levels of sharedness in the text only condition. Structured sorts did in fact produce better results than unstructured sorts as expected; however, in the structured sort, the picture only condition provided the best results. In addition, comparisons were made to a previous study suggesting that the proper instruction by modality combination should be evaluated based on domain and desired outcomes. The implications of these findings as well as the need for further research are discussed.

The cellular phone market has advanced rapidly and consequently has produced a variety of functional capabilities and interface designs. However, this same rapid proliferation has led to problems with the diffusion and organization of design guidelines. Given the current market demands on usability that include addressing the needs of users with disabilities, cellular phone developers need easy-to-access and comprehensible design guidelines. This study was conducted to develop a guidelines tool to support the development of usable cellular phone interfaces. The tool was developed on the basis of knowledge elicitation techniques of experts, including card sorting and category evaluation. Cluster and content analyses were used to analyze the quantitative and qualitative data. Categories of guidelines and their meanings are provided.

Given the increasing time pressure on parents and children, in contrast to the decreasing amount of time that families spend together at home, it is imperative that homes be designed to maximize the opportunity for family members to interact during the limited time they have together each day. This exploratory study was conducted to understand how homes can be designed to best support interaction. This issue was investigated through the use of design programming: a systematic process of creating guidelines for a building's optimal performance based on the analysis of information about the building and its users. Four requirements had to be met by the home to facilitate family interaction; the home had to (a) accommodate specific family activities, (b) accommodate multiple activities in the same location within the home, (c) facilitate household tasks, and (d) maximize the aesthetic appeal of family spaces. Activity logs, parental interviews, and a card sort method were used with 24 single parent and dual–earner families with elementary school–aged children. The findings from the activity logs indicated that families interact primarily around mealtime {dinner} and when viewing television. The primary place for interaction is the kitchen or kitchen/dining area or the living/family room. Analysis of the interviews indicated that families today value having a large kitchen area because family interaction often occurs while parents are cooking or cleaning in the kitchen area and children are doing homework or other activities immediately in the area. When asked to create their ideal place for family interaction parents most often created family/living areas. The study concluded that areas that support concurrent activities best facilitate family interaction.f

14 students enrolled in introductory psychology classes engaged in several trials of card sorting to acquire the rule for correct sorting They also were instructed to keep track of time simultaneously and to continue sorting for what they believed was 20 sec. The duration of card sorting was significantly greater for the first sorting trial, during which the rule was unknown, relative to the second consecutive trial of errorless sorting when the rule had been learned. After a shift in the rule mean sorting duration was reduced from the second errorless trial but returned to a value comparable to that on the first sorting trial. Results are interpreted in terms of the attention-allocation model of time estimation.

Card sorting and concept mapping are two popular techniques used for assessing structural knowledge. However, manual administration of these tasks has been cumbersome in the past. This demonstration serves as an introduction to the TPL-KATS – Card Sort and Concept Map software tool developed at the Team Performance Lab at the University of Central Florida. TPL-KATS is presented as a possible solution to the problems of manual administration. The software contains computerized versions of both techniques, with automatic administration and scoring capabilities. This demonstration provides an introduction to the card sort and concept map tasks, and describes the basic need for this software tool. A full description of TPL-KATS – Card Sort and Concept Map software is presented, along with a demonstration of its advantages over current techniques.

We studied the impact of Ecological Interface Design (EID) displays on mental models using the DURESS II simulator. To assess mental models we used a card sorting exercise, a network building exercise based on the Abstraction Hierarchy, and a system behavior prediction exercise. Participants in the EID condition performed significantly better on the card sorting and network building exercises, but poorer on the prediction exercise. Our results are mixed, suggesting that EID may improve the understanding of means- end links and yet, not improve the ability to predict system behavior. The relatively limited experience of participants with the interface as well as our selection of knowledge elicitation techniques may have influenced these results. More research is needed to fully evaluate the impact of EID on mental models.

The differences between the processing of textual and pictorial information have been a topic of research for some time now. Previous research concerning the modality of information has often concentrated on the speed of processing rather than the organizational differences that may exist. This experiment utilized card sorting to evaluate the changes in knowledge organization that occur when information is presented in text and picture formats. In addition to this, the structure of the elicitation task was manipulated to evaluate its effects on sharedness. It was found that textual stimuli produced a greater sharedness among participants in a free sorting task. However, for a structured sorting task, results reversed, and pictorial stimuli created a greater level of sharedness. Overall, structured sorting tasks produced a greater level of sharedness than the free sorting condition, regardless of modality.

The method of card-sorting is a very useful tool for interface design as it can help to uncover the mental models of users concerning the relationships among various interface items (links, menu choices, etc.). However, methodological difficulties associated with traditional sorting procedures such as time and space requirements, as well as tedious data analyses, have limited the practical utility of card-sorts. Some software tools have emerged that purport to automate the card-sorting procedure, as well as analysis of results, in order to attenuate or eliminate these difficulties. Three tools available for free download from the Internet were chosen for review. The tools were compared and contrasted in terms of efficiency and effectiveness, from the standpoint of the researcher as well as the participant. While all three tools were found to save much time and effort over manual sorting procedures, each also carries with it a set of unique capabilities and limitations, leaving open the possibility for improved automated sorting tools in the future.

One way to determine training needs and to evaluate learning is to measure how trainees organize knowledge using a card sorting task. While card sorting is a valid tool for assessing knowledge organization, it can be work intensive and error-prone when it is manually administered. For this reason, we developed a software tool that computerizes the card sort task. We present a study that was conducted to determine whether the computerized version of card sorting is comparable to the manual sort. One-hundred eight participants completed two card sorts, either two manual sorts, one manual and one computerized sort, or two computerized sorts. No differences were found between the administration methods with respect to card sort accuracy, test-retest scores, and number of piles created. Differences between the two methods were found in administration time and length of the pile labels. These differences disappeared after one computerized administration. We conclude that a computerized card sorting task is just as effective at eliciting knowledge as a manual card sort.

Knowledge elicitation and mental model assessment methods are becoming increasingly popular in applied psychology. However, there continue to be questions about the psychometrics of knowledge elicitation methods. Specifically, more needs to be known regarding the stability and consistency of the results over time (i.e., whether the methods are reliable) and regarding the degree to which the results correctly represent the underlying knowledge structures (i.e., whether the methods are valid). This paper focuses on the convergence among three different assessment methods: (a) pairwise relatedness ratings using Pathfinder, (b) concept mapping, and (c) card sorting. Thirty-six participants completed all three assessments using the same set of twenty driving-related terms. Assessment sequences were counterbalanced, and participants were randomly assigned to one of the six assessment sequences. It was found that the three assessment methods showed very low convergence as measured by the average correlation across the three methods within the same person. Indeed, convergence was lower than the sharedness across participants (as measured by the average correlation across participants within the same assessment method). Additionally, there were order effects among the different assessment sequences. Implications for research and practice are discussed.

Okanda, et al. (2010) reported new evidence concerning associations between language ability, bilingualism, and executive functioning early in development. The paper adds to a growing body of literature suggesting that bilingualism is associated with advantages in executive functioning generally, and the Dimensional Change Card Sort task in particular. However, as with all findings that hinge on between-group comparisons, there is a need to exercise caution before drawing firm conclusions about the effects of bilingualism on the development of executive control. Several lines of recent evidence are outlined that challenge key assumptions underlying the standard account of the bilingual advantage. Okanda, et al.'s findings are discussed in light of this evidence.

This research is the first part of a project on the relationship between cognitive and social abilities in childhood, with special attention on the analysis of the relationship between ‘exibility in thinking and co-operative versus competitive solutions of social conflicts with peers. Flexibility is defined as the ability to suppress a response in order to ”nd a new one. The hypothesis is that the more ‘exible children are the more able they are to detach themselves from the conflict situation and more able to consider it in an integrated way—therefore they are more able to ”nd a co-operative solution. We studied a sample of 152 7-year-old children. In this research, the Wisconsin Card Sorting Task was used for the evaluation of ‘exibility in thinking. To evaluate co-operative and competitive behaviour, an interactive drawing task was used, where children were asked to draw together with tied pencils. Two children, of the same sex and of the same level (high or low) of ‘exibility, were put to work together in the interactive task. Results confirm the hypothesis, and indicate that the more ‘exible children are the more able they are to co-operate with their peers, to take turns, and to verbalise about other topics not related to the task

Patient participation in various aspects of healthcare is rapidly expanding. However, patients' preferences with regard to participation in treatment decision-making remain relatively under-explored, as does the congruence between such preferences and patients' perceptions of their actual role in decision-making. The aims of this study were to identify and to compare treatment decision-making role preferences and perceptions of the actual decisional role in a sample of renal patients. A cross-sectional survey was conducted at a single regional renal unit in the North of England. A convenience sample of 405 patients was obtained (pre-dialysis N=155; dialysis N=103; transplant N=147). Preferred and perceived role were identified using a card sort technique (choice of five options, active to passive). Rationales for role preference were recorded verbatim. Most patients were found to prefer passive (N=186, 45.9%; 95% Cl 41.2- 50.9) or collaborative (N=147; 36.3%; 95% Cl 31.3-40.7) decisional roles; perceived role was typically passive (N=319; 78.8%; 95% Cl 74.8-82.8). Trust in health professionals, perceived lack of knowledge and severity of condition were the most frequently cited determinants of role preference. Patients' decisional role preferences were often at variance with their perceived roles. Taking a highly active role in treatment decision-making was undesirable to most patients. Trust in healthcare professionals was an important contributor to desire to defer or retain control over treatment decisions.

A seven-step, 36-card, vocational card sort mapping task with concurrent verbalizations is presented which may be used in the initial stages of career exploration to assist counselors in assessing a client's development and integration of occupational knowledge. Based on cognitive theory, each step in the assessment procedure is designed to elicit certain kinds of information pertaining to the manner in which occupational knowledge is represented in long-term memory. A formal cognitive mapping task procedure with concurrent verbalizations, employing a novice and an expert under standardized conditions, is used to demonstrate how the task elicits individual differences in cognitive structure. The article also demonstrates how the technique may be used for the diagnosis of a career problem and for the prescription of career interventions to enhance the development of occupational knowledge. The potential use of the technique for research in the development of occupational knowledge is also discussed.

52 members of the college community at a small liberal arts university (Ages 25–72) were administered a 60-min. battery of tests measuring different aspects of cognitive performance. Stepwise regression equations detected significant relationships between the predictor variables of age and education and measures of the WAIS-R, the Logical Memory subtests and Trails A. Age predicted performance on Trails A, the Digit Symbol subtest of the WAIS-R, the Logical Memory II component of the Wechsler Memory Scale-R, some measures of the Wisconsin Card Sorting Test, and Full Scale WAIS-R IQ. Education was a significant predictor of performance on the remaining WAIS-R subtests and the Failure-To-Maintain Response Sets measure of the card sorting. However, the results of the present study also suggest that over-all cognitive decline with advancing age is not an eventuality, at least among the highly educated, intellectually active individuals.

There are a variety of ways to analyse qualitative data, ranging from manual analysis by the time honoured process of coloured pens, to a card sorting system, to using a computer software program. Because of the voluminous quantities of data generated through the author's own research, a software program specifically developed for qualitative research, NUD.IST, was chosen to expedite analysis. This paper will describe the process taken to convert the raw data for a qualitative nursing study into preliminary chapters for inclusion in a doctoral thesis. The research took an ethnographic approach to explore the parenting practices of sole supporting teenage mothers. Data consisted of verbatim transcripts from interviews, field notes and personal reflective journalling. NUD.IST provided a tool which helped reduce the data into a more manageable form. It served both mechanical and conceptual functions.

The relationship between working memory (WM) capacity and three executive function tests, which were adopted from clinical neuropsychology, was studied. The subjects were normal 15–16-year-old students. A large set of WM measures included digit and word span, a modified memory-updating task, and five different complex WM span tasks. The complex span measures and the memory-updating task showed high intercorrelations. Of the three executive function tests, the Wisconsin Card Sorting Test (WCST) correlated significantly with WM tasks, the storage function of WM probably being a limiting factor in card sorting. The global performance measures of the WCST were more dependent on WM capacity than the number of perseverations. The two other executive function tests-the Tower of Hanoi and Goal Search Task-did not correlate with WM tasks. None of the executive function tests exhibited any significant intercorrelations. The results are in agreement with earlier studies, which have found separate executive functions. The present results and evidence from earlier studies suggest that there does not exist a unitary, limited-capacity central executive.

The California Card Sorting Test (CCST) is a relatively new measure of executive function which seems to hold promise for advancing our understanding of the cognitiveprocesses underlying high level problem-solving ability. Unfortunately, despite its manyadvantages, administration time may be prohibitively long for practical use in many clinical settings. It was hypothesized that the constituent stimulus subsets might be used individually as brief alternatives to the full-length CCST. Interset and set/total correlationswere examined in two groups of subjects (college students, children). Based on the findings of unacceptably low interset correlations, it was concluded that the subsets are notequivalent and should not be treated as interchangeable. However, the generally goodcorrelations between the subset and total test scores indicate that the use of the individual stimulus subsets as "short forms" is justified.

The present study sought to determine the utility of the unweighted sum as an alternative to conjoint measurement in forming a workload composite measure with the Subjective Workload Assessment Technique (SWAT). The data from a simulator study designed to identify pilot workload associated with alternative cockpit configurations in a night air interdiction mission were reanalyzed using a percentage measure (SUMPCT) based on the unweighted sum of the three SWAT rating scales. Results indicated that the CONJOINT and SUMPCT workload composites were highly correlated and were equally sensitive to detecting differences in the independent variables. The reasons for the equal sensitivity of the two composite measures are discussed. This study adds to the growing body of evidence that one can form a composite measure of workload using the SWAT rating scales without the necessity of having to conduct the time-consuming card sort.

Cognitive mapping techniques refer to methods used to elicit the structure and content of people's mental models. There is a growing interest in applying these techniques to the study of managers' mental models of strategic management issues. This paper reports on a new method for mapping managers' mental models of competitive industry structures, that is based upon recent developments in the cognitive psychology of the categorization of concepts. This method, a simple visual card sorting technique, is quick to administer and interpret. The method is evaluated with respect to its psychometric properties against the well established, but potentially cumbersome, repertory grid technique. The method is demonstrated to have good validity by comparing it with the repertory grid technique.

Effects of goal-setting and locus of control were studied in 188 undergraduate men in a laboratory setting using a card-sorting task. Two trials were employed with one of three goals assigned on Trial 2 (easy, moderately difficult, or hard goals representing, respectively, a 10%, 20%, or 50% increase over Trial 1 card sort). A curvilinear relationship moderated the relationship between goal-setting and card sorting when differences in locus of control were examined. However, when these individual differences were partialled out, a linear relation between goal-setting and card sorting typical of the goal-setting literature was obtained.

One important element of hypertext design is the presentation of the hypertext cue. This study compared learning relationships about animals in an oriental zodiac using one of two online, hypertext presentation styles and printed text. In one hypertext condition, the cues were embedded in the text; in the other condition, the cues were located in the margins. A control condition contained unrelated study materials. The database on the zodiac consisted of a multiple hierarchy. Two measures of learning were used. Reaction time before and after a brief study period was used to assess knowledge about the relationships among animals in the zodiac. Participants' ability to correctly reproduce the underlying model of the zodiac in the database using a card sorting task was used to assess the degree to which they learned the underlying model. Analysis of variance revealed a significant interaction between learning (pre- and post-study period) and presentation condition (hypertext embedded in text, hypertext in the margin, printed text, unrelated text material). An analysis of the results of the card sort data revealed that participants more accurately reproduced the underlying model after study with hypertext, regardless of the location of the cue, than with text. The results are discussed with respect to the design of hypertext information and the need for future research.

The literature suggests that there exists a need for fewer but more dearly defined subgroupings in behavior rating instruments. The present study focuses on the items from seven behavior rating instruments in an effort to organize the items conceptually into as few behavior clusters as possible. The procedures utilized extensive card sort activities and a panel of experts who examined each of the items and made decisions regarding subgroup placement of the items. The find ings provide one way to organize behavior descriptors and thereby unify behavior rating as recommended by several authors. In addition, the classification of behav iors provides the field with a conceptual organization which may form the basis for future conceptual and empirical activities. In recent years, there has been a proliferation of behavior rating instruments, each designed to assist in the assessment of the behavior of children and youth. These instruments have been used increasingly as part of the initial screening and identification procedures for students thought to have behavior problems (Bullock & Wilson, 1986; Smith, Wood, & Grimes, 1985). In addition, they have been used to aid in the development of intervention plans and tracking of behavior change in subjects (Bullock & Wilson, 1986; Wilson, 1980) and in research studies of various types (e.g., Achenbach & Edelbrock, 1986; Bullock, Wilson, & Sarnacki, 1988; Campbell, Bullock, & Wilson, 1989; Eaker, Allen, Gray, & Heckel, 1983; Quay & Peterson, 1983; Sarnacki, 1987). Behavior rating instruments have evolved from an intuitive clinical notion about behavior to complex analyses using exploratory factor analytic approaches to develop clusters of highly correlated behaviors. The latter approach has resulted in a great variety of labels being assigned to the various behavior clusters. Because of the lack of coordination or unifying theory in the factor analytic work that has taken place, cluster labels have not been applied consistently. In addition, exploratory factor analysis produces indeterminant solu tions (Kim & Mueller, 1978) such that any given solution depends upon (a) the items placed in the analysis, (b) the computation method, and (c) the type of rotation used for a final solution. Therefore, since factor solutions can be very different, the resulting dimensional interpretations can also differ depending on aspects of the analysis rather than the under lying dimensional realities of behavioral disorders. Some factor analytic work has been done to sort out the variety of item clusters represent ing behavior dimensions that presently exist in behavior rating scales. For example, Achenbach and Edelbrock ( 1978) examined 27 different scales and arrived at four different categories: overcontrolled, undercontrolled, pathological detachment, and learning prob lems. In a related study of 24 rating instruments, Hoge (1983) summarized 53 different item grouping names into three categories: personality adjustment, social adjustment, and adjustment to academic demands. A study of two instruments with confirmatory factor analysis conducted by Hale and Zuckerman (1981 ) found that a three-factor model was best for explaining the mathematical characteristics of a group of items' covariance pattern. A special thanks is extended to Juane Heflin. J. Michael Reese, Ronald L. Sarnacki, and Patti Westerlage who, in addition to the authors, served as the panel of experts for the project. Behavioral Disorders February 1990 [Vol.15 No. 2 87-99] 87 This content downloaded from 157.55.39.78 on Mon, 20 Jun 2016 07:33:25 UTC All use subject to http://about.jstor.org/terms These factors were identified as overreaction, underreaction, and inadequacy-immaturity. The three studies reported above show different results: there are two different dimension interpretations — Achenbach and Edelbrock (1978) and Hoge (1983); and two different number of factors — Hoge (1983) and Achenbach and Edelbrock (1978), fourfactors versus Hale and Zuckerman (1981), three factors. The lack of agreement in previous research suggestsa need forfurther investigation of the problem. Problems associated with indeter minacy, as pointed out earlier, suggest a need for a different approach. The results of all three studies imply, however, that there is a multitude of rating scale dimensions which can be simplified and any analysis of rating scale data should result in a relatively few number of dimensional categories into which behavioral rating items can be placed. One issue that is basic to the labeling of item clusters is the correspondence between the label and the categorical nature of the behaviors described by the items. Labeling a cluster of rating items is a result of the interpretation of the behavior described in the items. For example, the interpretation of an item describing drug consumption behavior that is included in a cluster labeled Aggressive Tendency would be different if the same behavior were in a cluster labeled Irresponsible Activities. The three studies cited above addressed the issue of labeling through the use of statistical analysis of behavior rating data. Statistical methodology, however, is appropriate for testing the results of investigations devised to reflect theoretical notions, but not generate them. The matter of grouping behaviors described by rating items is the basis of a theoretical interpretation of the behaviors. There is a need to generate a unified conceptualization of behavioral disorders so that research and any work that is based upon research can be considered comparable. That is, if work is conducted by different researchers with different interpretations of the notion of behavioral disorders, then similar work may result in different findings. If these different findings result in practical applications, then the applica tions may appear very different and may not produce the same results. Ultimately, differen tial interpretations of behaviors considered relevant to behavioral disorders will result in multiple systems of identification, treatment, and interpretation of the results of treatment, none of which will be consistently understood. This confusion presently exists in the area of behavioral disorders and will likely not be changed unless the theoretical differences can be defined and compared. This sort of definition can only be done by examining a large number of behavioral items and generating a unified theory about how they are grouped together and what is the most appropriate category title for the groupings. If it is possible to interpret the larger universe of behavior rating items with relatively few distinct well-defined categories, then the difference between subsets of items (i.e., particu lar rating instruments) can be better understood. The research reported here is an attempt to do just that; that is, to examine several behavior rating instruments and determine in a unifying fashion the basic elements of disordered behavior generally reflected in the behavior rating items used in the instruments. A unified theory would allow researchers to determine the comparability of research done with a variety of different theoretical catego ries; it would probably assist in the consistency of student classification, and it might also help those involved in the education of behaviorally disordered students to measure and interpret student behavior in a consistent fashion with results that are more comparable than with the present set of rating instruments.