Search

Introduction Health professions education faces transitions from monodisciplinary to integrated education and from soloist teachers to interdisciplinary teacher teams. Interdisciplinary teamwork has been found complex and prone to conflict. Teachers’ perceptions of why some teams work and learn as a real interdisciplinary team and others do not are lacking in this setting. We studied the factors that teachers perceive as enabling and/or inhibiting interdisciplinary team learning. Methods In this exploratory, qualitative study, we conducted 17 semi-structured, vignette-guided interviews with teachers recruited from diverse disciplines in undergraduate health professions programmes at Maastricht University, the Netherlands, through maximum variation sampling. Team learning research informed data collection and template analysis. Results We identified three themes representing the factors that teachers perceived to influence interdisciplinary team learning: ‘alignment/misalignment with the educational philosophy’ (regarding personal attributes, tendencies and motivation), ‘leadership practices’ (encompassing team vision, responsibility and reflection), and ‘involvement in organisational processes’ (covering organisational decision-making, support and learning opportunities). For interdisciplinary team learning in development of integrated education, teachers emphasised their personal ability to move beyond disciplinary boundaries. Shared team leadership enabled the creation of a shared vision, shared responsibility, and team reflection. Lastly, teacher involvement in educational management, peer support and learning was considered important. Discussion To work beyond disciplines in health professions education, teachers should take an interest in integrated education, share responsibility and work in an environment where people continuously learn from others. Organisations can facilitate this by involving teachers in decision-making processes and providing faculty development aimed to foster shared leadership and team reflection. Supplementary Information The online version of this article (10.1007/s40037-020-00644-7) contains supplementary material, which is available to authorized users.

Internal quality assurance (IQA) is one of the core support systems on which schools in the health professions rely to ensure the quality of their educational processes. Through IQA they demonstrate being in control of their educational quality to accrediting bodies and continuously improve and enhance their educational programmes. Although its need is acknowledged by all stakeholders, creating a system of quality assurance has often led to establishing a 'tick-box' exercise overly focusing on quality control while neglecting quality improvement and enhancement. This AMEE Guide uses the concept of quality culture to describe the various dimensions that need to be addressed to move beyond the tick-box exercise. Quality culture can be defined as an organisational culture which consists of a structural/managerial aspect and a cultural/psychological aspect. As such this AMEE Guide addresses tools and processes to further an educational quality culture while also addressing ways in which individual and collective awareness of and commitment to educational quality can be fostered. By using cases within health professions education of both formal and informal learning settings, examples will be provided of how the diverse dimensions of a quality culture can be addressed in practice.

The concept of quality culture has gained increased attention in health professions education, drawing on insights that quality management processes and positive work-related attitudes of staff in synergy lead to continuous improvement. However, the directions that guide institutions from quality culture theory to educational practice have been missing so far. A prospective qualitative case study of three health professions education programmes was conducted to explore how a quality culture can be enhanced according to the experiences and perspectives of educational leaders. The data collection was structured by an appreciative inquiry approach, supported with vignette-based interviews. A total of 25 participants (a selection of course coordinators, bachelor coordinators and directors of education) reflected on quality culture themes to learn about the best of what is (Discover), envision positive future developments (Dream), identify actions to reach the desired future (Design), and determine how to support and sustain improvement actions (Destiny) within their own educational setting. The results are presented as themes subsumed under these four phases. The experiences and perspectives of educational leaders reveal that peer learning in teams and communities, attention to professional development, and embedding support- and innovation networks, are at the heart of quality culture enhancement. An emphasis on human resources, (inter)relations and contextual awareness of leaders stood out as quality culture catalysts. Educational leaders are therefore encouraged to especially fuel their networking, communication, coalition building, and reflection competencies.

INTRODUCTION: Inclusive educational leaders promote teacher team functioning. To support leader inclusiveness, we designed and implemented a faculty development programme focusing on leader identity formation. We investigated (1) how participants' leader identity developed throughout the programme and (2) how the design principles contributed to this process, according to participants.METHODS: A design-based research approach was followed. Participants were 7 course coordinators, leading an interdisciplinary teacher team. To study leader identity development, participants repeatedly filled out a validated questionnaire. To investigate how design principles contributed, observational field notes, facilitator debriefings, a programme evaluation questionnaire and a semi-structured focus group were used. Thematic analysis was applied for qualitative data.RESULTS: Participants gained broader views on leadership, moving from individual dominance towards engaging team members. Most participants initially experienced a deconstruction of their former leader identity and became motivated to improve leadership qualities. Competence-building, reflecting and receiving feedback on workplace experiences, and practicing in a safe environment were perceived to be key for identity development.CONCLUSIONS: We developed and evaluated a leader identity programme which can convert teachers' classic leadership views towards views incorporating social interactions and relationships. We demonstrated how social interactions provide opportunities to learn from pe.ers in the work environment.

Phenomenon: Developing modern medical curricula requires collaboration between different scientific and clinical disciplines. Consequently, institutions face the daunting task to engage colleagues from different disciplines in effective team collaboration. Two aspects that are vital to the success of such teamwork are "team learning behavior" by all team members and "leader inclusiveness behavior" by the team leader. Team members display team learning behavior when they share information, build upon and integrate each other's viewpoints. The team leader can promote such team learning by exhibiting inclusiveness behavior, which aims to encourage diversity and preserve individual differences for an inclusive workplace, nurturing engagement in teamwork. There is a paucity of in-depth research on leader inclusiveness behavior in the field of medical education. This case study aimed to offer unique insight into how leader inclusiveness behavior manifests itself in a successful interdisciplinary teacher team, demonstrating team learning behavior in undergraduate medical education. Approach: We conducted a qualitative, ethnographic case study using different but complementary methods, including observations, interviews and a documentary analysis of email communication. By means of purposive sampling, we selected an existing interdisciplinary teacher team that was responsible for an undergraduate medical course at Maastricht University, the Netherlands, and that was known to be successful. Chaired by a physician, the team included planning group members and tutors with medical, biomedical, and social sciences backgrounds as well as student-representatives. In the course of one academic year, 23 meetings were observed and recorded, informal interviews were conducted, and over 100 email conversations were collected. All data were submitted to a directed content analysis based on team learning and leader inclusiveness concepts. Findings: Leader inclusiveness behavior became evident from verbal and non-verbal interactions between the team leader and team members. Leader inclusiveness behavior that facilitated team learning behavior manifested itself in five actions undertaken by the team leader: coordinating, explicating, inviting, connecting, and reflecting. Similarly, team members facilitated team learning behavior by participating actively, speaking up behavior, and mimicking leader inclusiveness behavior. These behaviors demonstrated engagement and feelings of inclusion, and reinforced leader inclusiveness behavior by creating additional opportunities for the leader to exhibit such behavior. Insights: This case study responds to the need for inclusive leadership approaches in medical education. Our findings build upon theoretical knowledge on team learning and leader inclusiveness concepts. By studying behaviors, interactions and documents we obtained in-depth information on leader inclusiveness. Our findings are unique in that they demonstrate how leader inclusiveness behavior manifests itself when leaders interact with their team members. This study provides health professionals who are active in education with practical suggestions on how to act as a successful and inclusive leader. Finally, the behaviors identified open up avenues for future professional development initiatives and future research on team leadership.

Supplemental Digital Content is available in the text., Purpose To explore team learning processes among interdisciplinary teacher teams in the development of integrated health professions education and to investigate students’ perspectives on the quality of the educational courses. Method Using an exploratory, sequential mixed-methods design, the first author conducted 17 vignette-guided, semistructured interviews with teachers originating from diverse disciplines. These teachers worked in different courses of integrated, undergraduate health professions programs at the Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands. The interview guide and vignettes were based on team learning research. The interviews sought to establish how interdisciplinary teacher team members work together on integrated curricula. The vignettes reflected constructs of team learning processes: sharing, co-construction, and constructive conflict. Data were collected between November 2017 and March 2018 and analyzed using template analysis. Sequentially, course evaluation data were used to provide a descriptive analysis of students’ perspectives on educational quality (course organization, structure, learning effect, and alignment). Results Three team approaches were identified. In fragmented teams or “hangouts,” teachers individually worked on tasks that they were interested in, leaving their disciplinary mark. Framework-guided teams or “distribution centers” aimed to work within the given frameworks and organizational expectations, striving for disciplinary balance. Integrated teacher teams or “melting pots” used an interdisciplinary approach on all topics and put students at the center. Integrated teams reflected high-level team learning processes and were most satisfied with their (team)work. In contrast, fragmented and framework-guided teams mainly reflected low-level team learning processes. Students evaluated courses of integrated teacher teams highest on all investigated quality items (course organization, structure, learning effect, and alignment). Conclusions Successful interdisciplinary teacher teams are represented by an integrated approach with high-level team learning behavior and the best course evaluations. Therefore, health professions education management should actively encourage and facilitate integrated teacher teamwork.

Meeuwissen et al. present 'team learning' as a process that helps explain how successful teams work, clarifying how behavioral dimensions of team learning lead to new, shared understandings.

Introduction: Student-staff partnerships as a concept to improve medical education have received a growing amount of attention. Such partnerships are collaborations in which students and teachers seek to improve education by each adding their unique contribution to decision-making and implementation processes. Although previous research has demonstrated that students are favourable to this concept, teachers remain hesitant. The present study investigated teachers’ conceptions of student-staff partnerships and of the prerequisites that are necessary to render such partnerships successful and enhance educational quality. Method: We conducted semi-structured interviews with 14 course coordinators who lead course design teams and also teach in 4 bachelor health programmes, using Bovill and Bulley’s levels of student participation as sensitising concepts during data analysis. Results: The results pointed to three different conceptions of student-staff partnerships existing among teachers: Teachers teach and students study; teachers teach and value students’ feedback; and teachers and students co-create. The prerequisites for effective co-creation teachers identified were: Teachers must be open to involve students and create dialogues; students must be motivated and have good communication skills; the organisation must be supportive; and teachers should have the final say. Conclusion: We conclude that teachers’ conceptions are consistent with Bovill and Bulley’s levels of student participation. Under certain conditions, teachers are willing to co-create and reach the highest levels of student participation.

Background Asian educators have struggled to implement problem-based learning (PBL) because students rarely discuss their work actively and are not sufficiently engaged in self-directed learning. Supplementing PBL with additional e-learning, i.e. ‘blended’ PBL (bPBL), could stimulate students’ learning process. Methods We investigated the effects of bPBL on tutorial group functioning (discussion, self-efficacy, self-directed learning, active participation, and tutor’s perceived authority) and students’ level of acceptance of the e-learning elements. We compared PBL and bPBL in a medical university in Japan. In the bPBL condition, the tutor’s instructions were replaced with online materials and short quizzes. After the course, a 13-item questionnaire using a 5-point Likert scale was distributed regarding the tutorial group functioning of the tutorial group (influence of discussion, self-efficacy, self-directed learning, active participation, and tutors’ authority). The mean scores of subscales were compared with analysis of covariance. Knowledge levels were measured using a pre-test post-test design. A multiple regression analysis was performed to explore the association between e-learning acceptance and the subscales related to PBL. Results Ninety-six students participated in the study (PBL: n = 24, bPBL: n = 72). Self-efficacy and motivation for learning triggered by group discussions was significantly higher for students in bPBL (p = 0.032 and 0.007, respectively). Knowledge gain in test scores was also significantly better in the bPBL condition (p = 0.026), and self-directed learning related positively to the acceptance of blended learning (p = 0.044). Conclusions bPBL seemed more effective in promoting active learning and improving knowledge, without affecting tutors’ authority. Implementing e-learning into PBL is suggested to be an effective strategy in the Asian context.

There is a growing belief that higher education institutions should nurture a 'quality culture' in which structural/managerial and cultural/psychological elements act in synergy to continuously improve education. Notwithstanding the positive connotation of the 'quality culture' concept, its exact configuration remains subject to debate. A realist review was conducted to identify inhibiting and promoting organisational context elements impacting quality culture, its working mechanisms and associated outcomes. Leadership and communication were identified as being of key importance in binding structural/managerial and cultural/psychological elements. Leaders are central 'drivers' of quality culture development through their ability to influence resource allocation, clarify roles and responsibilities, create partnerships and optimise people and process management. Adequate communication is considered a prerequisite to diffuse quality strategies and policies, evaluate results and identify staff values and beliefs. It is proposed that the working mechanisms of quality culture comprise increased staff commitment, shared ownership, empowerment and knowledge. Associated outcomes related to these mechanisms are positive effects on staff and student satisfaction, continuous improvement of the teaching-learning process and student and teacher learning and development. Institutions striving for the development of a quality culture should best operate from a contingency approach, i.e. make use of quality management intervention approaches which are tailored to the organisational context.

Purpose To explore (1) whether an instructional model based on principles of cognitive apprenticeship fits with the practice of experienced clinical teachers and (2) which factors influence clinical teaching during clerkships from an environmental, teacher, and student level as perceived by the clinical teachers themselves. The model was designed to apply directly to teaching behaviors of clinical teachers and consists of three phases, advocating teaching behaviors such as modeling, creating a safe learning environment, coaching, knowledge articulation, and exploration. Method A purposive sample of 17 experienced clinical teachers from five different disciplines and four different teaching hospitals took part in semistructured individual interviews. Two researchers independently performed a thematic analysis of the interview transcripts. Coding was discussed within the research team until consensus was reached. Results All participants recognized the theoretical model as a structured picture of the practice of teaching activities during both regular and senior clerkships. According to participants, modeling and creating a safe learning environment were fundamental to the learning process of both regular and senior clerkship students. Division of teaching responsibilities, longer rotations, and proactive behavior of teachers and students ensured that teachers were able to apply all steps in the model. Conclusions The theoretical model can offer valuable guidance in structuring clinical teaching activities and offers suggestions for the design of effective clerkships.

Purpose To assess the impact on full-time faculty's own clinical skills and practices of sustained clinical skills bedside teaching with preclerkship students. Method This was a longitudinal, qualitative study of faculty who provide dedicated ongoing bedside clinical skills teaching for preclerkship medical students. Interviews were conducted during 2003 to 2007 with 31 faculty of the Colleges program at University of Washington School of Medicine. Content analyses of interview transcripts were performed. Results Teachers perceived a strong positive impact of teaching on their own clinical skills. Six themes were associated with the influence of bedside teaching on teachers' skills and practices. One related to deterrents to change (e. g., reliance on tests/specialists) that narrowed teachers' practice skills prior to starting bedside teaching. Three related to expansion of the process of clinical care resulting from bedside teaching: expanded knowledge and skills, deconstructing the clinical experience (e. g., deepening, broadening, slowing one's practice), and greater self-reflection (e. g., awareness of being a role model). Two were perceived outcomes: improved clinical skills (e. g., physical examination) and more mindful practices (e. g., self-confidence, patient-centered). Conclusions Teachers perceived profound positive impact on their clinical skills from teaching preclerkship students at the bedside. Further studies are needed, including comparing teaching preclerkship students with teaching advanced students and residents, to assess whether teaching at other levels has this effect.

Purpose Clinical teaching's importance in the medical curriculum has led to increased interest in its evaluation. Instruments for evaluating clinical teaching must be theory based, reliable, and valid. The Maastricht Clinical Teaching Questionnaire (MCTQ), based on the theoretical constructs of cognitive apprenticeship, elicits evaluations of individual clinical teachers' performance at the workplace. The authors investigated its construct validity and reliability, and they used the underlying factors to test a causal model representing effective clinical teaching. Method Between March 2007 and December 2008, the authors asked students who had completed clerkship rotations in different departments of two teaching hospitals to use the MCTQ to evaluate their clinical teachers. To establish construct validity, the authors performed a confirmatory factor analysis of the evaluation data, and they estimated reliability by calculating the generalizability coefficient and standard error measurement. Finally, to test a model of the factors, they fitted a structural linear model to the data. Results Confirmatory factor analysis yielded a five-factor model which fit the data well. Generalizability studies indicated that 7 to 10 student ratings can produce reliable ratings of individual teachers. The hypothesized structural linear model underlined the central roles played by modeling and coaching (mediated by articulation). Conclusions The MCTQ is a valid and reliable evaluation instrument, thereby demonstrating the usefulness of the cognitive apprenticeship concept for clinical teaching during clerkships. Furthermore, a valuable model of clinical teaching emerged, highlighting modeling, coaching, and stimulating students' articulation and exploration as crucial to effective teaching at the clinical workplace.

Objectives: Students in Problem-Based Learning (PBL) are assumed to adopt a deep learning approach and not a surface approach. This study investigated: 1) the reliability and validity of version of the Revised Study Process Questionnaire adapted to the PBL context (PBL-R-SPQ) and 2) the extent to which PBL students use deep or surface approaches, and whether this differs between first and second year students. Methods: The items of the R-SPQ were reformulated to better fit with a PBL environment, resulting in the PBL-RSPQ. In total 262 students from Maastricht Medical School responded to the PBL-R-SPQ. Results: A Confirmatory Factor Analysis (CFA) demonstrated that a 9-item Deep Approach scale and a 9-item Surface Approach scale fitted the observed set of data well. Cronbach alphas for the Deep and Surface scales were 0.76 and 0.74, respectively. First year students reported signifi

Learning in clinical practice can be characterised as situated learning because students learn by performing tasks and solving problems in an environment that reflects the multiple ways in which their knowledge will be put to use in their future professional practice. Collins et al. introduced cognitive apprenticeship as an instructional model for situated learning comprising six teaching methods to support learning: modelling, coaching, scaffolding, articulation, reflection and exploration. Another factor that is looked upon as conducive to learning in clinical practice is a positive learning climate. We explored students' experiences regarding the learning climate and whether the cognitive apprenticeship model fits students' experiences during clinical training. In focus group interviews, three groups of 6th-year medical students (N = 21) discussed vignettes rep- resenting the six teaching methods and the learning climate to explore the perceived occurrence of the teaching methods, related problems and possibilities for improvement. The students had experienced all six teaching methods during their clerkships. Modelling, coaching, and articulation were predominant, while scaffolding, reflection, and exploration were mainly experienced during longer clerkships and with one clinical teacher. The main problem was variability in usage of the methods, which was attributed to teachers' lack of time and formal training. The students proposed several ways to improve the application of the teaching methods. The results suggest that the cognitive apprenticeship model is a useful model for teaching strategies in undergraduate clinical training and a valuable basis for evaluation, feedback, self-assessment and faculty development of clinical teachers.

CONTEXT Many undergraduate medical education programmes offer integrated multi-disciplinary courses, which are generally developed by a team of teachers from different disciplines. Research has shown that multi-disciplinary teams may encounter problems, which can be detrimental to productive co-operation, which in turn may diminish educational quality. Because we expected that charting these problems might yield suggestions for addressing them, we examined the relationships between team diversity, team processes and course quality. METHODS We administered a questionnaire to participants from 21 interdisciplinary teams from 1 Dutch and 1 German medical school, both of which were reforming their curriculum. An adapted questionnaire on team learning behaviours, which had been validated in business contexts, was used to collect data on team processes, team learning behaviours and diversity within teams. We examined the relationship between the team factors and educational quality measures of the courses designed by the teams. RESULTS A total of 84 teachers (60%) completed the questionnaire. Bivariate correlation analysis showed that several aspects of diversity, conflict, working climate and learning behaviour were correlated with course quality. CONCLUSIONS The negative effects of the diversity measures, notably, value diversity, on other team processes and course quality and the positive association between psychological safety and team learning suggest that educational quality might be improved by enhancing the functioning of multi-disciplinary teams responsible for course development. The relationship between team processes and educational quality should be studied among larger study populations. Student ratings should also be considered in measuring educational quality.

This article addresses the choice of the most appropriate procedure for the assessment of portfolios used in teacher and lecturer assessment. A characteristic of modern assessment modes, including portfolios, is that the information they provide is often qualitative and derived from different contexts. Unambiguous, objective rating of portfolios is difficult to achieve, because the richness and uniqueness of the contents of the portfolio necessitate interpretation and taking account of the context before judgement can be passed. From this it follows that the traditional approaches and criteria for evaluating quality are no longer appropriate. We advocate the use of a hermeneutic, interpretative approach to the judging of portfolios as well as the use of quality criteria to establish trustworthiness in performing and monitoring portfolio assessment. A constructivist approach aimed at generating maximum diversity in interpretations and debate on ideas by the different stakeholders is proposed. Finally, we o...

Aim: Tutor performance and tutorial group productivity interact with each other in a complex manner. The aim of this study was to investigate how tutor performance, tutorial group productivity and the effectiveness of a PBL unit interact with each other. Three hypotheses were tested: (1) Does the tutor performance score differ across different levels of group productivity? (2) Does the group productivity score differ across different levels of tutor performance? and (3) Is the learning effectiveness score of a PBL unit related to tutor performance and group productivity? Method: Students rated the tutor performance, the tutorial group productivity and the effectiveness of the PBL unit. In total 287 unique tutors were involved and were categorized as having a relatively low, average or relatively high score on tutor performance. This was also done for the group productivity score. For each combination, average effectiveness score were computed. Furthermore, partial correlation coefficients were computed. Results: The results demonstrated that the average tutor performance score was higher if the productivity score was higher (hypothesis 1). The results also demonstrated that the average productivity score was higher if the tutor performance score was higher (hypothesis 2). Furthermore it was found that the effectiveness score was higher if the productivity score was higher. The effectiveness score was also higher if the tutor score was higher. The results furthermore demonstrated that the correlation coefficient between tutor performance and effectiveness of the PBL unit is 0.01 (n.s.) if the correlation between the two variables was controlled for group productivity (was 0.36). In addition, the correlation coefficient between group productivity and effectiveness became 0.39 if the correlation between these two variables was controlled for tutor performance (was 0.51) (hypothesis 3). Conclusion: It is concluded that tutor performance differs across different levels of group productivity and that the group productivity score differs across different levels of tutor performance. In addition, it is concluded that both group productivity and the tutor’s performance have an impact on the effectiveness of a PBL unit, although the correlation between group productivity and effectiveness is higher than the correlation between tutor performance and effectiveness. This finding illustrates the importance of the group’s productivity in PBL and the tutor’s importance and implies that schools should put more efforts in improving tutorial group functioning as well as in improving a tutor’s performance.

Context Problem-based learning (PBL) is widely used in higher education. There is evidence available that students and faculty are highly satisfied with PBL. Nevertheless, in educational practice problems are often encountered, such as tutors who are too directive, problems that are too well-structured, and dysfunctional tutorial groups. Purpose The aim of this paper is to demonstrate that PBL has the potential to prepare students more effectively for future learning because it is based on four modern insights into learning: constructive, self-directed, collaborative and contextual. These four learning principles are described and it is explained how they apply to PBL. In addition, available research is reviewed and the current debate in research on PBL is described. Discussion It is argued that problems encountered in educational practice usually stem from poor implementation of PBL. In many cases the way in which PBL is implemented is not consistent with the current insights on learning. Furthermore, it is argued that research on PBL should contribute towards a better understanding of why and how the concepts of constructive, self-directed, collaborative and contextual learning work or do not work and under what circumstances. Examples of studies are given to illustrate this issue.

In higher education, approaches to teaching are becoming more student-centred, which demands different teaching competencies. Therefore, it is necessary to have an adequate framework of teaching competencies that can be used for evaluation purposes. The weaknesses of the existing frameworks are that they do not pay attention to the person as teacher, they are too narrowly defined, they are not validated and they are not adjusted to modern approaches to teaching. The aim of this study was to develop and validate a framework of teaching competencies in higher education. A framework for teaching competencies was therefore constructed containing the following domains: The Person as Teacher, Expert on Content Knowledge, Facilitator of Learning Processes, Organiser and Scholar/Lifelong Learner. The framework was validated using a Delphi method. Educational experts (N = 63) were asked: ``How important are the following teaching competencies in each domain for an experienced teacher in higher education?'' A confirmatory factor analysis was conducted to assess the adequacy of the domains. After two rounds, the shift in ratings was minimal, so the results were considered stable. The response rate was 82%. From the original list with 134 items, 30 items were omitted because many experts failed to respond to them. The experts reached consensus on 61 out of the resulting 104 items (59%). A confirmatory factor analysis on the three best scoring items in each domain confirmed the model. A framework of teaching competencies was developed and validated that can be used as a starting point for teacher evaluation in higher education.

This article reports the design of a prototype for a teaching portfolio for development and assessment purposes. A portfolio prototype was developed according to theoretical choices based on portfolio literature and Zeichner and Wray's (2001) conceptual framework. We conducted an interview study in which portfolio experts were asked to judge to what extent our theoretical choices were reflected in the prototype developed and which modifications were needed. In general, the experts agreed that our choices were reflected in the prototype developed. However, there were some controversial topics, which are discussed, as well as key issues for further research.

This study investigated whether supervision, patient mix, and numbers of students influence the effectiveness of clinical rotations.The authors administered a questionnaire to 1,208 medical students in 1999 to evaluate the rotations' quality. They computed four variables--overall effectiveness, supervision, patient mix, and number of students contemporaneously involved--and analyzed the data using analysis of variance.Supervision and patient mix positively influenced a rotation's effectiveness. A higher level of supervision and a higher level of patient mix led to higher overall effectiveness scores. The number of students did not significantly influence the effectiveness score, although a low number led to a higher score as compared with a high number. Furthermore, the two-way interaction between supervision and patient mix was significant. Supervision more strongly influenced the rotation's effectiveness when patient mix was limited than when it was high. Also, when the patient mix was limited and supervision was high, the average effectiveness score was still 7.7 (scale 1-10).The effectiveness of clinical rotations depends on the supervision provided and patient mix, but not the number of students. Furthermore, supervision more strongly influences effectiveness when patient mix is limited than when it is high. Finally, high-quality supervision guarantees at least a sufficient score for the rotation's effectiveness, irrespective of the level of patient mix.

The tutor role in problem-based learning (PBL) has attracted the interest of many researchers and has led to an abundance of studies. This article reports on major trends in studies investigating the tutor during the past 10 years. Three major trends were observed by the authors while analysing the studies conducted: studies on the differential influence of content expert and non-content expert tutors on student achievement, studies on process variables, and studies on the relationship between tutor characteristics and differential contextual circumstances. The aim of this article is to summarize the main findings of the studies conducted so far within the three trends observed, to provide directions for educational practitioners and policy makers, and to suggest directions for future research questions. The studies included were selected by conducting a literature search in medical journals, which was complemented with the personal archives of the authors. The results of the studies conducted within the three trends of research have led to advanced insights in tutoring. The outcomes revealed that content expert tutors tend to use their subject-matter expertise more to direct the discussion in the tutorial group, whereas non-content expert tutors tend to use their process-facilitation expertise more to direct the tutorial group. Furthermore, a tutor's performance is not a stable characteristic but is partly situation specific. It is concluded that a tutor should both know how to deal with the subject matter expertise and should know how to facilitate the learning process. Faculty and policy makers should put substantial efforts into designing curricula and cases and developing tutors' skills by faculty development strategies that stimulate reflection. The research agenda should be driven more by modern educational theories of learning in which tutoring is a process aimed at stimulating constructive, self-directed, situated and collaborative learning by students. Furthermore, more qualitative studies should be conducted to gain better insights in teachers' conceptions about the tutor role and student learning to better understand their behaviours.

Background Teachers play an important role in seminars as facilitators and content experts. However, contextual factors like students’ preparation, group size, group interaction, and content appear to influence their performance. Understanding the impact of these contextual factors on students’ perception of teaching performance may help to further understand seminar teaching. Besides that, it may help curriculum organisers and teachers to get more insight in how to optimise their versatile role in seminars. The aim of this study is to investigate how students’ perception of teaching performance in seminars is explained by students’ extent of preparation, seminar group size, group interaction, and content. Methods The Utrecht Seminar Evaluation (USEME) questionnaire was used to collect information on teaching performance and the aforementioned explanatory variables. To account for intra-student, intra-seminar, and intra-teacher correlation in the data, multilevel regression was used to analyse 988 completed questionnaires in 80 seminars with 36 different teachers. Results Group interaction and seminar content had large (B = 0.418) and medium (B = 0.212) positive effects on perceived teaching performance scores, whereas the effects of students’ preparation (B = -0.055) and group size (B = -0.130) were small and negative. Conclusions This study provides curriculum organisers and teachers indications on how to optimise variables that influence perceived teaching performance in seminars. It is suggested that teachers should search for the most appropriate combination of motivating and challenging content and facilitation method within seminars to optimise discussion opportunities between students. Electronic supplementary material The online version of this article (doi:10.1186/1472-6920-14-203) contains supplementary material, which is available to authorized users.

Introduction Problem-based learning (PBL) has gained a foothold within many schools in higher education as a response to the problems faced within traditional education. Discussion Working with PBL tutorial groups is assumed to have positive effects on student learning. Several studies provide empirical evidence that PBL stimulates cognitive effects and leads to restructuring of knowledge and enhanced intrinsic interest in the subject matter. However, staff members do not always experience the positive effects of group work which they had hoped for. When confronted with problems in group work, such as students who only maintain an appearance of being actively involved and students who let others do the work, teachers all too often implement solutions which can be characterized as teacherdirected rather than student-directed. Teachers tend to choose solutions which are familiar from their own experience during professional training, i.e. using the teacher-directed model. These solutions are not effective in improving group work and the negative experiences persist. Conclusion It is argued that teachers should hold on to the underlying educational philosophy when solving problems arising from group work in PBL, by choosing actions which are consistent with the student-directed view of education in PBL.

The aim of this study was to identify the relationship between elements that are important for the tutorial group process and the individual learning process in a problem-based curriculum. The variables under investigation were student-generated learning issues, individual learning process, reporting in the tutorial group, and achievement.

Due to changes in the position of healthcare workers, greater emphasis should be placed on the assessment of the humanistic qualities of professional behaviour. To achieve this, students must receive feedback on their professional behaviour in different situations relevant to future practice. In the Maastricht Medical School it means that from the first year till the clerkships professional behaviour of students is assessed experimentally for judgements and feedback in different educational settings. The criteria of those judgements are discussed in this paper. Unprofessional behaviour during clerkships is reported to the academic adviser or to the examination committee or both. The academic adviser offers support to find solutions for the student when needed. The examination committee judges the severity of the problem and will act accordingly. Good collaboration between the academic adviser and the examination committee is necessary to cope with the problem of inadequate professional behaviour.

Objectives The aim of this study was twofold. The first question concerns the way students make use of the learning issues they generate (as strict guidelines or as global guidelines) and whether this changes across years of training. The second question concerned the relationship between the way students make use of learning issues and the time spent on individual study and achievement on two tests of knowledge. Design A questionnaire was developed, containing seven items that measured to what extent students study strictly according to the student-generated learning issues and six items that measured to what extent students study beyond the student-generated learning issues. The questionnaire also contained one question in which students had to estimate the mean time spent on individual study. Achievement was measured by two forms of tests of knowledge, a block test assessing course content and a progress test assessing long-term functional knowledge. Setting Medical School of Maastricht University, the Netherlands. Subjects Medical students (response=69%) from the problem-based curriculum at the Maastricht University. Results During their first year students study strictly according to the content of the learning issues, whereas in later years students studied more according to their own learning needs and interests. In addition, students who tended to study beyond the generated learning issues spent more time on individual study and achieved better on both tests. Conclusions Students in a problem-based curriculum seem to become better self-directed learners during the years of training.

Background: Many studies have been conducted on tutor performance in problem-based curricula. In the past, the implicit assumption behind these studies was that tutor performance is a relatively stable characteristic. More recent studies demonstrate that a tutor's performance may be dependent on other circumstances, such as the level of structure in the curricular materials. The aim of this study was to investigate whether a tutor's performance is also dependent on the tutorial group's productivity. Purpose: The idea is that low-productive tutorial groups require much more input from a tutor than high-productive groups. In the problem-based curriculum under investigation, most tutors guide 2 tutorial groups within the same unit. A salient finding in this problem-based curriculum was that some tutors who guide 2 tutorial groups within the same unit have discrepancies in their tutor performance across the 2 groups. This finding might be explained by differences in both tutorial groups. In this study, first ...

Background Many medical schools have embraced small group learning methods in their undergraduate curricula. Given increasing financial constraints on universities, active learning groups like seminars (with 25 students a group) are gaining popularity. To enhance the understanding of seminar learning and to determine how seminar learning can be optimised it is important to investigate stakeholders’ views. In this study, we qualitatively explored the views of teachers on aspects affecting seminar learning. Methods Twenty-four teachers with experience in facilitating seminars in a three-year bachelor curriculum participated in semi-structured focus group interviews. Three focus groups met twice with an interval of two weeks led by one moderator. Sessions were audio taped, transcribed verbatim and independently coded by two researchers using thematic analysis. An iterative process of data reduction resulted in emerging aspects that influence seminar learning. Results Teachers identified seven key aspects affecting seminar learning: the seminar teacher, students, preparation, group functioning, seminar goals and content, course coherence and schedule and facilities. Important components of these aspects were: the teachers’ role in developing seminars (‘ownership’), the amount and quality of preparation materials, a non-threatening learning climate, continuity of group composition, suitability of subjects for seminar teaching, the number and quality of seminar questions, and alignment of different course activities. Conclusions The results of this study contribute to the unravelling of the ‘the black box’ of seminar learning. Suggestions for ways to optimise active learning in seminars are made regarding curriculum development, seminar content, quality assurance and faculty development.

Whole-task models of learning and instructional design, such as problem-based learning, are nowadays very popular. Schools regularly encounter large problems when they implement whole-task curricula. The main aim of this article is to provide 12 tips that may help to make the implementation of a whole-task curriculum successful. Implementing whole-task curricula fails when the implementation is not well prepared. Requirements that must be met to make the implementation of whole task models into a success are described as twelve tips. The tips are organized in four clusters and refer to (1) the infrastructure, (2) the teachers, (3) the students, and (4) the management of the educational organization. Finally, the presented framework will be critically discussed and the importance of shared values and a change of culture is emphasized.

PURPOSE The aim of this study was to investigate to what extent ratings of tutor performance remain stable in the long term. At many schools, teaching performance is assessed and these evaluations are consulted as part of the decision-making process for promotion, tenure, and salary. Since this information may have summative value, it is crucial that the reliability of the data be assessed. A previous study had shown that a single evaluation of a tutor is reliable when the responses of six students are used (interrater reliability). The present study focused on the stability of tutor evaluations over repeated occasions of evaluation. METHOD A generalizability study was conducted to estimate the number of occasions required to demonstrate stability. The study took place during three academic years (1992-93, 1993-94, and 1994-95) at the problem-based medical school of the University of Limburg (now Maastricht University). A total of 291 ratings were analyzed (97 tutors rated during three sequential tutoring occasions). Two types of scores were used: an aggregate score calculated from ratings of 13 items and an overall judgment. RESULTS The results indicate that when the scores are used to interpret the precision of individual scores, two evaluation occasions should be available for the overall judgment and four occasions for the aggregate score. If the tutor scores are consulted only to determine whether performances are above or below a cutoff score, a reliable decision can be made after only a single occasion of evaluation. CONCLUSION The results demonstrate that data collected over an extended period of time can be reliably used as part of the decision-making process for promotion, salary, and tenure.

To test whether there are effects of tutor expertise on student performance under conditions of curricular materials that have low or high levels of structure and that are poorly or well matched to students' levels of prior knowledge.The study was conducted in 1994-95 at the medical school of the University of Limburg. The data set used for analysis included 135 tutorial groups (with ten to 12 students per group), 119 tutors (each running only one group per unit), and 15 units in four curriculum years. The analysis was conducted at the level of tutorial groups since a tutor's level of expertise might differ for distinct units. Tutors were asked to judge their levels of expertise related to the cases discussed, based on which a distinction was made between expert and non-expert tutors. The degrees of structure of curricular materials and students' levels of prior knowledge were rated by the students. Using analyses of variance, students' scores on end-of-unit tests (each with about 150 true-false items) were compared for groups led by expert and non-expert tutors under conditions of low and high levels of structure and low and high levels of prior knowledge.No difference was found between the test scores of groups led by expert and non-expert tutors. The interaction effects between expertise and structure and expertise and prior knowledge also turned out to be not statistically significant.The results suggest that expert tutors do not compensate for lack of curricular structure or students' lack of prior knowledge. This finding is not consistent with that of a recent study that expert tutors do compensate for lack of structure and lack of prior knowledge. This discrepancy may be accounted for by a much smaller range within which the structuredness of the curriculum and students' levels of prior knowledge varied in the present study compared with the previous study. An implication might be that faculty should put their efforts into designing structured curricula that are well matched to students' levels of prior knowledge instead of selecting hyper-expert tutors.

An instrument has been developed to assess tutor performance in problem-based tutorial groups. This tutor evaluation questionnaire consists of 13 statements reflecting the tutor's behaviour. The statements are based on a description of the tasks set for the tutor. This study reports results on the validity and reliability of the instrument. Confirmatory factor analysis showed that a three-factor model fitted the data reasonably well. The three factors are: (1) guiding students through the learning process, (2) content knowledge input, and (3) commitment to the group's learning. Generalizability studies indicated that the rating scales provide reliable information with student responses of existing tutorial group sizes. It is concluded that the tutor evaluation questionnaire is a fairly valid and reliable instrument that can be used in staff development programmes.

The improvement of instructional effectiveness has gained considerable interest during the last few years. Within this framework, the evaluation of teachers' performance has been stimulated in order to provide individual teachers with feedback about their performance. However, only providing faculty members with feedback will not increase their effectiveness. A faculty development programme is required in which is specified: (1) which formal requirements should be satisfied to fulfil the teaching role; (2) how faculty dialogue will be organized; (3) how the information will be used in promotion and tenure decisions; and (4) what kind of remedial teaching programmes will be offered to poorly scoring teachers. The purpose of this article is to demonstrate how tutors within a problem-based curriculum can be evaluated and provided with adequate feedback about their performance in order to improve their behaviour. In addition, the focus will be on plans for action that could be undertaken with regard to the as...

Background: Preclerkship clinical-skills training has received increasing attention as a foundational preparation for clerkships. Expectations among medical students and faculty regarding the clinical skills and level of skill mastery needed for starting clerkships are unknown. Medical students, faculty teaching in the preclinical setting, and clinical clerkship faculty may have differing expectations of students entering clerkships. If students’ expectations differ from faculty expectations, students may experience anxiety. Alternately, congruent expectations among students and faculty may facilitate integrated and seamless student transitions to clerkships. Aims: To assess the congruence of expectations among preclerkship faculty, clerkship faculty, and medical students for the clinical skills and appropriate level of clinical-skills preparation needed to begin clerkships. Methods: Investigators surveyed preclinical faculty, clerkship faculty, and medical students early in their basic clerkships at a North American medical school that focuses on preclerkship clinical-skills development. Survey questions assessed expectations for the appropriate level of preparation in basic and advanced clinical skills for students entering clerkships. Results: Preclinical faculty and students had higher expectations than clerkship faculty for degree of preparation in most basic skills. Students had higher expectations than both faculty groups for advanced skills preparation. Conclusions: Preclinical faculty, clerkship faculty, and medical students appear to have different expectations of clinical-skills training needed for clerkships. As American medical schools increasingly introduce clinicalskills training prior to clerkships, more attention to alignment, communication, and integration between preclinical and clerkship faculty will be important to establish common curricular agendas and increase integration of student learning. Clarification of skills expectations may also alleviate student anxiety about clerkships and enhance their learning. Keywords: preclinical medical education; clerkships; curriculum; clinical skills; bedside teaching (Published: 6 August 2010) Citation: Medical Education Online 2010, 15: 5295 - DOI: 10.3402/meo.v15i0.5295

Objective To explore student perceptions of factors contributing to the effectiveness of discussions in the reporting phase of the problem-based learning (PBL) process, where students report and synthesise the results of self-study. Methods Forty-eight Year 1 and 2 medical students participated in 6 focus group interviews about the characteristics of effective group discussions and possible improvements. The data were analysed qualitatively in several stages. Results The analysis yielded 4 main characteristics of effective discussions: asking for, giving and receiving explanations; integrating and applying knowledge; discussing differences with regard to learning content, and guiding and monitoring the content and the group process of the discussion. Integrating and applying knowledge included structuring, relating and summarising information and providing examples from practice. Discussing different opinions included discussing a variety of literature resources and disagreements. The main learning effects mentioned by the students were retention, understanding, integration and application of knowledge. Conclusions Students have clear ideas about what promotes effective discussions during the reporting phase. Their PBL experience has provided them with some insights that are in line with theory and research on collaborative learning. Future research should examine differences between student and tutor perceptions of the quality of discussions. Introductions to PBL for students and tutors should include training in asking open but focused questions, supporting explanations with arguments and dealing with conflicts about learning content. Tutors should be trained in giving effective and personal feedback. Collaborative creation of external knowledge representations (i.e. concept maps) should be advocated, as should variety of literature resources.

Portfolios are increasingly being used to stimulate teachers' reflections. Frameworks for reflection on teaching often emphasize competencies and behaviours. However, other aspects of teacher functioning are also important, such as the teaching environment and individual teachers' beliefs, professional identity and mission. In a study among five medical school teachers, we explored how a portfolio stimulated reflections on the various aspects of teaching functioning. Outcomes of written portfolio assignments were collected and analysed to identify examples of reflections on the various aspects of teacher functioning. Examples of reflections on all aspects of teacher functioning were found, although examples of reflections on competencies were easier to find than those on beliefs, identity and mission. This study might help teachers and their trainers and coaches to recognize different aspects of teacher functioning when discussing portfolios for professional development purposes. However, further development of assignments and other methods to stimulate reflections on beliefs, identity and mission are needed. Furthermore, apart from the content of teachers' reflections, teachers' reflection processes should be researched, as well as the effects of portfolio meetings with peers and coaches.

BACKGROUND Whether teaching portfolios achieve their aim of stimulating teachers' professional development is favourably affected by the incorporation of a balanced structure and effective social interactions, such as coaching. We explored teachers' experiences with a teaching portfolio that was structured by teaching roles, portfolio assignments and conversation protocols. The related social interactions consisted of meetings with peers and personal coaches. Teachers' and coaches' opinions were sought about the benefits and the enabling and disabling factors of the portfolio structure and the social interactions involved. METHOD Five teachers and their personal coaches were individually interviewed about their experiences with the portfolio. Protocols were analysed using a top-down strategy. RESULTS The teaching roles were helpful in analysing teaching, but the assignments and the conversation protocols were too detailed and directive. The social interactions were perceived as very valuable for professional development, particularly the meetings with personal coaches. CONCLUSIONS The portfolio structure and social interactions, especially personal coaching, appeared to be helpful for teachers in analysing their teaching practice. The assignments and protocols, however, were too detailed and directive. This may be resolved by using assignments and conversation protocols more flexibly and enabling adjustment to personal learning needs. The high appreciation of personal coaching might be related to the freedom for teachers to choose their own coach. The results can be used by portfolio designers as guidelines for how to achieve a balanced structure and effective social interactions and how these might be combined to increase the benefits of working with a portfolio to teachers.

INTRODUCTION Collaborative learning, including problem-based learning (PBL), is a powerful learning method. Group interaction plays a crucial role in stimulating student learning. However, few studies on learning processes in medical education have examined group interactions. Most studies on collaboration within PBL used self-reported data rather than observational data. We investigated the following types of interactions in PBL tutorial groups: learning-oriented interactions (exploratory questioning, cumulative reasoning and handling conflicts about knowledge); procedural interactions, and irrelevant/off-task interactions. AIM The central question concerned how much time is spent on the different types of interaction during group sessions and how the types of interaction are distributed over the meeting. METHOD Four tutorial group sessions in Year 2 of the PBL undergraduate curriculum of Maastricht Medical School were videotaped and analysed. The sessions concerned the reporting phase of the PBL process. We analysed the interactions using a coding scheme distinguishing several verbal interaction types, such as questions, arguments and evaluations. RESULTS Learning-orientated interactions accounted for 80% of the interactions, with cumulative reasoning, exploratory questioning and handling conflicts about knowledge accounting for about 63%, 10% and 7% of the interactions, respectively. Exploratory questioning often preceded cumulative reasoning. Both types occurred throughout the meeting. Handling conflicts mainly occurred after the first 20 minutes. CONCLUSIONS Task involvement in the tutorial groups was high. All types of learning-orientated interactions were observed. Relatively little time was spent on exploratory questions and handling conflicts about knowledge. Problem-based learning training should pay special attention to stimulating discussion about contradictory information.

PURPOSE We investigated the influence of harsh grading by tutors on tutor performance rating by students. METHODS A total of 187 tutors assessed students' professional behaviour in tutorial groups. Students rated tutor performance after receiving their grades for professional behaviour. In addition, students were asked to indicate whether they perceived their professional behaviour grades as too positive, adequate or too negative. This was considered to reflect tutors' harshness of grading. Students also rated the quality of the feedback they received from tutors with respect to their grades. RESULTS Professional behaviour grades that students perceived as too negative, adequate or too positive were associated with tutor performance ratings of 7.4 (SD = 0.9, scale 1-10, n = 33), 7.7 (SD = 0.9, scale 1-10, n = 95) and 7.5 (SD = 0.8, scale 1-10, n = 59), respectively. Harshness of grading did not influence tutor performance ratings significantly. Tutor ratings were predicted more effectively by the quality of the feedback tutors provided on grades than by the harshness of grading. CONCLUSIONS Tutor performance ratings were not related significantly to harshness of grading. Two explanations can be given: (1) tutor performance ratings were based on rating by groups of students and (2) the percentage of tutors who rated students' professional behaviour as unsatisfactory was low. The strong relationship between tutor performance ratings and the adequacy of the feedback given by tutors suggests that the tutor performance ratings collected in this study are a valid measure of the quality of their teaching, although, for a full picture of teaching quality, more measures will be needed.

Collaborative learning attracts attention because of its potential as a powerful learning strategy. This also holds for PBL. However, group work in PBL sometimes encounters problems and the quality of interaction is not always at the desired level. The aim of the present study was to develop and validate a questionnaire to assess the quality of learning-oriented group interactions in PBL in an uncomplicated way. The questionnaire, to be completed by students involved in PBL, contained items on three group-interaction dimensions: exploratory questions, cumulative reasoning and handling conflicts. It was validated by means of confirmatory factor analysis and regression analysis, the latter to investigate the relation between the three-dimension model and the tutorial group's productivity. The factors underlying the questionnaire were confirmed by the data in a linear structural analysis of the data. The regression analysis showed that the 'exploratory questions' and 'cumulative reasoning' factors explained together 26% of the variance of the tutorial group's productivity. This study provided evidence for the validity of the questionnaire. The instrument contains tips for students and tutors to stimulate deep processing interactions in the tutorial group. Nevertheless, it seems useful to investigate the external validity of the questionnaire.

OBJECTIVE To identify essential psychomotor skills for all the medical graduates of an undergraduate programme in Pakistan. MATERIALS AND METHODS Twenty-five physicians practising in a tertiary care centre and ninety primary care physicians used a Likert's scale, ranging from "very essential" to"not required at all", to mark 99 psychomotor skills in the undergraduate medical curriculum in Pakistan. RESULTS Overall the opinions of both the groups about the essential skills matched except for a few areas. CONCLUSION This study provides baseline data about psychomotor skills that a medical graduate in developing countries should be able to perform. Further studies will be undertaken by involving other stakeholders to identify and incorporate these skills in the undergraduate medical curriculum, thereby enabling graduates to practice in all the settings in Pakistan.

An academic department of education serving the entire university and a strategic choice by the Faculty of Medicine to support educational innovation through education research are the historical cornerstones of the education research program of the University of Maastricht. Over the years, the department's initial exclusive research focus on the evaluation of problem-based learning has widened to include theory-based applied research covering the broad domain of education. The program focuses on themes: the learning of students and teachers, characteristics of powerful learning environments, and assessment and evaluation of learning and teaching. Although modest in terms of resources, the program is firmly anchored within the Faculty's organizational structure. Educational relevance and professional alignment are the most prominent determinants of the success of the program. These features sustain the institutional mission of educational excellence as well as the high ranking of the Faculty of Medicine's medical training program among the training programs of the Netherlands' medical schools. A break in this self-perpetuating mechanism--due either to internal politics or to staffing problems--forms the main risk factor for the continuation of the department.

Many educational institutions use instructional approaches such as problem-based learning (PBL), in which collaborative learning plays an important role. There is little research, however, that describes which factors are responsible for the success of collaboration. The purpose of this study was twofold, i.e. to explore cognitive interactions taking place between students in tutorial groups and to examine whether the coding system of Van Boxtel (2000) is usable to analyze these interactions. The focus was on elaborations and co-constructions, which are indicators of individual and collaborative knowledge construction in a group. Videotapes of three PBL sessions were transcribed, in which tutorial groups of the Maastricht Medical School were discussing a problem. The results showed that cognitive interactions could be found in the tutorial groups and that it was possible to analyze them. Co-constructions seemed most easy to elicit from the transcripts.

Background In problem-based learning (PBL), problems represent the starting point of students' learning activities. Therefore, the quality of these problems should be high, in that they should be of an adequate level of complexity and structuredness. Previous research has proposed several guidelines for constructing problems, but some of them are rather vague and are not based on empirical evidence. The present study aimed to validate a short questionnaire that can be used to assess the degree of complexity and structuredness of PBL problems. Method This paper outlines Jonassen's theory, on which the questionnaire is based, and its relationship and applicability to PBL problems. The questionnaire was validated by means of confirmatory factor analysis. Results The results showed that students were able to distinguish PBL problems that were too simple and those that were too well-structured, but found it difficult to distinguish problems that were too complex or too ill-structured. Conclusion The questionnaire may be used to measure the levels of complexity and structuredness of a problem as perceived by students and can provide teachers with feedback about the quality of problems.