Search

This article is about differences between, and the adequacy of, response rates to online and paper-based course and teaching evaluation surveys. Its aim is to provide practical guidance on these matters. The first part of the article gives an overview of online surveying in general, a review of data relating to survey response rates and practical advice to help boost response rates. The second part of the article discusses when a response rate may be considered large enough for the survey data to provide adequate evidence for accountability and improvement purposes. The article ends with suggestions for improving the effectiveness of evaluation strategy. These suggestions are: to seek to obtain the highest response rates possible to all surveys; to take account of probable effects of survey design and methods on the feedback obtained when interpreting that feedback; and to enhance this action by making use of data derived from multiple methods of gathering feedback. (Contains 3 tables.)

Despite widespread recognition of the need to improve assessment in higher education, assessment tasks in individual courses are too often dominated by conventional methods. While changing assessment depends on many factors, improvements to assessment ultimately depend on the decisions and actions of individual educators. This paper considers research within the "heuristics and biases" tradition in the field of decision-making and judgement which has identified unconscious factors with the potential to limit capacity for such change. The paper focuses on issues that may compromise the process of improving assessment by supporting a reluctance to change existing tasks, by limiting the time allocated to develop alternative assessment tasks, by underestimating the degree of change needed or by an unwarranted overconfidence in assessment design decisions. The paper proposes countering these unconscious limitations to change by requiring justification for changing, or not changing, assessment tasks, and by informal and formal peer review of assessment task design. Finally, an agenda for research on heuristics and biases in assessment design is suggested in order to establish their presence and help counter their influence.

A review of articles published in "Assessment and Evaluation in Higher Education," over the last eight years (2006-2013) on assessment in higher education, since the introduction of the Bologna process, is the subject of the paper. The first part discusses the key issue of assessment in higher education and the method used for selecting articles. The second part presents results according to the main emerging themes arising from data analysis: assessment methods, modes of assessment and assessment related to a given teaching and learning method. The paper concludes that the foci of the studies are aligned with assessment practices other than the written test, in accordance with a learner-oriented perspective. Although the implementation of the Bologna process has had different kinds of impact in different European countries, the review shows that the use and effects of a diversity of assessment methods in higher education have been investigated, particularly those pointing to the so-called alternative methods. Implications of the findings are discussed.

An increasingly regulated higher education sector is renewing its attention to those activities referred to as "moderation" in its efforts to ensure that judgements of student achievement are based on appropriate standards. Moderation practices conducted throughout the assessment process can result in purposes identified as equity, justification, accountability and community building. This paper draws on the limited studies of moderation and wider relevant research on judgement, standards and professional learning to test commonly used moderation practices against these identified purposes. The paper concludes with recommendations for maximising the potential of moderation practices to establish and maintain achievement standards.

As higher education systems and institutions expand, more energy is being invested in ensuring that sufficient learning has been achieved to warrant the award of a qualification. Many commonly used assessment approaches do not scale well, and there remains a pressing need for reform. This paper distils insights from international investigations of student learning outcomes assessment, using this analysis to chart frontiers for innovation. This paper sets out principles for guiding change in this field, presents an evaluation of progress via a review of signature assessment initiatives, reviews likely facilitators and blockers and, through these analyses, derives a strategy for spurring development.

This paper reviews the literature about peer and self-assessment in university courses from the point of view of their use, and the suitability of their use, in the first year of university study. The paper is divided into three parts. The first part argues that although first-year students are involved in many of the studies that report on the use of peer and self-assessment in higher education, the proportion of these studies that do so is somewhat less than in other year levels. In addition, relatively little of this work directly and explicitly discusses the suitability of peer and self-assessment for students and courses at this year level. The second part of the paper provides an introductory exploration of the relationship between peer and self-assessment, and specific features of first-year assessment, learning and teaching. Three issues relating directly to the suitability of peer and self-assessment in the first year are explored. In the third part, the paper briefly discusses the desirability of implementing peer and self-assessment, in general, before seeking to extend this specifically to the first year. The paper concludes by recommending that greater use can and should be made of peer and self-assessment in the first year of university study. [ABSTRACT FROM AUTHOR]

Higher education institutions across Europe are currently involved in a major process of reforming and restructuring as part of the Bologna process which stresses the role of competences and outcomes in curriculum design. This paper reports on the findings of a research project whose purpose was to assess the clarity and the appropriate calibrations of oral production learning outcomes specifically developed for students of engineering and architecture in higher education. Two studies to evaluate the learning outcome descriptors in terms of clarity and calibration to levels of the common framework for language were carried out on three groups of engineering students and another on a group of 13 experienced educators. The results show that problematic learning outcomes can be classified into three types; oral production outcomes describing strategy use; outcomes describing technical activities and outcomes whose language was found to be difficult to understand by the learners, especially outcomes containing dead metaphors. This paper considers the practical implications of these findings for future research on the learning outcomes. (Contains 1 table, 1 figure, and 1 note.)

Today's teacher education programmes across the world strive to equip future teachers with the high-quality knowledge, skills and dispositions necessary to teach students. The assessment of teacher dispositions has thus become essential to cultivate those qualities. However, the current approach to disposition assessment in the United States focuses on personal characteristics and character-related dispositions and is frequently used as a sorting device to identify those who appear to be inadequately disposed to teaching. Expanding on earlier work by the author and colleagues, this paper examines the issue of whether more efforts should be made to incorporate elements to assess competence-related dispositions in conjunction with the character-related dispositions across assessment tools and, if so, how this could be accomplished. In addition, this paper will clarify some dispositional concepts and terms used interchangeably that actually differ from one another and can confuse the consistency of disposition assessment. Finally, a framework for assessing technology disposition as an example of competence-related disposition and for broadening the focus of disposition assessment is suggested.

This paper analyses the current research regarding student evaluations of courses and teaching. The article argues that student evaluations are influenced by racist, sexist and homophobic prejudices, and are biased against discipline and subject area. This paper's findings are relevant to policymakers and academics as student evaluations are undertaken in over 16,000 higher education institutions at the end of each teaching period. The article's purpose is to demonstrate to the higher education sector that the data informing student surveys is flawed and prejudiced against those being assessed. Evaluations have been shown to be heavily influenced by student demographics, the teaching academic's culture and identity, and other aspects not associated with course quality or teaching effectiveness. Evaluations also include increasingly abusive comments which are mostly directed towards women and those from marginalised groups, and subsequently make student surveys a growing cause of stress and anxiety for these academics. Yet, student evaluations are used as a measure of performance and play a role in hiring, firing and promotional decisions. Student evaluations are openly prejudiced against the sector's most underrepresented academics and they contribute to further marginalising the same groups universities declare to protect, value and are aiming to increase in their workforces. [ABSTRACT FROM AUTHOR]

Assessment of prior learning (APL) refers to the process of validating individuals' learning in a variety of contexts, representing a relatively new practice in many countries. In higher education, APL is used to receive access and credits based on skills and knowledge acquired mainly outside of formal academic settings. This paper focuses on validity of admission decisions based on this type of assessment in higher education. The study examines decisions made by higher education institutions for approximately 600 applicants who have used APL in order to receive admission to, and credits in the vocational teacher education programme in Sweden. The results are analysed and presented in relation to a validity discussion. A conclusion is that the existing practice of APL needs improvements in order to obtain validity and trustworthiness in the decisions made in relation to APL. (Contains 3 tables and 2 figures.)

Potential employers require graduates to be able to demonstrate competent teamwork skills in initiating ideas and solving problems cooperatively. Teamwork is prevalent in educational institutions and often included as a way of enriching learning and assessment. Whilst group working can provide a rich opportunity for cooperative learning, its assessment can be the cause of much anxiety amongst students. This paper examines the phenomenon of "free-riding" and explores methods of managing potential abuse. Six approaches were trialled in a UK university business school on modules of study involving assessed group work and the views of students and tutors analysed. Findings from the study indicate that students (like academics) value teamwork even when it is assessed. Any method to moderate "free-riding" is appreciated by students. (Contains 1 table.)

Graduate attributes have received considerable attention in recent years as universities seek to renew and articulate their purposes. Though many claims are made with respect to the implementation of graduate attributes, there is growing acceptance of the proposition that the strongest evidence of their achievement is their explicit embedding in assessment. While university policies espouse the assessment of graduate attributes, studies have suggested that the effectiveness of this undertaking is often limited by the influence of a range of diverse but interrelated factors. This paper discusses these influential factors and argues that the assessment of graduate attributes is a complex issue that is unlikely to be resolved unless these factors are addressed at a systemic level. (Contains 1 figure and 1 note.)

Introduction: Medical student concern that the submission of named examination scripts to examiners could cause bias initiated a study on the effect of identified and de-identified scripts on assessment outcome. Methods: Data were collected from a convenience examination sample of Year 1 (n = 88 students; n = 29 questions) and Year 2 scripts (n = 75 students; n = 27 questions). Scripts were randomized for presentation to examiners with or without identification for all of a given student's work. Assessment outcomes, by year and marking condition, were a non-normal distribution. Results: Non-parametric analysis determined that there were no systematic differences in assessment outcome under the two marking conditions (MW less than 0.05). Conclusion: We continue for a range of pedagogical reasons to present identified papers to examiners. Importantly, this study also demonstrates a research-led approach to resolving student queries. (Cotains 3 tables.)

Common measures used by academic developers, e.g. lecturer satisfaction with workshops, are necessary but insufficient indicators of impact since they tell only the first half of the story. Documenting whether positive responses translate into teaching actions that influence students is the second half of the story. In this study, the project team undertook to document this second half: the influence of a workshop for lecturers on student experience of learning in courses designed during the workshop. This paper describes the results of the study and also examines the implications--the difficulties and value of reconciling the desirable with the do-able in undertaking studies such as this. (Contains 8 tables and 3 notes.)

The development of e-learning has opened up new opportunities for innovation in assessment practices in higher education. This descriptive case study draws upon staff and student experiences of teaching and learning on a web-based Masters programme in primary health care to explore how specific features of the online environment can be exploited to promote assessment as part of learning. It begins by identifying different ways of conceptualising assessment in order to highlight the fundamental value choices facing those developing and delivering assessment systems, and then describes our own approach to assessment. In the second part of the paper we explore two key ways in which the online learning environment enables assessment to contribute to learning--through its potential to support collaborative learning, and through facilitating high quality feedback between teachers and students.

Introducing undergraduates to the peer-review process helps them understand how scientific evidence is evaluated and communicated. In a large biology course, technical writing was taught through mock peer-review. Students learned to critique original research through a multistep framework, calibrating their assessment skills with class standards. We hypothesized students' assessment of their own and their peers' writing becomes more accurate as they gain academic experience, and that novice students are more overconfident in their writing abilities. Counter to many instructors' assumptions, our results indicated that regardless of level of academic experience, students' self-assessed grades were higher than peer-assessed grades (p < 0.01). Both self- and peer-assessed grades were significantly higher than the instructor's grade (p < 0.01). Overall, the peer-review exercise was positively viewed by students. Most students found the peer-review process beneficial, and 88% reported taking it seriously, despite knowing they would receive instructor's feedback at the final stage of the writing exercise. Ultimately, the peer-review process provides an effective way to teach technical writing to undergraduates, but instructors should not assume that returning students with greater academic experience are better judges of their own performance. This study illustrates that all students are equally challenged to accurately assess the quality of their own work. [ABSTRACT FROM AUTHOR]

In this article we explore how examiners assess a thesis that includes published work. An online survey was used to gather data on approaches to assessing publication-based theses (PBTs). The respondents were 62 supervisors who had experience examining PBTs across a range of disciplines at a research-intensive university in New Zealand. Nearly half of the respondents had examined ‘hybrid’ theses with papers inserted as chapters, 41% had examined theses with publications appended and 14% had examined PhDs by publication (i.e. papers alone). Twenty-nine per cent of examiners used their own extended set of criteria to assess PBTs, 48% found them easier to assess but 26% wanted more guidance. Our analysis also indicated that 86% of the examiners were highly influenced by publications in top-ranked journals and international peer-reviewed journals. Among the concerns of the examiners was the intellectual input of the candidate in any multi-authored publication, as well as the coherence of the thesis. We recommend the need for clearer guidelines for doctoral candidates, supervisors and examiners managing PBTs. [ABSTRACT FROM PUBLISHER]

This paper explores ways in which examiners, supervisors and others interpret the concept of ‘originality’ when evaluating candidates’ achievements in the final Ph.D. examination. It is based on institutional responses to a question in a 2006 discussion paper on doctoral assessment about how universities define originality for the purpose of Ph.D. assessment and what other criteria are used to guide examiners in making judgements in the final examination. The paper also includes emerging findings from the first stages of data generation for a Ph.D. study about how examiners assess the Ph.D. This study involves viva observations and interviews with candidates, examiners, supervisors and independent chairs of vivas. Most participants, irrespective of role, discipline or field, confirmed that ‘originality’ or ‘a contribution to knowledge’ is required for a candidate to pass the Ph.D. examination, and that this criterion is interpreted in a subject-specific context by examiners. The studies suggest that the discipline or field in which the candidate’s topic is situated influences the examiners’ interpretation of ‘originality’ and their expectations of the candidate. However, ‘originality’ is not the only criterion for the Ph.D., and students are also required to demonstrate other achievements to pass the doctorate. The studies show that there is some degree of consistency among examiners in the general criteria used to evaluate the candidate’s thesis and performance in the viva. These findings are discussed in the light of the literature. [ABSTRACT FROM PUBLISHER]

Student feedback literacy has been the subject of much conceptual literature; however, relatively little intervention research has investigated how and if it can be developed. Further, no evaluation of the current empirical literature has been conducted to assess which elements of feedback literacy can be successfully improved in practice, and which elements need further investigation. This paper seeks to explore how different aspects of feedback literacy have been developed in higher education. A scoping review was conducted to address the foci, nature and success of interventions. The review found evidence that educational interventions enhanced feedback literacy in students, such as managing perceptions and attitudes, and having more confidence and agency in the feedback process. While some interventions have an effect on influencing student feedback literacy, both improved study design and intervention design are required to make the most of future feedback literacy interventions. [ABSTRACT FROM AUTHOR]

While feedback in education is important, formative feedback that leads to improvement in students' learning outcomes and emotional disposition is vital. This argument has been made repeatedly in the Western literature, both by researchers and by students participating in their research in higher education. This paper presents both affective and non-affective improvement as the most frequently occurring theme identified in a mixed-methods study conducted at a National Key University in mainland China, which investigated 232 Chinese students' experiences and expectations of feedback from their Chinese tutors. This paper, thus, concludes that mainland Chinese university students and Western researchers and students likely agree on the primary function of tutor feedback. The desire for tutor feedback to also or chiefly play a formative role appears not to be a geographically or culturally specific phenomenon. [ABSTRACT FROM AUTHOR]

Many documents are produced in evaluation processes; documentation of student evaluation can therefore be time-consuming. Nevertheless, we have little knowledge about how these documents are used. This paper is based upon analysis of documents that direct and report on internal student evaluation practice at a Norwegian university. Interviews with academics and students are used as . We analyse how student evaluation practice is described in university documents compared to what academics and students express. The aim is to explore how documents can contribute to the constitution of evaluation practice. The study shows that the university mainly requested information about how evaluation was carried out and followed up. The academics reported on challenges with evaluation practice, use and results. These reports were supposed to serve as background for educational quality reports at departments and faculty level. However, the information from programme reports was nearly absent from reports at higher organisational levels; they mainly described a well-established evaluation practice. There were misalignments between how student evaluation practice was described at different levels, in documents and in interviews. This may hinder improvements to evaluation practice and may also contribute to an understanding of student evaluation as an accountability tool more than a practice promoting learning. [ABSTRACT FROM AUTHOR]

As the parameters of the field of educational assessment have extended past testing into learning, assessment concepts have evolved and become ever more nuanced. It is frequently lamented in the English language literature that there is insufficient conformity and clarity in the way they are defined and used. This paper offers a survey of the problem over the past five decades and scrutinises conceptualisations of a number of key assessment terms. Additionally, it argues that some of these may not, or may no longer, be necessary, and recommends the phrase 'evaluation for learning' as the most suitable term for embodying the spirit of using testing for improving learning and teaching. It closes by offering suggestions for tackling the problem. [ABSTRACT FROM AUTHOR]

Constraints in resourcing and student dissatisfaction with assessment feedback mean that the effectiveness of our feedback practices has never been so important. Drawing on findings from a three-year study focused on student engagement with feedback, this paper reveals the limited extent to which effectiveness can be accurately measured and challenges many of the assumptions and beliefs about effectiveness of feedback practices. Difficulties relating to multiple purposes of feedback, its temporal nature and the capabilities of evaluators reveal that measuring effectiveness is fraught with difficulty. The paper argues that the learner is in the best position to judge the effectiveness of feedback, but may not always recognise the benefits it provides. Therefore, the pedagogic literacy of students is key to evaluation of feedback and feedback processes. [ABSTRACT FROM AUTHOR]

This paper reports on a study of student learning about collaboration and discusses the effectiveness of different forms of assessment in facilitating learning. The study was conducted in a large health and social care faculty in which all students on pre-qualifying professional programmes learn together in modules aimed at developing collaborative skills. Data about student learning were collected through interviews with 42 students and analysis of 53 students' completed assignments. The paper focuses on two questions: (1) What did students learn about collaborating in groups and about their own collaborative skills? (2) Which forms of assessment were effective in recording this learning? Interview and assignment data demonstrated that students learned about groups and group participation, about themselves in group situations and about the relevance of interprofessional learning to working collaboratively in professional practice. Module 3 (third year) assessments provided evidence of transference of learning from module to practice. Whereas learning logs, completed during the module as a form of reflective assessment, appeared to promote self-awareness about own collaborative skills, reflective essays, completed after module sessions had ended, provided more opportunities for analysis and to link theory to practice. [ABSTRACT FROM AUTHOR]

One issue universities face is the need to demonstrate excellence in postgraduate research supervision at the individual, faculty and university level. While poor supervision might become obvious over time, with grievances, withdrawals and poor completion times and rates, this paper focuses specifically on identifying and demonstrating supervisory excellence. Currently, the amount and range of evidence used to support claims of supervisory excellence tends to be limited, leaving supervisors, faculties and institutions in a position where demonstrating excellence remains difficult. This paper proposes two inter-dependent ideas which, considered together, help to redress this problem. The first is a 'map' for the collection and use of evidence of supervisory excellence. The second is a 'template' for a 'supervisory excellence report'. The 'map' details the organisational elements, uses of data, and data types which can be considered. The 'report' explains one simple and potent way to organise and present these data for multiple purposes. Together they constitute a much-needed framework for promoting and recognising excellence in the supervision of research students. [ABSTRACT FROM AUTHOR]

In higher education, student evaluation of teaching is widely used as a measure of an academic's teaching performance despite considerable disagreement as to its value. This paper begins by examining the merit of teaching evaluations with reference to the factors influencing the accuracy of the teaching evaluation process. One of the central assumptions on which student evaluation of teaching is based is that there is a relationship between student achievement and student rating of teachers. However, the findings of the majority of studies do not support this assumption. The absence of a strong link between student achievement and teaching evaluations suggests that there is scope for examining other approaches to measuring effective classroom dynamics. This paper presents such an approach based on the notion of transformational classroom leadership. [ABSTRACT FROM AUTHOR]

This paper reflects on the ongoing debate surrounding the usefulness (or otherwise) of multiple‐choice questions (MCQ) as an assessment instrument. The context is a graduate school of business in Australia where an experiment was conducted to investigate the use of assertion‐reason questions (ARQ), a sophisticated form of MCQ that aims to encourage higher‐order thinking on the part of the student. It builds on the work of Connelly (2004) which produced a quantitative analysis of the use of ARQ testing in two economics course units in a flexibly‐delivered Master of Business Administration (MBA) program. Connelly's main findings were that ARQ tests were good substitutes for the more conventional type of multiple‐choice/short‐answer type questions and, perhaps more significantly, ARQ test performance was a good predictor of student performance in essays—the assessment instrument most widely favoured as an indicator of deeper learning. The main focus of this paper is the validity of the second of these findings, analysis of questionnaire data casting some doubt over whether student performance in ARQ tests can, indeed, be looked upon as a sound indicator of deeper learning—student reactions and opinions suggesting instead that performance might have more to do with one's proficiency in the English language. [ABSTRACT FROM AUTHOR]

This paper places a discussion of assessment in higher education (HE) in the UK within the wider policy context. It argues for a critical sociological analysis to consider some of the issues, themes and debates in relation to assessment in HE at this time. Recent trends in assessment policy and practice are discussed, alongside a consideration of the purposes of assessment. It examines the moral panic around standards, especially in the context of widening participation, and moves on to discuss concerns of equity in relation to assessment. Issues of identity and power relations are central to these debates, and the paper concludes with a plea for social justice, rather than selectivity and inequality, to be prioritized. [ABSTRACT FROM AUTHOR]

The paper reports the results of a phenomenographic study on academics' conceptions of e-assessment. A cohort of 21 academics from 17 disciplines participated in semi-structured interviews exploring their experiences of using web-based technologies for formative and summative assessment purposes. Through iterative analysis of the interview transcripts, the study identified four qualitatively different ways in which academic teachers understand e-assessment. It was seen as a means of: (a) efficiently managing and streamlining the assessment process; (b) facilitating dialogue and student engagement; (c) enhancing student learning; and (d) developing (digital) identity and the community. Six interrelated dimensions of variation were also established: the benefit of e-assessment, the role of the assessing teacher, the role of the assessed student, the role of the medium, the purpose, quality and level of collaboration, and, finally, the relationship between e-assessment and teaching and learning. The results thematise how university teachers relate to technology-enabled assessment and represent incrementally expanding levels of agency within relatively recent, often hybrid assessment milieus. More importantly, the reported dimensions of variation can be utilised to inform which aspects of e-assessment warrant further attention for the improvement of formative and summative assessment design and practice. [ABSTRACT FROM AUTHOR]

This paper presents an effort to create a unified model for conducting and assessing undergraduate dissertations, shared by all disciplines involved in computer game development at a Swedish university. Computer game development includes technology-oriented disciplines as well as disciplines with aesthetical traditions. The challenge has been to create a unified process and assessment procedure while maintaining the individual disciplines’ academic standards. The unification has been achieved through the development of the Interdisciplinary Dissertation Model – a shared model where the assessment is focused on fundamental academic principles, and there is a clear separation between summative and formative assessment and well-defined interaction loops between examiners, supervisors and students. Examiners from all involved disciplines have regular meetings focusing on creating an interdisciplinary view on the summative assessment of undergraduate dissertations. The model developed has been successful in that it allows for a unified process for disciplines from fundamentally different academic traditions. The quality of the examined dissertations has been evaluated in a national evaluation. The result shows that all participating disciplines meet the requirements from their respective communities. This shows that the proposed model successfully unifies disciplines without sacrificing the quality in any discipline. [ABSTRACT FROM PUBLISHER]

This paper analyses how middle managers perform and experience their role in enacting policy in Scottish higher education institutions. The policy focus is the quality enhancement framework (QEF) for learning and teaching in higher education, which was launched in 2003. The data-set was collected between 2008 and 2010, during the evaluation of the QEF by means of focus groups with middle managers at nine Scottish institutions. The metaphor of a policy implementation staircase helps to situate middle managers’ position in enactment, and analyse their experience of the role. Despite the values of collegiality and ownership upheld by the QEF, middle managers’ accounts of their practices reveal that their position at the interface between university management and core academic activities continues to be a delicate one, marked by contradictory allegiances to institutional strategies and the concerns of academic colleagues. While emphasising middle managers’ pivotal role in the implementation of the QEF policy, the data paradoxically suggest that the systemic positioning of middle managers is more influential in shaping their role of mediation than the values of collegiality and ownership promoted by this enhancement approach to quality specific to Scotland. [ABSTRACT FROM PUBLISHER]

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors. [ABSTRACT FROM PUBLISHER]

There is a lively discussion on how to evaluate competence-based higher education in both evaluation and competence research. The instruments used are often limited to course evaluation or specific competences, taking a rather narrow perspective. Furthermore, the instruments often comprise predetermined competences that cannot be adapted to higher education needs. The aim of this paper is to provide a flexible screening instrument into which practitioners/evaluators of study programmes can themselves insert the competences they are interested in, i.e. competences students ought to acquire through their study programme. The present study describes the development of the Competence Screening Questionnaire for Higher Education (CSQ-HE) in different domains of higher education. Results show that the CSQ-HE provides plausible screening information whilst fulfilling evaluation standards. It can be integrated into a higher education institution's internal quality management system to demonstrate a programme's legitimacy and enhance competence-based teaching and learning. Strengths, weaknesses and implications for practice are discussed. [ABSTRACT FROM AUTHOR]

This case study presents a longitudinal, evidence-based approach to health science curriculum reform and evaluation. Curriculum in higher education must meet the needs of diverse stakeholders and must respond to dynamic local, national and international contexts, and this creates challenges for evaluation. The long lead time prior to the introduction of new or revised university curriculum (at least two years), the length of degree programmes (three to five years) and the lag time in the availability of objective indicators of degree outcomes (at least one year post-graduation) mean that findings may be obsolete before an evaluation can be completed. Few would argue that evaluation is important and methods and approaches for conducting curriculum evaluation are proposed in the literature, although few published reports of the outcomes of comprehensive evaluations are evident. This paper discusses potential for evaluation to establish responsive communication between students, teaching staff and programme administrators, ensuring a match between the intended, implemented and attained curriculum. [ABSTRACT FROM AUTHOR]

Group projects form a large and possibly growing component of the work undertaken for assessing students for postgraduate degrees in business. Yet the assessments sources, methods and purposes result in an array of combinations that the literature on assessment fails to capture in its full complexity. This paper builds on a new framework for discussing the assessment of group projects and reports on an exploratory piece of empirical research to examine the questions posed in the title: Is there more learning? The answer is a solid, if less than emphatic yes. Is it less fair? Probably, but what is the alternative? [ABSTRACT FROM AUTHOR]

This paper discusses the evaluation of Web-based tools to support Personal Development Planning (PDP) in order to illustrate issues associated with the introduction and assessment of the effectiveness of online resources. The aims of the evaluation were based on concerns about the very complex situation that offering online resources including PDP poses for part-time off-campus students. In this paper the authors utilize a modified RUFDATA framework (Saunders, 2000) and consider the benefits a range of methods, including surveys and observational studies, can offer evaluations of electronic resources. They found that students used and valued some aspects of the provision but that even with best efforts of instructional designers, and user testing at the draft stage, students do not always behave as intended. This evaluation highlighted the need for a range of strategies for evaluating online resources and that regardless of student attitude there are real worries about eLearning imposing extra workload burdens on 'time poor' adults. [ABSTRACT FROM AUTHOR]

This paper reports a case study that investigated the practice of peer assessment in a professional course, with particular emphasis on students' experience. It was found that peer assessment processes were beneficial to students' learning and development as professionals. The case study pointed to several conditions for effective implementation of peer assessment, which also relate to assessment in general. These conditions include: (a) adequate and appropriate preparation for the use of peer assessment; (b) alignment of assessment, learning objectives and the broader purpose of the course (e.g. preparation as professionals); (c) the availability of assistance from a teacher throughout the peer assessment process; and (d) constructive discussions following peer assessment, sensitively handled by a teacher. [ABSTRACT FROM AUTHOR]

This article explores some specific issues involved in online learning and assessment. It draws on data from a postgraduate course for professional educators, delivered globally online, and highlights the relationship between students' online discussion and their written assessed work, arguing that we need to focus on both of these in terms of the writing demands they make on students. In so doing it utilizes a theoretical framework which conceptualizes writing as contextualized social practice. The paper illustrates the complexity of the rhetorical demands being made on students in these new environments of teaching and learning and, in focusing on writing, complements present approaches to online learning which have, to date, tended towards collaborative and constructivist perspectives. The article highlights the relationship between pedagogy, technology and assessment. It concludes with a discussion of the design of an online writing resource to support student writers on this particular masters programme. [ABSTRACT FROM AUTHOR]

This paper reports on findings relating to a project on gender and essay assessment in HE. It focuses on one aspect of the study: the assessment of and feedback given to two sample essays by 50 historians based at universities in England and Wales. We found considerable variation both as to the classification awarded to the essays and to positive and negative comments made about their quality, supporting the argument that the 'quality' of a piece of writing for assessment is ultimately constructed by the reader of the essay and cannot be objectively ascertained. Gender issues emerging from the data are explored in the paper, relating to lecturers' perceptions of the essays' qualities; views concerning the way feedback should be presented; and the content and style of feedback given by lecturers on the sample essays. We found that gender constructions were manifested more in presentation than practice. These findings on the situated practice of assessment have implications for the conception of the 'reliability' of essay assessment in HE. [ABSTRACT FROM AUTHOR]

This paper presents an example of using evaluation as a tool for encouraging democratic participation in the development of professional learning and course delivery. The new professional role of Connexions Personal Adviser (CPA) is being clarified and developed through participation of CPAs on a new national training programme, the Diploma for Connexions Personal Advisers, and the approach to evaluating the programme builds on the espoused values and principles of the Connexions Service. Whilst using findings from an evaluation of a specific training programme, the paper advocates a method that could be used on other professional programmes. A theme to emerge from the evaluation has been the identification of a gap between the rhetoric of the national policy and the reality of awareness at local delivery level. These findings have been instrumental in informing future developments and research in particular into the effectiveness of approaches to action learning, and an exploration of strategies to enable the effective transfer and integration of learning from education to workplace. [ABSTRACT FROM AUTHOR]

In this paper, we present an in-depth case study of a single student who failed an online module which formed part of a master's programme in Professional Education and Leadership. We use this case study to examine assessment practices in higher education in the online environment. In taking this approach, we go against the current predilection for Big Data which has given rise to 'learning analytics', a data-intensive approach to monitoring learning. In particular, we draw attention to the model of the learner produced by learning analytics and to issues of 'dataveillance' in online learning. We also use the case to examine assessment in higher education more broadly, exploring the tensions between the requirements for certification and the need for learning. We conclude that assessment practices in higher education may have more to do with 'quality assurance' and regulatory frameworks than with 'enhancing the student experience' and inculcating the qualities that mark out higher education as an ethical project. [ABSTRACT FROM AUTHOR]

Little work is being undertaken in South Africa to systematically and intentionally prepare undergraduate students to pursue postgraduate studies. This is concerning given the shortage of postgraduate students and the small scale of postgraduate studies. The few programmes and endeavours that exist to prepare students for postgraduate studies are not necessarily evaluated to assess their achievements and shortcomings. This paper provides an evaluation of an academic year-long postgraduate preparation programme, and is specifically concerned with examining subsequent postgraduate enrolment and improvement of participants’ marks. The study draws on both quantitative and qualitative data. The findings indicate that the majority of programme participants proceeded to subsequently enrol in postgraduate studies immediately after completing the programme and that the programme played a role in the pursuit of postgraduate studies. However, the findings also indicate that overall participants’ marks did not improve after participating in the programme. The study brings to light that, while some achievements are possible, the limits of the programme must also be acknowledged. [ABSTRACT FROM PUBLISHER]

The aim of this research was to identify the factors that could be used for quality assessments of the placement centres used by the University of Barcelona’s Faculty of Education. To achieve this, a multiple case study method (bachelor’s degrees in Education, Social Education and Social Work) was used, which was based on a survey methodology. A questionnaire, which identified seven quality criteria, was applied to 291 placement centres. An analysis of the placement centres’ average quality scores revealed that most of them had high scores for consideration of the training activity, tutor–student ratio, type of placement, capacity of providing career opportunities, the centre’s tutoring, the tutors’ tasks and the coverage of the competences. We considered it necessary to improve the definition of quality placement plans, and the recognition of the tutor’s work at the centre; therefore, this paper proposes an instrument to measure and assess the quality indicators described, aiming at improving the placements system at the University by better connecting universities and centres. [ABSTRACT FROM PUBLISHER]

The impact of innovative assessment on student experience in higher education is a neglected research topic. This represents an important gap in the literature-given debate around the marketisation of higher education, international focus on student satisfaction measurement tools and political calls to put students at the heart of higher education in the UK. This paper reports on qualitative findings from a research project examining the impact of assessment preferences and familiarity on student attainment and experience. It argues that innovation is defined by the student, shaped by diverse assessment experiences and preferences, and therefore its impact is difficult to predict. It proposes that future innovations must explore assessment choice mechanisms which allow students to shape their own assessments. Cultural change and staff development will be required to achieve this. To be accepted, assessment for student experience must be viewed as a complementary layer within a complex multi-perspective model of assessment, which also embraces assessment of learning, assessment for learning and assessment for lifelong learning. Further research is required to build a meta-theory of assessment to enhance the synergies between these alternative approaches and minimise the tensions between them. [ABSTRACT FROM AUTHOR]

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact. [ABSTRACT FROM AUTHOR]

This paper describes the evaluation of a teaching and learning centre (TLC) five years after its inception at a mid-sized, midwestern state university. The mixed methods process evaluation gathered data from 209 attendees and non-attendees of the TLC from the full-time, benefit-eligible teaching faculty. Focus groups noted feelings of inclusiveness because of the sole concentration on teaching. The survey found that attendees felt more positive about the value and effectiveness of the TLC than non-attendees, and no one on campus expressed negative feelings about its effectiveness. We discuss implications for future evaluation research. [ABSTRACT FROM PUBLISHER]

In this paper, we examine workplace educators’ interpretations of their assessment practices. We draw on a critical practice lens to conceptualise assessment practice as a social, relational and situated practice that becomes critical through critique and emancipation. We conducted semi-structured interviews followed by roundtable discussions with physiotherapy clinical educators in the workplace. Employing a critical hermeneutic approach, data was scrutinised to understand relational and critical dimensions of assessment practices. Findings revealed that participants were socialised into assessment practices. They were aware of the challenges, and they grappled with the tension between their practice-based judgments and documentation of competence. Despite this tension, there was a reluctance to move their awareness of practice tensions towards emancipatory action which could relieve these tensions. We conclude that it is important to acknowledge the judgements which inform assessment practice and the tensions assessors feel between objective and subjective assessments. Assessors who deliberately engage with these tensions can emancipate their assessment practices towards a critical, wise and self-defining practice that nurtures lifelong learning in students and assessors alike. [ABSTRACT FROM PUBLISHER]

The paper reports the findings of evaluative research that attempted to rigorously assess the efficacy of a feed-forward, formative assessment intervention. The aim was to improve participants’ conceptions of quality, and hence improve the quality of a complex piece of summative assessment, by asking them to mark exemplars produced by former students. Feed-forward assessment has theoretical support in the literature, but empirical confirmation has been slight. Research findings were encouraging. A statistical model incorporating feed-forward was developed which accounted for a large effect in the improvement of results for the summative item. Importantly, there was improvement across student ability levels. Students, in the main, made accurate judgements about different levels of exemplar quality, although they had some difficulty discerning different pathways to high-quality products. Qualitative analysis indicated improved student conceptions of coherence and integration in the summative piece. [ABSTRACT FROM AUTHOR]

Despite the wide use of peer assessment, questions about the helpfulness of peer feedback are frequently raised. In particular, it is unknown whether, how and to what extent peer feedback can help solve problems in initial texts in complex writing tasks. We investigated this research gap by focusing on the case of writing literature reviews in an academic writing course. The dataset includes two drafts from 21 students, sampled to represent a wide range of document qualities, and 84 anonymous peer reviews, involving 1,289 idea units. Our study revealed that: (1) at both substance and high prose levels, drafts of all quality levels demonstrated more common problems on advanced writing issues (e.g. counter-argument); (2) peer feedback was driven by difficulty of the problem rather than overall draft quality, peer comments were not well aligned with the relative frequency of problems, more comments were given to less difficult problems; (3) peer feedback had a moderate impact on revision, and importantly, receiving multiple comments on the same issue led to more repairs and improvement of draft quality, but consistent with the comments received, authors tended to fix basic problems more often. Implications for practice and research are drawn from these findings. [ABSTRACT FROM AUTHOR]

This paper aims to inspire stakeholders working with quality of higher education (such as members of study boards, study programme directors, curriculum developers and teachers) to critically consider their evaluation methods in relation to a focus on student learning. We argue that many of the existing methods of evaluation in higher education are underpinned by a conception of learning that is de-contextualised. As a consequence, many data collection methods do not address aspects that affect students' learning. This is problematic because the core aim of higher education is to facilitate student learning. We propose a contextualised evaluation methodology, guided by 10 key questions, which can help evaluators address concepts and questions of student learning in their evaluations. [ABSTRACT FROM AUTHOR]