Search

While several test concordance tables have been published, the research underpinning such tables has rarely been examined in detail. This study aimed to survey the publically available studies or documentation underpinning the test concordance tables of the providers of four major international language tests, all accepted by the Australian Department of Home Affairs for Australian visa purposes. To evaluate the concordance studies, we first identified the good practice principles in concordance research through a review of both the relevant literature and leading professional standards in the field of educational measurement and language assessment. Next, we reviewed the concordance studies against the identified good practice principles. Our findings revealed that the information supplied by test providers varied, with some making the full research papers available, whereas others providing little information about their underpinning research. None of the concordance studies fulfilled all the good practice principles. Based on the findings of this study, we offer recommendations for future concordance research in the field of language testing as well as suggestions for practice.

The medical field is facing a physician-scientist shortage, threatening future medical research and development. Medical institutions can contribute to developing physician-scientists by stimulating students' involvement in research. In this study, a medical undergraduate teaching module to develop research skills and encourage interest in research was held in Chongqing, China. Undergraduate medical students at Chongqing Medical University completed research skills training. Before and after the training, an online, anonymous, self-reported questionnaire was administered. The self-reported questionnaire investigated students' demographic characteristics, students' perception of attitudes toward conducting research, research skills (e.g., identifying and utilizing information, critical appraisal of literature, paper writing, and understanding of research), and feedback on scientific research training. The module was implemented with 25 students from July 2019 to October 2019, and 96.0% (N = 24) of participants responded to the questionnaire both prior to and after the training. In the evaluation of research skills, results showed that use of appropriate tools for research (Z = -3.340, p < 0.01), students' ability to undertake a focused literature search (Z = -3.40, p < 0.01), identifying and utilizing information (Z = -3.34, p < 0.01), and paper-writing skills (Z = -3.49, p < 0.01) were significantly improved after the undergraduates participated in the scientific research training. A qualitative analysis of the feedback showed that students found that the training helped to enhance their knowledge, improve their study scores, and motivate them to conduct research in the future. Early scientific research ability training strengthened the research skills of medical undergraduates and motivated them to pursue research.

This paper examines the quality evaluations of the Greek Universities highlighting those which offer a full-fledged study program of primary education. There are eight principles-criteria according to which each university is evaluated. For each principle, scores may range from 1 (noncompliance with the principle) to 4 (fully compliance). I present and compare results of the Greek university evaluations completed by the Hellenic Quality Assurance and Accreditation Agency (HQA). Surprisingly, no university has been graded as non-compliant in any of the eight principles-criteria. These evaluations are performed by an alleged "external" and "independent" committee. For all practical purposes, they are, nevertheless, based on subjective and biased opinions of academics affiliated with international universities with links to Greece and its universities. Independent and evidence-based evaluations paint a different picture. Universities which get perfect scores in the quality evaluation processes perform badly in the pertinent international ranking systems. As a case study, their primary education departments have lower research performance with high variability between faculty members and departments. Given that one criterion of evaluating quality is research, then not all Greek universities can be evaluated as highly performing research institutions, either in absolute or relevant terms. This criterion of quality is not satisfied by the primary education departments of Greek universities. According to this evidence, using the HQA as an agency to assess and accredit quality is useless. It should be abolished. A new system should be based on objective criteria such as independent teaching evaluations and research performance. These do not require any committee to evaluate performance and can be constructed on evidence-based policy. The latter relies solely on rigorously established objective facts.

The quality of a research study application sends a distinct signal to the institutional review board (IRB) about the skills, capacities, preparation, communication, experience, and resources of its authors. However, efforts to research and define IRB application quality have been insufficient. Inattention to the quality of an IRB application is consequential because the application precedes IRB review, and perceptions of quality between the two may be interrelated and interdependent. Without a clear understanding of quality, IRBs do not know how to define quality and researchers do not know how to achieve quality. This position has not been systematically studied to date, and future research could provide much-needed empirical validation. This paper lays the conceptual groundwork for future investigation into what constitutes quality in an IRB application. It includes a landscape review of multidisciplinary research on quality, as well as a discussion of quality frameworks analogous to research with human participants that exist in the published literature. It also examines the background and significance of federal research regulations, regulatory burdens, researchers' regulatory literacy, and the roles and responsibilities of IRB professionals within this ecosystem.

This paper is concerned with the process of selecting between the increasingly popular bi-factor model and the second-order factor model in measurement research. It is indicated that in certain settings widely used in empirical studies, the second-order model is nested in the bi-factor model and obtained from the latter after imposing appropriate parameter constraints. These restrictions can be directly tested within the framework of the latent variable modeling methodology employing widely circulated software. The outlined model selection procedure provides a readily applied means of choosing between the two models of growing interest to measurement scholars, and is illustrated using numerical data.

Although encompassing a variety of research approaches, qualitative research in music education shares the assumption that reality is socially constructed; it takes this construction to be based on the specific perspective of the individual human; and it considers epistemology and ontology to be different fields of study. The posthuman theory of agential realism, on the other hand, argues that the world is becoming through intra-actions; it decentres the individual humanist subject; and it studies onto-epistemology -- practices of knowing-in-being. Considering these fundamental differences, the quality criteria of qualitative research are not applicable to posthuman music education research. Nevertheless, posthuman research is concerned with the ethics of research and how to response-ably and ethically take part in the world's becoming. Thus, the emerging field of posthuman music education research must develop other ways of evaluating research beyond the quality criteria found in qualitative methodologies. This paper argues that diffraction -- both as a physical and musical phenomenon and as a philosophical concept -- might be a fruitful approach. By reading quality criteria diffractively through the agential realist concept of response-ability, it poses critical and creative questions, moving us towards evaluating the quality of posthuman music education research on its own terms.

This paper provides an introduction to issues surrounding the participation rights of young people in research and the implications of their growing involvement in research as well as providing a discourse on the ethical implications related to consent. The unique contribution of this paper is that it considers children's rights in respect to the increasing opportunities for young people to take part in evaluation research. The aim of this paper, therefore, is to acknowledge the growing involvement for young people in research and the implications of ensuring that their rights of participation are respected. Secondly, we will consider the children's rights legislation and our obligations as researchers to implement this. Finally, we will explore consent as an issue in its own right as well as the practicalities of accessing participants. This paper will postulate that any research about young people should involve and prioritize at all stages of the research process; including participation in decision-making. We conclude by identifying five key principles, which we believe can help to facilitate the fulfilment of post-primary pupils' ability to consent to participate in trials and evaluative research.

Impact assessment research has developed theory-based approaches to trace the societal impact of scientific research. Impact assessment typically starts from the perspective of a research investment, organization, or project. Research users, non-academic actors involved in knowledge production, translation, and application, are well represented in many of these approaches. Researcher users are usually positioned as contributors to research, recipients of research outputs, or beneficiaries of research-driven outcomes. This paper argues that impact assessment would benefit from a more comprehensive understanding and analysis of research valorization processes from the user perspective. The first half of the paper reviews key impact assessment literature to identify how research users are positioned and portrayed in relation to valorization processes. In the second half of the paper, we use the results of this review to propose a set of principles to guide a systematic approach to constructing user perspectives on research impact. We suggest four concepts for operationalization of this approach. The paper concludes that the addition of a more comprehensive research user perspective on research valorization would complement and enhance existing impact assessment approaches.

The purpose of this research is to make a meta-evaluation of the program evaluation studies on teacher training programs in Turkey. Meta-evaluation is the process of revealing the deficiencies and errors of the research as the last stage of the program evaluation process. In this context, the steps of the meta-evaluation processes were followed methodically. In the in this research, 9 program evaluation studies conducted between 2010 and 2020 about teacher training programs, using any program evaluation approach and model, were meta-evaluated. The sampling method of the research is criterion sampling.. Research data was collected using the "Program Evaluation Standards Checklist" developed by the researchers. During the development of the relevant data collection tool, the "Program Evaluation Standards" created by the Joint Committee on Standards for Educational Evaluation (JCSEE) were benefited. The studies included in the research were examined by 6 program experts who formed the meta-evaluation team. Each expert evaluated 3 studies using the checklist. Research data was analyzed using descriptive analysis method. Research findings indicate that the examined program evaluation studies meet the program evaluation standards by 55.67%. From this point, some suggestions that are believed to contribute to future program evaluation and meta-evaluation studies on teacher training were presented. [This study was an oral presentation in "International Pegem Conference on Education", Diyarbakir, Türkiye, 16-18 September 2020.]

Although robust evidence has been accumulating over the last decades supporting the effectiveness of career counseling in various populations, yet there are scarce conceptual or empirical studies investigating the impact of distinct approaches, such as traditional-"psychometric," narrative, or a mixed (integrative) one. The primary aim of this paper is to make a contribution in the field, providing a synthetic overview of contemporary research findings regarding the effectiveness of narrative and mixed career counseling approaches with a special focus on university students. The recent trend of utilizing mixed approaches in career assessment and intervention is also emphasized in the article, suggesting they may have a great potential in helping young adults advance meaning-making about career and life design. Finally, a discussion of both benefits and challenges relating to narrative and mixed interventions along with recommendations for extending research in the area of assessing career interventions' effectiveness for university students is provided.

Purpose: The purpose of this study is to systematically review the practice of reverse mentoring and draw a timeline of the research over the past two decades. Considering the novelty of this intervention, this paper proposed an agenda for future research on this burgeoning topic. Design/methodology/approach: By adopting narrative literature review and Gregory and Denniss' (2018) four-step process, this paper reviewed 54 studies grounded in conceptual, literature review and empirical research published between 1998 and 2020. Findings: The articles included in the literature review on reverse mentoring research were summarized according to journal publications, research methodologies, contextual settings, theoretical framework, purpose and outcomes. Reverse mentoring studies are dominantly published in educational journals using primarily qualitative and conceptual approaches to explore both academic and business contexts within the USA and Europe. Theories frequently used to frame and examine the need of reverse mentoring included social exchange theory and leader-member exchange theory. The fundamental purpose of reverse mentoring research is to transfer knowledge and to bridge the technology divide between intergenerational groups. Reverse mentoring has been used to promote inclusivity between multiple generations in relation to gender, ethnicity and culture. Originality/value: As per the knowledge, this is the first-ever comprehensive English summary of reverse mentoring research done in the past two decades. Findings from this research can be used to better understand reverse mentoring research trends and directions.

There is general consensus that teaching and learning centres have a positive impact on promoting and supporting a culture that values teaching and learning, yet there is limited evidence on how centres evaluate their work. For this purpose, a survey was developed and administered to the directors of 88 Canadian centres; 46 of which completed the survey. In this paper, we share centres' approaches to evaluation that emerged from the survey findings and the artifact analysis. By raising awareness of evaluation approaches, we are better positioned to identify further opportunities to develop a more systematic and coherent approach to evaluation.

As curricular reforms are implemented, there is often urgency among scholars to swiftly evaluate curricular outcomes and establish whether desired impacts have been realized. Consequently, many evaluative studies focus on summative program outcomes without accompanying evaluations of implementation. This runs the risk of Type III errors, whereby outcome evaluations rest on unverified assumptions about the appropriate implementation of prescribed curricular activities. Such errors challenge the usefulness of the evaluative studies, casting doubt on accumulated knowledge about curricular innovations, and posing problems for educational systems working to mobilize scarce resources. Unfortunately, however, there is long-standing inattention to the evaluation of implementation in health professions education (HPE). To address this, we propose an accessible framework that provides substantive guidance for evaluative research on implementation of curricular innovations. The Prescribed-Intended-Enacted-Sustainable (PIES) framework that is articulated in this paper, introduces new concepts to HPE--with a view to facilitating more nuanced examination of the evolution of curricula as they are implemented. Critically, the framework is theoretically grounded, integrating evaluation and implementation science as well as education theory. It outlines when, how, and why evaluators need to direct attention to curricular implementation, providing guidance on how programs can map out meaningful evaluative research agendas. Ultimately, this work is intended to support evaluators and educators, seeking to design evaluation studies that provide more faithful, useful representations of the intricacies of curricular change implementation.

Increased charter school demand creates a critical need for reliable information on outcomes. Stanford's Center for Research on Education Outcomes (CREDO) has a series of influential reports on charter schools using the virtual control record (VCR) method. However, the VCR method has been criticized and the validity of CREDO's findings challenged, leading to confusion among stakeholders. In this paper, we synthesize charter school evaluation literature, explain CREDO's methods as well as other evaluation methods, and consider the limitations of matching in this context. We find that, while the matching techniques are imperfect, they provide necessary information on the greatest percentage of charter school students.

Rating scale development in the field of language assessment is often considered in dichotomous ways: It is assumed to be guided either by expert intuition or by drawing on performance data. Even though quite a few authors have argued that rating scale development is rarely so easily classifiable, this dyadic view has dominated language testing research for over a decade. In this paper we refine the dominant model of rating scale development by drawing on a corpus of 36 studies identified in a systematic review. We present a model showing the different sources of scale construct in the corpus. In the discussion, we argue that rating scale designers, just like test developers more broadly, need to start by determining the purpose of the test, the relevant policies that guide test development and score use, and the intended score use when considering the design choices available to them. These include considering the impact of such sources on the generalizability of the scores, the precision of the post-test predictions that can be made about test takers' future performances and scoring reliability. The most important contributions of the model are that it gives rating scale developers a framework to consider prior to starting scale development and validation activities.

This paper focuses upon the many functions that are served by evaluations, and by the process of conducting them. Many forms or types of evaluation have evolved to serve these different functions; and a brief account is given of some of the most common of these forms and the issues or controversies that they have engendered. The discussion opens, after a brief historical note, by describing the differing views of Lee Cronbach and Michael Scriven about whether an evaluator should aim to educate stakeholders about the program so that they can make informed decisions about it, or whether evaluators should go further and make a value judgement about it. The discussion then moves on to the importance of not overlooking the unintended effects of a program that is under study; and after presenting a list of functions that evaluations can have, the remainder of the discussion deals with the "pros" and "cons" of, and the differences between, formative and summative evaluations.

The National Institute for the Evaluation of Education (INEE) in Mexico has begun to meet the challenges in evaluating indigenous children and teachers and the educational programs and policies targeted to them. Several evaluation projects are described in this paper. One is the "Previous, Free and Informed Consultation of Indigenous People," which focuses on quality of education they receive. A second is the design of a protocol for reducing cultural and linguistic bias in standardized tests, which requires oversampling of indigenous students and the involvement of anthropologists, linguists and indigenous teachers in item development. A third is an indigenous language evaluation for candidates for entry into the teaching profession, which they must pass before they can work in indigenous schools. A fourth is the development of a qualitative instrument for evaluating teacher performance. The instrument asks evaluated teachers to contextualize their planning; scorers decide whether the plan is adapted to the cultural context and the characteristics of the children. The projects described are only a starting point. In the near future, several dilemmas, such as the apparent trade-off between contextualization and quality, have to be faced and solved.

Assessment is an integral part of the teaching-learning process in both conventional and distance education contexts. Literature suggests that with the increase in the use of Information and Communications Technology in the delivery of learning, a number of institutions are resorting to formative assessment practices that are mediated by technology to not only provide flexible and more efficient means of assessment but also attain improved learning outcomes. This paper investigated student perceptions of the effectiveness of different types of formative assessment used in online learning environments. A 31-item questionnaire was used to gather data on student perceptions. On the level of difficulty, students generally perceived the various types of formative assessment as having no significant differences. Results further indicated that students received more prompt feedback from peer assessment and computer-marked assessment, compared to teacher-marked assessment. The findings of this study will support practitioners in eLearning to use formative assessment and feedback mechanisms more effectively to influence student engagement as well as learning outcomes.

The importance of temporality in learning has been long established, but it is only recently that serious attention has begun to be paid to the precise identification, measurement, and analysis of the temporal features of learning. From 2009 to 2016, a series of temporality workshops explored temporal concepts and data types, analysis methods for exploiting temporal data, techniques for visualizing temporal information, and practical considerations for the use of temporal analyses in particular contexts of learning. Following from these efforts, this two-part Special Section serves to consolidate research working to progress conceptual, technical and practical tools for temporal analyses of learning data. In addition, in this second and final editorial we aim to make four contributions to the ongoing dialouge around temporal learning analytics to help us move towards a clearer mapping of the research space. First, the editorial presents an overview of the five papers in Part 2 of the Special Section on Temporal Analyses, highlighting the dimensions of data types, learning constructs, analysis approaches, and potential impact. Second, it draws on the fluid relationship between 'analyzed time' and 'experienced time' to highlight the need for caution and criticality in the purposes temporal analyses are mobilized to serve. Third, it offers a guide for future work in this area by outlining important questions that all temporal analyses should intentionally address. Finally, it proposes next steps learning analytics researchers and practitioners can take collectively to advance work on the use of temporal analyses to support learning.

The implementation of the national Research Engagement and Impact Assessment in Australia provides a timely opportunity to review attempts to improve the non-academic impact of academic research. The impact agenda represents a new phase in academic research evaluation and funding, characterised by a heightened need to demonstrate a return on public investments in research. New imperatives seek the reorientation of some academic research towards more directly driving national innovation, meeting the needs of business, and contributing to improved social and economic outcomes. This paper reviews the policy journey of research impact in Australia from the proposed, but never implemented, Research Quality Framework (RQF) to the National Innovation and Science Agenda (NISA). Our analysis of policy developments from the Howard to the Turnbull Governments highlights the controversial nature of research impact assessment and the political and methodological challenges that have accompanied its implementation.

The issue of connecting research evidence with practice referred to as evidence-informed practice (EIP) has received considerable critical attention in the field of education. However, assessment literature has failed to fully acknowledge the significance of this issue. This conceptual paper concentrates on the process of informing classroom assessment practices by policy-makers, researchers, and teachers. It draws on the important literature on EIP in education and its scarce literature in assessment to highlight a possible productive relationship between assessment research and practice. The paper aims to propose a set of principles as to address the tension between AfL research and practice by defining the roles of each party in the process based on cultural and contextual issues. Accordingly, a cyclical process of collaborative research conduction is proposed. This process provides policy-makers, researchers and practitioners with important implications for practicing AfL research evidence.

With the push for evidence-informed policy and practice, schools and policy makers are now increasingly encouraged and supported to use and enagage with research evidence. This means that consumers of research will now need to be discerning in judging the quality of research evidence that will inform their decisions. This paper evaluates the quality of evidence behind some well-known education programmes using examples from previous reviews of over 5,000 studies on a range of topics. It shows that much of the evidence is weak, and fundamental flaws in research are not uncommon. This is a serious problem if teaching practices and important policy decisions are made based on such flawed evidence. Lives may be damaged and opportunities missed. The aim of this paper is to show how widespread this problem is and to suggest ways by which the quality of education research may be improved. For example, funders of research and research bodies need to insist on quality research and fund only those that meet the minimum quality criteria. Journal editors and reviewers need to be cognizant of fundamental flaws in research and reject such submissions. One way to do this is to encourage submission of the research design and research protocol prior to acceptance, so acceptance or rejection is based on the design and not on the outcomes. This helps prevent publication bias and biased reporting. Individual researchers can improve quality by making it their moral responsibility to be truthful and transparent.

The acceptance of the Scholarship of Teaching and Learning (SoTL) as a legitimate form of scholarly investigation and the shape that it takes in post-secondary education are inherently discipline-specific. This paper examines how the character and heritage of public administration influence the acceptance of SoTL, and the form that it takes. It argues that the applied nature of public administration and its interdisciplinary character have influenced SoTL in the discipline. This study concludes systematic self-reflection by disciplines may be needed to identify potential factors that limit the acceptance and/or direction of SoTL in a discipline.

Many practitioner researchers strive to understand which assessment practices have the best impact on learning, but in authentic educational settings, it can be difficult to determine whether one intervention, for example the introduction of an online quiz to a course studied by diverse students, is responsible for the observed effect. This paper uses examples to highlight some of the difficulties inherent in assessment research and suggests some ways to overcome them. Problems observed in the literature include: assuming that if two effects are correlated then one must have caused the other; confounding variables obscuring the true relationships; experimental approaches that are too far removed from reality; and the danger that self-reported behaviour and opinion is sometimes different from student's actual behaviour. Practical solutions include: the use of an experimental or pseudo-experimental approach; the use of mixed methods; and the use of meta-analysis.

This article is written to inform educational community particularly in the respect of new tendencies in educational assessment and present a clear-cut picture of the recent studies conducted in the Black Sea Region. The review paper refers to the following countries: Georgia, Russia, Turkey and Ukraine. It stresses the prevalent approach detected in these countries and sees the role of assessment as a tool to enhance the quality of education and obtain the reliable data that help professionals to make appropriate decisions in teaching and assessment.

This study compares the use and efficacy of assessment grading tools within postgraduate education courses in a regional Australian university and a regional university in the US. Specifically, we investigate how the quality of postgraduate education courses can be improved through the use of assessment rubrics or criterion referenced assessment sheets (CRA sheets). The researchers used a critical review of rubrics from Master of Education courses, interviews and a modified form of the Delphi method to investigate how one can assure the quality of assessment grading tools and their effects on student motivation and learning. The research resulted in the development of a checklist, in the form of a set of questions, that lecturers should ask themselves before writing rubrics or CRA sheets. The paper demonstrates how assessment grading tools might be researched, developed, applied and constantly improved in order to advance the Scholarship of Teaching and Learning.

The challenge of supporting the development of meaningful learning is prevalent in chemistry education research. One of the core activities used in the learning process is assessments. The aim of this paper is to illustrate how the semantics dimension of Legitimation Code Theory can be a helpful tool to critique the quality of assessments and reveal how this quality potentially contributes to meaningful learning. For this purpose we analysed an exam paper from an introductory chemistry module, using the semantics dimension as a framework. We discuss the tools that were designed for this analysis and how it was applied to reveal the weakness in this particular assessment. Suggestions for how this assessment can be improved is also discussed. This study illustrates how the semantics dimension can inform assessment practice and potentially contribute to the development of meaningful learning.

George Grob presented the fifth and final Eleanor Chelimsky Forum address at the 2017 annual meeting of the Eastern Evaluation Research Society. In this commentary, I respond to several points that George raises in the "American Journal of Evaluation" paper based on that address. An overarching theme of my comments involves the potential for strengthening the linkages between evaluation theory and evaluation practice. Research on evaluation should also be added to this pairing.

This paper reports on what has happened since Elliott ("Dynamic Assessment in Educational Settings: Realising Potential," 2003) in those applications of dynamic assessment that he considered. There continues to be two broad applications, one, largely researcher led, and the other, largely practitioner led, although there are examples of researcher-practitioners. Since Elliott, arguably, it is now relatively easy both to reach consensus over constructs and assessment processes and also to mutually respect differences through clarity over professional and assessment purposes. There is a fuller account than Elliott provided of educational psychologists' practice in the UK, which partly serves to highlight the difference in purpose and imperative to research and evaluate outcomes that persists for researchers and practitioners. The paper concludes by arguing that the premise in Elliott's title of "realising potential" was always contestable.

Existing literature indicates that physical attractiveness positively affects variables such as income, perceived employee quality and performance evaluations. Similarly, in the academic arena, studies indicate instructors who are better looking receive better teaching evaluations from their students. Previous analysis of the website RateMyProfessors.com confirms this, indicating that instructors who are viewed by students as "hot" receive higher "quality" ratings than those who are "not." However, psychology literature indicates that perceptions of attractiveness are influenced by positive illusions, a property whereby individuals with higher quality relationships view each other more positively than objective observers. This paper uses data from Rate My Professors to investigate the existence of positive illusions in the instructor-student relationship. It finds that positive illusions exist, suggesting that existing literature overestimates the premium associated with physical attractiveness. Furthermore, the source of these illusions varies significantly between male and female instructors with important implications for the role of gender in workplace evaluations, hiring, promotion, and tenure.

The main problem of the educational evaluation validity is that it just copies the conceptual framework system of validity from educational measurement to its own conceptual system. The validity conceptual system that fits the need of theory and practice of educational evaluation has not been established yet. According to the inherent attributive character of educational evaluation, this paper constructs eight concepts of validity in educational evaluation: objective validity, construct validity, content validity, cross validity, subject validity, process validity, function validity and result validity, and elaborates its basis and connotation.

Assessment pervades the learning process. This paper provides an overview of the application of technology to support and enhance diagnostic, formative and summative assessment. The focus is on examining how it can replace what already exists, improve the functionality, catalyse a redesign of the process and in some circumstances, make possible what was previously inconceivable. The paper considers formal and informal individual learning environments, group settings and assessment at a national or international scale.

Background: The use of research in policy settings is complex, unpredictable and influenced by a range of poorly understood social factors. This makes it difficult to plan for, facilitate and evaluate policy impacts arising from research. Aims and Objectives: 1. Propose and test tools for planning for and facilitating research impact, based on a new logic model combined with a novel approach to public/stakeholder analysis. 2. Propose and test methods for establishing causal links between research and policy impacts. 3. Use case study findings to provide new empirical insights into the social processes that mediate the generation of impact from research. Methods: Social Network Analysis, qualitative analysis of semi-structured interviews, and analysis of secondary data were used in a case study of peatland climate change research in Scottish Government policy. Findings: Boundary organisations and centrally-positioned, well-trusted individuals, were crucial to the development of a trusted body of research in which policymakers were sufficiently confident as the basis for policy. Discussion and conclusions: The non-linear social dynamics that characterise science-policy networks can be understood and evaluated. By using the tools described in this paper, researchers and other stakeholders can better plan, facilitate and evaluate research impact.

There is a lively discussion on how to evaluate competence-based higher education in both evaluation and competence research. The instruments used are often limited to course evaluation or specific competences, taking a rather narrow perspective. Furthermore, the instruments often comprise predetermined competences that cannot be adapted to higher education needs. The aim of this paper is to provide a flexible screening instrument into which practitioners/evaluators of study programmes can themselves insert the competences they are interested in, i.e. competences students ought to acquire through their study programme. The present study describes the development of the Competence Screening Questionnaire for Higher Education (CSQ-HE) in different domains of higher education. Results show that the CSQ-HE provides plausible screening information whilst fulfilling evaluation standards. It can be integrated into a higher education institution's internal quality management system to demonstrate a programme's legitimacy and enhance competence-based teaching and learning. Strengths, weaknesses and implications for practice are discussed.

Assessment as a field of investigation has been influenced by a limited number of perspectives. These have focused assessment research in particular ways that have emphasised measurement, or student learning or institutional policies. The aim of this paper is to view the phenomenon of assessment from a practice perspective drawing upon ideas from practice theory. Such a view places assessment practices as central. This perspective is illustrated using data from an empirical study of assessment decision-making and uses as an exemplar the identified practice of 'bringing a new assessment task into being.' It is suggested that a practice perspective can position assessment as integral to curriculum practices and end separations of assessment from teaching and learning. It enables research on assessment to de-centre measurement and take account of the wider range of people, phenomena and things that constitute it.

This paper updates a review of dynamic assessment in education by the first author, published in this journal in 2003. It notes that the original review failed to examine the important conceptual distinction between dynamic testing (DT) and dynamic assessment (DA). While both approaches seek to link assessment and intervention, the former is of particular interest for academic researchers in psychology, whose focus is upon the study of reasoning and problem-solving. In contrast, those working in the area of dynamic assessment, often having a practitioner orientation, tend to be particularly concerned to explore the ways by which assessment data can inform educational practice. It is argued that while some authors have considered the potential value of DA in assisting classification, or in predicting future performance, the primary contribution of this approach would seem to be in guiding intervention. Underpinning this is the view that DA can shed light on the operation of underlying cognitive processes that are impairing learning. However, recent research has demonstrated that the belief that deficient cognitive/executive functions could be identified and ameliorated, and subsequently result in academic progress, has not been supported. Where gains in such processes/functions have sometimes been found in laboratory training studies, these have tended not to transfer meaningfully to classroom contexts. The review concludes by pointing out that DA continues to be supported primarily on the basis of case studies and notes that the 2003 call for research that systematically examines the relationship between assessment and intervention has yet to be realised.

In this paper, we focus on participatory evaluation in the context of international development and specifically on the emerging empirical knowledge base. In a prior review and critique of research on participatory evaluation (Cousins and Chouinard 2012), we examined 121 studies, with only 21 (17%) situated in development contexts. However, the circumstances and challenges for international development and for development evaluation are distinct from those found in developed countries and therefore warrant separate consideration. To provide a more focused and detailed understanding of participatory evaluation in international development contexts, in this paper we augment our initial sample to a total of 40 studies on participatory evaluation in development published over the past 16.5 years. Based on an analysis of this research and related theoretical and conceptual contributions we identify and discuss eight emergent themes: multiplicity of relationships, consequences of stakeholder selection, characterization of participation, contextual complexity, methodological requirements, cultural influence, politics and power and learning and capacity building. We conclude with some reflections on an agenda for research.

Without deliberate and resourced translation, research evidence is unlikely to inform policy and practice. This paper describes the processes and practical solutions used to translate evaluation research findings to improve the readability of print materials in a large scale worksite health programme. It is argued that a knowledge brokering and translation role, clear planning, effective communications, strong senior leadership support and strategic alignment, and mechanisms to engage practitioners and research users are key factors required to help achieve policy and practice change directly based on research evidence.

In educational research that calls itself empirical, the relationship between validity and reliability is that of trade-off: the stronger the bases for validity, the weaker the bases for reliability (and vice versa). Validity and reliability are widely regarded as basic criteria for evaluating research; however, there are ethical implications of the trade-off between the two. The paper traces a brief history of the concepts, and then describes four ethical issues associated with the validity-reliability trade-off in educational research: bootstrapping, stereotyping, dehumanization, and determinism. The article closes by describing emerging trends in social science research that have the potential to displace the validity-reliability trade-off as a central concern for the evaluation of educational research: the introduction of translational sciences, a shift from significance to replicability, a move from inference to Big Data, and the increasing importance of consequential validity.

Purpose: In the higher education sector, the evaluation of learning and teaching projects is assuming a role as a quality and accountability indicator. The purpose of this paper is to investigate how learning and teaching project evaluation is approached and critiques alignment between evaluation theory and practice. Design/Methodology/Approach: The emergent realism paradigm provides the theoretical framework with a pragmatic approach to mixed-methods data collection. Thematic analysis was used to analyze the transcripts of interviews with 15 project leaders. Findings: Four key themes on project evaluation emerged: how evaluation is conceptualized, particularly the overlap, even conflation, between evaluation and research; capability building within the sector; resourcing in terms of time and money; and the role of an action-oriented approach to evaluation. The authors conclude that misalignment exists between evaluation theory and the practice of project evaluation and that this relationship can be further inhibited by a project leader's perception of evaluation. Practical Implications: A series of strategies for developing capacity across the higher education sector for project evaluation are presented. These include the development and provision of: a time allocation for evaluation in future and ongoing project plans with procedures to revisit the project and assess impact; models of how to incorporate evaluation into the research cycle; constructive feedback on evaluation reports from the university funding body; and networking opportunities to disseminate learnings from project evaluations. Originality/Value: This study focusses on the under-researched area of evaluation of learning and teaching projects in higher education, providing research-based evidence for strategies to develop sector capacity.

In the vast evaluation literature, there are numerous accounts describing the emergence of the field of evaluation. However, texts on evaluation history often describe how structural conditions for conducting evaluation have changed over time, often from an American perspective. Inspired by the Oral History Team, the purpose of this article is to provide an oral history description of the professional development of a person who has contributed to the emergence of evaluation in Sweden. The article concerns Evert Vedung, undoubtedly Sweden's most well-known evaluation researcher. Vedung is intellectually distinct from many other prominent evaluation researchers, given his background in political science. He believes that evaluation is closely connected to representative democracy, which affects his view on what value criteria are to be used in evaluation. This article presents an edited conversation with Evert Vedung on his research trajectory. The article aims to describe how serendipities have shaped Vedung's professional development.

From a utilization-focused evaluation perspective, the success of an evaluation is rooted in the extent to which the evaluation was used by stakeholders. This paper details the "Expectations to Change" (E2C) process, an interactive, workshop-based method designed to engage primary users with their evaluation findings as a means of promoting evaluation use and building evaluation capacity. The process is uniquely suited for contexts in which the aim is to assess performance on a set of indicators by comparing actual performance to planned performance standards for the purpose of program improvement. The E2C process was originally developed for use in a participatory evaluation with a non-profit human service organization. Evidence from a one-year follow-up survey suggested that the method increased participants' conceptual, instrumental, and process use. The general utility of the E2C process is discussed.