Search

In recent times, there has been a surge of impact-oriented, transdisciplinary research programmes and projects integrating multiple disciplines, types of knowledge and practices. An essential element often mentioned in the literature to improve the performance of these programmes and support impact delivery is continuous reflection and learning via evaluation. We argue that because a standard format for organizing transdisciplinary research does not exist, tailor-made evaluation approaches designed around the specificities of each programme are needed. The existing evaluation literature provides useful building blocks that can be integrated and adapted to specific transdisciplinary research contexts. In this article, we develop the valorization cycle and apply it to evaluate a transdisciplinary research programme in the water sector. The building blocks of our approach are: understanding of valorization as a cyclical process; theory of change as a logic model to structure the evaluation; productive interactions as a process indicator articulated in learning outcomes (cognitive, relational, and strategic); and impact pathways as narratives to explain impact dynamics. Our framework is based both on research evaluation and learning literature and on our personal experience in the evaluation of transdisciplinary research. The evaluation of the programme showed how the valorization cycle can point to different learning outcomes across the research process that are conducive to impact, and provided useful insights to the programme managers to adjust the programme. The principle of tailoring an evaluation approach to the specificities of the programme evaluated using building blocks from the literature will be applicable in other transdisciplinary contexts too.

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies' findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

Spin-off companies are generally considered a promising vehicle for developing academic knowledge into products that are ready for the market. In this paper the authors explore under what circumstances spin-off companies can serve as a source of knowledge for the research organization from which they originate. The paper brings together literature from different academic fields to construct an analytical framework for investigating knowledge feedback mechanisms between spin-off companies and their parent research organizations. The authors illustrate the application of this framework in six case studies of parent-spin-off couples in the water technology sector. These case studies show that the interaction with spin-off companies can yield important cognitive benefits for the academic research process, such as an improvement of the research agenda and new insights about the practical operation of theoretical models and technologies. These benefits were facilitated mainly by staff exchange, collaborative research and personal contacts.

Distinguishing between research collaboration, consultancy, dissemination, and commercialization of research results, this paper analyses the determinants of researchers' societal engagement. The analytical framework integrates societal engagement as part of the credibility cycle. Several variables extend previous findings on determinants and mechanisms—herein scientific recognition and funding sources. A novel method to investigate the relationship between scientific recognition and societal engagement is explored. Drawing on a large-scale survey of European-based researchers in physics, cardiology, and economics, we find that several factors are associated with different modes of societal engagement in complex and intersecting ways. Scientific recognition is positively associated with research collaboration and dissemination, while organizational seniority is associated with all modes except for research collaboration with non-scientific actors. Female gender is positively associated with dissemination and external funding sources are positively associated will all. The findings intersect with differences in the three research fields. [ABSTRACT FROM AUTHOR]

This paper aims to explore disciplinary variation in valuation practices by comparing the way research groups accumulate credibility across four epistemic cultures. Our analysis is based on case studies of four high-performing research groups representing very different epistemic cultures in humanities, social sciences, geosciences and mathematics. In each case we interviewed about ten researchers, analyzed relevant documents and observed a couple of meetings. In all four cases we found a cyclical process of accumulating credibility. At the same time, we found significant differences in the manifestation of the six main resources that are part of the cycle, the mechanisms of conversion between these resources, the overall structure and the average speed of the credibility cycle. The different ways in which the groups use data and produce arguments affect the whole cycle of accumulating credibility. In some cultures, journal publications are the main source of recognition, but in others one can earn significant amounts of recognition for conference contributions or service to the academic community. Moreover, the collaboration practices in the respective fields strongly influence the connection between arguments and publications. In cultures where teams of researchers collaboratively produce arguments, it is more strongly embedded in the process of writing publications. We conclude that the credibility cycle can only be used as an analytical tool to explain the behavior of researchers or research groups when taking differences across epistemic cultures into account.

Over the past decades, science funding shows a shift from recurrent block funding towards project funding mechanisms. However, our knowledge of how project funding arrangements influence the organizational and epistemic properties of research is limited. To study this relation, a bridge between science policy studies and science studies is necessary. Recent studies have analyzed the relation between the affordances and constraints of project grants and the epistemic properties of research. However, the potentially very different affordances and constraints of funding arrangements such as awards, prizes and fellowships, have not yet been taken into account. Drawing on eight case studies of funding arrangements in high performing Dutch research groups, this study compares the institutional affordances and constraints of prizes with those of project grants and their effects on organizational and epistemic properties of research. We argue that the prize case studies diverge from project-funded research in three ways: 1) a more flexible use, and adaptation of use, of funds during the research process compared to project grants; 2) investments in the larger organization which have effects beyond the research project itself; and 3), closely related, greater deviation from epistemic and organizational standards. The increasing dominance of project funding arrangements in Western science systems is therefore argued to be problematic in light of epistemic and organizational innovation. Funding arrangements that offer funding without scholars having to submit a project-proposal remain crucial to support researchers and research groups to deviate from epistemic and organizational standards.

This paper investigates the consequences of institutional changes on academic research practices in eight fields of natural science in the Netherlands. The authors analyse the similarities and differences among the dynamics of these different fields and reflect on possible explanations for the changes observed. The study shows that the increasing pressure for productivity, as measured in bibliometric terms, can counteract the pressure for practical utility. Moreover, the work indicates that the dynamics of science varies much more across scientific fields than most of the literature suggests is the case. (Contains 9 notes, 2 figures and 6 tables.)

In many Western science systems, funding structures increasingly stimulate academic research to contribute to practical applications, but at the same time the rise of bibliometric performance assessments have strengthened the pressure on academics to conduct excellent basic research that can be published in scholarly literature. We analyze the interplay between these two developments in a set of three case studies of fields of chemistry in the Netherlands. First, we describe how the conditions under which academic chemists work have changed since 1975. Second, we investigate whether practical applications have become a source of credibility for individual researchers. Indeed, this turns out to be the case in catalysis, where connecting with industrial applications helps in many steps of the credibility cycle. Practical applications yield much less credibility in environmental chemistry, where application-oriented research agendas help to acquire funding, but not to publish prestigious papers or to earn peer recognition. In biochemistry practical applications hardly help in gaining credibility, as this field is still strongly oriented at fundamental questions. The differences between the fields can be explained by the presence or absence of powerful upstream end-users, who can afford to invest in academic research with promising long term benefits. (Contains 1 figure, 5 tables, and 20 footnotes.)

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies' findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value. [ABSTRACT FROM AUTHOR]

A transition towards a transdisciplinary (TD) approach in science is increasingly called for, yet our understanding of how such transitions actually occur in practice remains limited. This paper analyses a Dutch water research programme in transition from a mono- to a TD structure. The main findings of the research are first that efforts to embed a TD approach stimulated interaction between different scientific disciplines (interdisciplinarity), but in some cases hampered the integration of non-academic knowledge and participation in research (co-creation). Second, the composition of TD research groups matters deeply; in our research, we found that practitioners had the biggest say, with consequences for research practice. Third, unlike the case of other studies on transdisciplinarity, most researchers in our study recognized the importance of integrating experiential knowledge in research. [ABSTRACT FROM AUTHOR]

Many countries witness the rise of 'excellence initiatives'. These policies promote vertical differentiation in the science system by funding top research performers and expecting positive spillover effects. However, current understanding of the functioning and (potential) effects of these instruments is limited. We compare policies aimed at promoting excellence in four countries (the UK, Germany, Denmark, and Switzerland), using secondary sources and fourteen expert interviews. Using the notion of coordination approaches as a heuristic tool, we characterise each policy in terms of the coordinating actor, the system addressed, the activities that are coordinated, the specific interventions taken, and the types of relationships affected. We find that countries adopt very different approaches to reach similar goals and thus bring into question appealing but simplistic ideas of 'excellence' as an agreed concept. Remarkably, excellence policies are more prone to reveal existing but tacit diversity in the system than to generate new relational patterns. [ABSTRACT FROM AUTHOR]

Strategic research consortia as policy instruments for research coordination have been on the rise for more than a decade. Despite their rising popularity as coordination structures, there has been little comparative analysis of the actual coordination approaches such consortia develop. In order to enhance our understanding of consortia as coordination structures, this paper makes a systematic and in-depth comparison of the coordination approaches of two Dutch consortia. The analysis shows that research consortia coordinate their activities in very different ways. A consortium’s coordination approach turns out to be strongly influenced by its internal characteristics. The observed influence of internal consortium characteristics implies that the eventual coordination approach of consortia will not always match the rationale behind a policy measure to support these consortia. We recommend policy-makers to foster strategic research consortia with a heterogeneous composition that have organised sufficient flexibility for reacting to unforeseen developments. [ABSTRACT FROM PUBLISHER]

National large-scale interdisciplinary research programmes funded to boost scientific, economic and societal advancement often have difficulty realizing their goals. Reasons include: the monodisciplinary nature of academic research; insufficient interaction with societal actors; and funding conditions. To better understand the mechanisms for dealing with these tensions, we analysed the dynamics of a large and promising interdisciplinary programme in the emerging field of ecological genomics. As part of the programme, with responsibility for its societal valuation, we were able to investigate its initiation and implementation processes in depth, using participative observations, interviews, and process interventions. We adapted the multi-level perspective to analyse the constraining and enabling conditions, and derived practical recommendations for improving the societal outcomes of large research programmes. Our main conclusion is that programmes with an interdisciplinary and societal mission should work explicitly towards this mission from the start, as reorientation at later stages of the programme can be problematic. [ABSTRACT FROM AUTHOR]

There is wide agreement about the importance of transdisciplinary research to address complex sustainability issues. Although there is a growing body of literature about the management of transdisciplinary research programmes as well as the challenges relating to the collaboration between academic researchers and practitioners, empirical research has to date paid little attention to the challenges related to the variation of practitioners involved in these programmes. This paper presents a comparative analysis of three transdisciplinary programmes addressing climate change and sustainability and identifies factors that contribute to fruitful collaboration between heterogeneous practitioners. Contrary to what could be expected from the literature, differences between the practitioners involved have only created significant issues in one of the programmes, and this programme has developed a way to cope with this difficulty. Effective strategies to avoid and limit tensions among practitioners include a focus on bilateral collaborations, the careful selection of programme participants, and the appointment of dedicated project monitors who are responsible for social learning processes. [ABSTRACT FROM AUTHOR]