Search

The peer review of research results submitted for journal publication raises elementary issues of fairness and reliability.' Peer review of proposals to perform research in the future, however, is even more problematic. For many reasons, judging untested ideas is inherently more uncertain than evaluating completed work. Research has a way of evolving in directions that are unchartable in advance-certain data may prove unattainable, new discoveries by the researchers or by others may point to reorientation, or personnel may change. In a 1974 study, Grace Carter found that ratings of National Institutes of Health (NIH) initial grant applications were correlated with independent ratings of their later reapplications by a thin 0.4 (i.e., only 16% of the variance was accounted for by the other rating).2 While this figure includes unreliability due to an independent rerating, it also reflects changes in the perception of the value of specific projects as research progresses. One National Science Foundation (NSF) proposal reviewer made special note of "exemplary staffing plans" of a proposal he evaluated.3 Ironically, that same project changed staff repeatedly to the extent that it was not possible even to identify a project leader. In essence, then, peer review of proposals is a difficult business. Peer review as a process has engendered strong

Study of citation patterns has become a leading analytical tool in the sociology of science, and offers an enticing way to evaluate the research productivity of individual scientists. However, there remain some basic conceptual and nmethodological concerns. Debate touches on the validity of citation counts as an indicator of 'scientific quality' (of papers or authors), of recognition, of scientific diffusion, and of utilization (who cites whom).I To Menard, citations are the only enduring record that a paper is used,2 and Cole and Cole take citations to be a roughly valid indicator of influence.3 But Kaplan wonders what citations mean, in the absence of know'ledge about actual norms and practices.4 Plausible reasons for citation include selectively supporting one's contentions (i.e. 'persuasion'), indicating knowledge of the literature, providing clues about content to citation index scanners -as well as acknowledging genuine communication.5 Citation does not actually prove a reference has been read (or even seen),6 yet journal citations appear to be the best practical indicator of the worth of research.7 Citations may be used to evaluate papers (either individually or in groups e.g. experimental vs. theoretical physics; or short notes vs. full reports); or, similarly, to evaluate journals; or scientists. This paper concerns citation analysis of scientist-authors (both individuals and groups), and discusses the methodological problems involved. The problems to be faced depend on the uses to which the measures are to be put. The problems and opportunities of a supervisor evaluating the citation profile of a young scientist (an arguable practice) are poles apart from those of a sociologist studying groups of scientists in an international specialty, or of a historian studying its development. However, there are some generalizable concerns: these include the possibility that radical work will go unrecognized (and uncited) by contemporaries: that outstanding ideas become widely accepted and not formally cited (e.g. the Mossbauer effect);8 and that citation counts do not

Crossdisciplinary research plays a major role in addressing many problems of intellectual and societal importance. We describe a framework to characterize such projects that replaces "discipline "by "intellectual skills." We applied this framework to study 40 National Science Foundation-sponsored projects. We found that the intellectual skills varied widely among projects in depth, range, andform. Contrary to the conventional wisdom, we found that many academic departments were "open" to broad ranging skill acquisition and use. Unfortunately our evidence found NSF peer review processes unfavorably disposed to multiskill projects. Lastly, we discuss various approaches to organizing and practicing such research.