Bibliometric Measures of Epistemic Change

Bibliometrics has a long-standing interest in tracing epistemic change. The question of how epistemic change is reflected in publications is implicit to evaluative bibliometrics because the quality of a paper is measured through its impact, which can be understood as the extent to which a paper changes later research. Beyond this idea underlying bibliometric assessments, the question of how to identify and trace emerging fields or emerging technologies has attracted considerable attention (Small et al. 2014, Rotolo et al. 2015), and the dynamics of topics has been traced with bibliometric methods (McCain 2010, Leydesdorff and Rafols 2011). These approaches either identify emerging topics and technologies by searching for rapidly growing clusters of publication or trace changes that are known to exist and trace them using citation patterns, keywords or combinations thereof. The current interest in the identification of exceptional (e.g., so-called novel or disruptive) papers reflects a similar interest in epistemic change (Evans 2010, Azoulay et al. 2011, Klavans et al. 2014, Wang et al. 2017, Shibayama and Wang 2020, Park et al. 2023).

The application of bibliometric methods for tracing epistemic change also appears to be hindered by constraints that are not necessary but appear to be somewhat self-inflicted in that they are due to methodological traditions that are taken for granted in bibliometrics. These ‘soft’ limitations of bibliometric indicators are exemplified by the recent deluge of new indicators which purportedly describe epistemic change. Indicators like Innovativeness, Novelty, Originality or Disruptiveness describe single papers and attempt to codify the epistemic change they bring about (e.g. Ayoubi et al. 2021, Park et al. 2023). This design is based on three implicit assumptions. First, the construction of indicators of epistemic change from metadata of individual papers implies the assumption that the epistemic change is an event rather than a process. Only an epistemic event could be described by a single paper, while describing an epistemic process requires the consideration of a set of papers which report research processes building on each other. Second, the construction of these indicators also assumes that the event that is measured by the indicator is fully reported in exactly one paper. Each indicator only processes metadata of one single paper. Third, indicators sacrifice specificity for comparability. As a consequence, they attempt to describe “how much” epistemic change is represented by the paper while being unable to report the kind of change that has occurred.

These three limitations, which appear to be the results of choices made by indicator designers rather than ‘hardwired’ limitations of bibliometrics, come on top of the only truly hardwired limitation, namely the dependence of bibliometrics on the signal about a publication’s content that is contained by the publication’s metadata. This limitation raises a fundamental question about the validity of bibliometric measures of epistemic change. Since we don’t know how the content of a paper is reflected in the bibliometric metadata, we cannot assess the content or strength of the signal metadata contain and thus cannot know if the signal is validly codified by an indicator.

The special collection presents papers that were developed in a series of workshops which addressed these four challenges. It includes the following topics:

Conceptualisation and operationalisation: How can epistemic change be conceptualised in a way that supports the operationalisation of concepts for the bibliometric tracing of epistemic change?

Ground truths: How can ground truths be constructed for the development and tests of bibliometric methods for tracing epistemic change?

Metadata: Which information about epistemic change is included in bibliographic metadata and can thus be exploited by bibliometric methods?

Validation: How do established bibliometric indicators fare when trying to recover the ground truths?

Artificial intelligence: How can epistemic change identified in publications with the application of LLMs to full texts of publications?

References

Ayoubi, C., M. Pezzoni and F. Visentin (2021). "Does It Pay to Do Novel Science? The Selectivity Patterns in Science Funding." Science and Public Policy 48(5): 635-648.

Azoulay, P., J. S. Graff Zivin and G. Manso (2011). "Incentives and creativity: evidence from the academic life sciences." The RAND Journal of Economics 42(3): 527-554.

Evans, J. A. (2010). "Industry Induces Academic Science to Know Less about More." American Journal of Sociology 116(2): 389-452.

Klavans, R., K. Boyack, H. Small, A. A. Sorensen and J. P. A. Ioannidis (2014). Indicators of Innovative Research. Context counts: proceedings of the STI 2014. E. Noyons. Leiden, Universiteit Leiden: 314-320.

Leydesdorff, L. and I. Rafols (2011). "Local emergence and global diffusion of research technologies: An exploration of patterns of network formation." Journal of the American Society for Information Science and Technology 62(5): 846-860.

McCain, K. W. (2010). "Core journal literatures and persistent research themes in an emerging interdisciplinary field: Exploring the literature of the evolutionary development biology." Journal of Informetrics 4: 157-165.

Park, M., E. Leahey and R. J. Funk (2023). "Papers and patents are becoming less disruptive over time." Nature 613(7942): 138-144.

Rotolo, D., D. Hicks and B. R. Martin (2015). "What is an emerging technology?" Research Policy 44(10): 1827-1843.

Shibayama, S. and J. Wang (2020). "Measuring originality in science." Scientometrics 122(1): 409-427.

Small, H., K. W. Boyack and R. Klavans (2014). "Identifying emerging topics in science and technology." Research Policy 43(8): 1450-1467.

Wang, J., R. Veugelers and P. Stephan (2017). "Bias against novelty in science: A cautionary tale for users of bibliometric indicators." Research Policy 46(8): 1416-1436.