Date Published: August 5, 2015
Publisher: Public Library of Science
Author(s): Stefanie Ringelhan, Jutta Wollersheim, Isabell M. Welpe, Pablo Dorta-González.
Due to the increasing amount of scientific work and the typical delays in publication, promptly assessing the impact of scholarly work is a huge challenge. To meet this challenge, one solution may be to create and discover innovative indicators. The goal of this paper is to investigate whether Facebook likes for unpublished manuscripts that are uploaded to the Internet could be used as an early indicator of the future impact of the scientific work. To address our research question, we compared Facebook likes for manuscripts uploaded to the Harvard Business School website (Study 1) and the bioRxiv website (Study 2) with traditional impact indicators (journal article citations, Impact Factor, Immediacy Index) for those manuscripts that have been published as a journal article. Although based on our full sample of Study 1 (N = 170), Facebook likes do not predict traditional impact indicators, for manuscripts with one or more Facebook likes (n = 95), our results indicate that the more Facebook likes a manuscript receives, the more journal article citations the manuscript receives. In additional analyses (for which we categorized the manuscripts as psychological and non-psychological manuscripts), we found that the significant prediction of citations stems from the psychological and not the non-psychological manuscripts. In Study 2, we observed that Facebook likes (N = 270) and non-zero Facebook likes (n = 84) do not predict traditional impact indicators. Taken together, our findings indicate an interdisciplinary difference in the predictive value of Facebook likes, according to which Facebook likes only predict citations in the psychological area but not in the non-psychological area of business or in the field of life sciences. Our paper contributes to understanding the possibilities and limits of the use of social media indicators as potential early indicators of the impact of scientific work.
Assessing and evaluating the impact of research articles is a fundamental process in science that serves the advancement of knowledge in our society . Finding relevant research, processing (current) research, and evaluating research are increasingly difficult and time-consuming undertakings [2,3]. Although performance assessments based on specific quantitative indicators (e.g., counting published journal articles) are fiercely criticized in the literature due to their neglect of qualitative aspects of scientific performance [1,4,5], the strong interest in performance evaluations to indicate the impact of scientific work is understandable [5–7].
While additional analyses in Study 1 have shown that there are differences across disciplines in the predictive value of (non-zero) Facebook likes for citations, Study 1 is limited in its generalizability because the data were collected on one particular website. Therefore, in Study 2, we collected new data from a website in the life sciences.
This study set out to investigate Facebook likes as a potential early predictor of the impact of scientific work. Across both studies reported in this paper, we observed that Facebook likes of manuscripts cannot predict the journal-level indicators Immediacy Index or the Impact Factor, except for Facebook likes of non-psychological manuscripts from the HBS website, which negatively predict the Impact Factor. Another outcome of this paper is that citations are predicted by Facebook likes for psychological manuscripts from the HBS website. As indicated in our additional analyses, the significant prediction of citations by non-zero Facebook likes of manuscripts from the HBS website stems from the psychological and not from the non-psychological manuscripts on the HBS website. In Study 2, we observed no significant predictions of citations by Facebook likes of the manuscripts uploaded on bioRxiv. These outcomes demonstrate that Facebook likes may only have a predictive value in some disciplines. The data suggests that Facebook likes predict citations in the field of psychology but that they do not predict citations in, for example, life sciences. This interdisciplinary difference is also reflected in the fact that the HBS manuscripts (N = 170) that we categorized as being psychological in our study (45.9%) received 68.2% of the Facebook likes, whereas the manuscripts that we categorized as being non-psychological (54.1%) received 31.8% of the Facebook likes (note that similarly, 33.3% of the psychological manuscripts received no Facebook likes at all compared to 53.3% of the non-psychological manuscripts). Furthermore, interdisciplinary differences in social media behavior, i.e., uploading of papers  and the presence of altmetrics , have also been reported in the literature, with a higher involvement from social sciences and humanities . Thus, these findings indicate a similar direction.
In summary, our paper contributes to the assessment of the impact of scientific work via social media indicators. We compared Facebook likes for manuscripts with traditional indicators of success and find that Facebook likes may be used as an addmetric and early impact indicator of psychological manuscripts, not, however, manuscripts from non-psychological areas of business or the area of life sciences. In light of the discipline-depending results, Facebook likes at least in part resemble something other than the impact measured by traditional impact measures. Thus, we are convinced that science would benefit from an open-minded allowance of diverse indicators to assess the multitude of aspects of scientific performance observed from the angle of different stakeholders. However, applying diverse indicators and assessment methods should be performed in a reflective manner, keeping the shortcomings of these indicators in mind. The contributions of our paper are relevant for scholars themselves and for the governance of academia, (social) media, society and other stakeholders. Future research in this direction is desirable to clarify the specific type of impact that Facebook likes measure and also who likes manuscripts in different disciplines. Our study is also conducive to elucidating the validity and reliability of Facebook likes as an addmetric.