Date Published: October 7, 2008
Publisher: Public Library of Science
Author(s): Neal S Young, John P. A Ioannidis, Omar Al-Ubaydli
Abstract: John Ioannidis and colleagues argue that the current system of publication in biomedical research provides a distorted view of the reality of scientific data.
Partial Text: This essay makes the underlying assumption that scientific information is an economic commodity, and that scientific journals are a medium for its dissemination and exchange. While this exchange system differs from a conventional market in many senses, including the nature of payments, it shares the goal of transferring the commodity (knowledge) from its producers (scientists) to its consumers (other scientists, administrators, physicians, patients, and funding agencies). The function of this system has major consequences. Idealists may be offended that research be compared to widgets, but realists will acknowledge that journals generate revenue; publications are critical in drug development and marketing and to attract venture capital; and publishing defines successful scientific careers. Economic modelling of science may yield important insights (Table 1).
In auction theory, under certain conditions, the bidder who wins tends to have overpaid. Consider oil firms bidding for drilling rights; companies estimate the size of the reserves, and estimates differ across firms. The average of all the firms’ estimates would usually approximate the true reserve size. Since the firm with the highest estimate bids the most, the auction winner systematically overestimates, sometimes so substantially as to lose money in net terms . When bidders are cognizant of the statistical processes of estimates and bids, they correct for the winner’s curse by shading their bids down. This is why experienced bidders sometimes avoid the curse, as opposed to inexperienced ones [1–4]. Yet in numerous studies, bidder behaviour appears consistent with the winner’s curse [5–8]. Indeed, the winner’s curse was first proposed by oil operations researchers after they had recognised aberrant results in their own market.
Successful publication may be more difficult at present than in the past. The supply and demand of scientific production have changed. Across the health and life sciences, the number of published articles in Scopus-indexed journals rose from 590,807 in 1997 to 883,853 in 2007, a modest 50% increase. In the same decade, data acquisition has accelerated by many orders of magnitude: as an example, the current Cancer Genome Atlas project requires 10,000 times more sequencing effort than the Human Genome Project, but is expected to take a tenth of the time to complete . In the current environment, the distinction between raw data and articles (telling for sure what more an article has compared with raw data) can sometimes become difficult. Only a small proportion of the explosively expanded output of biological laboratories appears in the modestly increased number of journal slots available for its publication, even if more data can be compacted in the average paper now than in the past.
The authority of journals increasingly derives from their selectivity. The venue of publication provides a valuable status signal. A common excuse for rejection is selectivity based on a limitation ironically irrelevant in the modern age—printed page space. This is essentially an example of artificial scarcity. Artificial scarcity refers to any situation where, even though a commodity exists in abundance, restrictions of access, distribution, or availability make it seem rare, and thus overpriced. Low acceptance rates create an illusion of exclusivity based on merit and more frenzied competition among scientists “selling” manuscripts.
When scientific information itself is the commodity, there is uncertainty as to its value, both immediately and in the long term. Usually we do not know what information will be most useful (valuable) eventually. Economists have struggled with these peculiar attributes of scientific information as a commodity. Production of scientific information is largely paid for by public investment, but the product is offered free to commercial intermediaries, and is culled by them with minimal cost, for sale back to the producers and their underwriters! An explanation for such a strange arrangement is the need for branding—marking the product as valuable. Branding may be more important when a commodity cannot easily be assigned much intrinsic value and when we fear the exchange environment will be flooded with an overabundance of redundant, useless, and misleading product [39,45]. Branding serves a similar and complementary function to the status signal for scientists discussed above. While it is easy to blame journal editors, the industry, or the popular press, there is scant evidence that these actors bear the major culpability [46–49]. Probably authors themselves self-select their work for branding [10,11,50–52].
We may consider several competing or complementary options about the future of scientific publication (Box 1). When economists are asked to analyse a resource-allocation system, a typical assumption is that when information is dispersed, over time, the individual actors will not make systematic errors in their inferences. However, not all economists accept this strong version of rationality. Systematic misperceptions in human behaviour occur with some frequency , and misperceptions can perpetuate ineffective systems.