THE PERFECT TIME TO REFORM PEER REVIEW

The world is full of people who have opinions. Some of those are taken more seriously than others. The world of the intelligentsia, or those who produce takes and opinion columns and articles, is certainly no less hierarchical than the rest of our unequal society, and possibly more so. Within this world, as elsewhere, advantage quickly turns to greater advantage. The more followers you start with, the easier it is to grow your audience on social media. If I have an article do well in a newspaper or professional journal, it is easier to persuade the editor to take a chance on my next piece. Journalists will repeatedly turn to those who have given good quotes before. And so it goes.

This is a process of cumulative advantage: the rich inevitably become richer, and the gap between them and those at the bottom tends to grow. The sociologist Robert Merton ([1968]) memorably dubbed this the ‘Matthew effect’, after Matthew 13:12 wherein Christ says, ‘For whoever has, to him more shall be given, and he will have an abundance; but whoever does not have, even what he has shall be taken away from him’. The various gatekeeping institutions and mechanisms we have in place tend to reinforce this Matthew effect in the attention economy.

Should we take steps to equalize this highly unequal attention economy? After all, deciding what topics are worthy of collective attention—what claims are worth taking seriously and perhaps making the basis of political action—is a decision with huge social ramifications. Malign corporate interests show by their actions that they know this: witness the huge amount of time and money spent by consulting firms working on behalf of oil companies or tobacco companies trying to shape what sort of studies come to public attention (Oreskes and Conway [2010]). Are we really sure we want to trust this responsibility to… whoever is deciding what is worth listening to now? Do we know that their gatekeeping efforts will line up with the public good in a way that we can trust and rely upon?

There are many instantiations of the Matthew effect and gatekeeping, and there is a lot to say about each of them. For now, let us limit our attention to purportedly scientific opinions—about, say, the effectiveness of COVID-19 vaccines, the likelihood of extreme weather events in the future, the causes of crime, and so on. These topics directly affect our lives in important ways, so we would like to know which opinions about them to believe or take seriously. It would be terribly dull to have to dredge through everything that anyone has ever seen fit to write down. How do we solve this problem? We have peer-reviewed journals! Experts scrutinize and filter proposals. We, the curious public, can then apportion our attention by whether or not an idea has appeared in a reputable scientific publication. Journalists can use what appears in these journals as a guide to what sort of (purported) new results are conversation-worthy in the public sphere. Thus our problem is solved.

It might even be thought that there is no real alternative: without peer-reviewed journals, how else would we know which purported scientific opinions to trust or take seriously? This would be too quick, however, as peer review is not as intimately bound up with the scientific method as is sometimes suggested. It is true that Henry Oldenburg introduced some version of peer review in his role as editor of the Philosophical Transactions of the Royal Society, founded in 1655 and one of the first scientific journals. Yet peer review did not take on the central role it currently occupies in the scientific attention economy until halfway through the twentieth century. This is illustrated by Einstein’s indignant reply upon receiving his first reviewer report in 1936: ‘I sent the paper for publication, not review’ (Kennefick [2005]). For most of the history of modern science, peer review played at most a peripheral role.

And there are costs to using peer review as our main way of deciding which scientific work is worth our attention. For one, peer review takes time, sometimes a lot (Himmelstein [2015]), during which exciting new ideas are not available to the broader public. This is especially problematic in rapidly developing situations such as during the COVID-19 pandemic; indeed, COVID-19 put great strain on peer review in the relevant disciplines (Packer [2020]). For another, peer review generates social pathologies all of its own. For instance, every now and again the peer-review system generates reviewer rings (for example, Ferguson et al. [2014]; Callaway [2015]). In these cases, reviewers collude with authors: the experts ‘scrutinizing’ the work of their peers resolve to simply let through the work of their friends. Such cases are sensational and rare, but they also show that the very fact of having a peer-review system changes how scientists behave and how they spend their time. Most obviously, it requires that scientists spend some of their time reviewing each other’s work. Even if it does save scientists and the public time reading bad work, is peer review a net good in how it reallocates scientists’ time?

In our recent article—yes, it passed peer review, get the laughs out of your system now—we argue that no, it is not. More carefully, we argue that for all the present evidence, where it is determinate enough to allow a judgement, peer review seems to either make no difference in helping scientists better allocate their time, or actually make things worse. For this reason, we propose a more organic model of peer review, where scientists decide for themselves when to publish their work, and the scientific community determines its worth over time through ‘post-publication peer review’ and by engaging, criticizing, or ignoring the work.

While post-publication peer review sounds complicated and technical, what we envision will be familiar to anyone who has used websites such as Reddit or Rotten Tomatoes. Scientists comment on each other’s work and vote on it. As on Reddit, positive feedback will make something easier to find for others, while negative feedback will make it harder. As on Rotten Tomatoes, we can distinguish ‘certified experts’ (such as those with a PhD in a relevant field) from ‘putative experts’ (everyone else). Readers can sort by certified expert scores or overall scores, as they prefer.

What would the benefits of such a move be? First, scientific work would be available sooner, allowing fellow experts to build on it while it is still being vetted for broader consumption. We have already seen this happen during the COVID-19 pandemic (Flier [2020]), but this process would have been much more controlled if it was already the standard model of publishing.

Second, scientists would be freer to decide for themselves how to divide their time between reviewing, replicating, and original research (compared to the current situation where journal peer review is among many factors putting pressure on scientists’ time). This allows them to spend their time more efficiently, especially if a greater degree of specialization between these tasks occurs (Romero [2018]).

Third, as depositing an article in an open access archive (for example, arXiv, bioRxiv, or OSF) becomes the primary means of publication, exorbitantly expensive (Monbiot [2018]) commercial publishers are circumvented, saving university libraries vast amounts of money (Van Noorden [2013]) while broadening access to scientific work.

Perhaps most importantly, we would put the fate of scientific contributions in the hands of the broad-based judgement of the scientific community, rather than the possibly idiosyncratic views of a journal editor and one to three reviewers. This will make judgements of scientific quality more democratic and less biased. This is not only good for science but also for scientists, as it makes their career success (closely tied to the success of their contributions) a little less arbitrary.

We now return to the question we started with: how will other scientists and interested outsiders know what to read among the sheer endless quantity of scientific papers? It is worth pointing out that it is unclear how helpful peer-reviewed journals as they currently exist really are in this respect, as there is shockingly little evidence that papers appearing in, say, Nature or Science, are more worth reading than those in less prestigious journals (Brembs et al. [2013]; Brembs [2018]). In contrast, shaking up peer review along the lines we suggest will stimulate lots of innovation as new methodologies and metrics are devised for sorting papers. We already suggested metrics based on certified expert scores, which may be contrasted with those that take the opinion of both certified and putative experts into account, but the possibilities are endless. We expect that before long some of these new metrics will turn out to be more reliable than journal prestige (Arvan et al. [2025]).

We, the curious public, can then read new research quickly and for free, apportioning our attention by using a mixture of carefully curated metrics and our own interests and curiosity, and thus our problem is solved. We can then turn our attention to other hierarchical systems of information distribution, such as in journalism and social media. It is worth thinking about whether these can and should similarly be democratized.