[UPDATED] Peer review or perish: The problem of free riders in philosophy

Beth Hannon

As any journal editor will tell you (at length, possibly via the medium of rant), the trickiest part of the job is not the papers, not the authors, and not even the typesetters. It’s the referees. It is no mean feat to secure referees who are, first, reliable in their academic judgement, second, responsive to emails, and third, willing to return reports when they say they will. But the frustrations of editors aside, the far more pressing concern is for the career prospects of early-career researchers. Jobs and funding can depend on timely decisions. Indeed, whether an early-career researcher gets to become a mid- or late-career researcher can depend on whether a decision is made in a reasonable amount of time.

Common bad behaviour from referees includes (but is not limited to!):

1. Failing to respond to invites in a timely fashion(where timeliness is calculated in days not weeks), even if it’s only to decline the invitation;
2. Agreeing to act as referee and to return the report within an agreed timeframe (in the BJPS’s case, four weeks), only to substantially exceed this timeframe (by weeks, sometimes months) and
   • asking for this substantial extra time for the weakest of reasons*;
   • not communicating with the relevant editors whatsoever;
3. Returning a report long past the agreed timeframe, and that report being almost useless;
4. Not returning the report and not responding to emails enquiring about the report.

Opinions differ on the obligations of academics as referees. Is it unpaid labour, an act of charity towards the community that ought only to be gratefully received? As much a part of the job as teaching and writing? Something in between? Whatever the answer, authors need more from referees than they ever have done; more depends on papers being reviewed in a professional, timely manner. And at the very least, there’s a ‘pay it forward’ case to be made: A paper sent to the BJPS that isn’t desk rejected can be expected to be read by at least six people (and that’s not counting the work that goes into any resubmissions). For every paper an author submits, other people have attended to their work in detail. The author, qua referee, might be expected to return the favour.

I’ve been lucky to witness some extremely productive philosophical engagement between authors and referees. When it’s good, it’s so good. The only shame is that so much of this is hidden. The process viewed en masse—the view one gets as an editor—is primarily one of cooperation and collegiality, and it’s a wonder that puts the lie to the notion of philosophy as anything like an individualistic endeavour.

But what to do about the bad referees, the system’s free riders? Relentless pestering and various forms of emotional blackmail fall on deaf ears. At the BJPS, we operate a flag system for persistent offenders, but all this amounts to is bad referees being asked to perform fewer reviews, while good referees carry more of the load.

So here’s a radical suggestion, using the only weapon motivational device editors have: If someone fails to fulfil their duties as referee, the journal will not accept submissions from that referee, for some period of time to be determined. The time period should reflect the severity of the dereliction of duties. For instance, agreeing to act as a referee but then disappearing off the radar might warrant the most substantial ban. Delivering a meagre report that’s extremely late, and without communicating with the relevant editor about the delay, might mean some shorter period of time on the bench. First-time offenders surely deserve different treatment to persistent re-offenders. And the embargo period will need to be substantial enough to be effective (too short and it will have no real impact; too long and it’s probably not practical due to the changes in the editorial team). The details can be ironed out.

It’s not just badly behaved referees that stand to suffer here. There’s a risk for the journal in question too: bad referees aren’t necessarily bad authors, and we risk losing good papers to other journals by refusing those authors’ papers. But the problem is so rife and its upshot so dire for early-career researchers that maybe something more radical is required to make clear what is expected of referees and ameliorate, at least to some degree, the problem of free-riders. All thoughts on this proposal very welcome!

‘I decided to go on holidays’ and ‘I have other deadlines that I decided to prioritise after agreeing to referee this paper’ are the problems, not the excuse. On the other hand, there are perfectly good reasons to be delayed in returning a report. Not only do we understand, we’ve been there. You are not the droids we’re looking for.

---

[Update: 23 May 2018]
Thanks to everyone, here and elsewhere, for their feedback—it’s been really helpful. I thought I’d add some clarifications to the original post and respond to some of the alternative suggestions. Some concerns stem, I think, from the thought that we’re concerned with a wider set of behaviours than is the case. Some alternative suggestions can’t be accommodated for ethical or practical reasons. I’m sure I haven’t covered every issue here, and so very happy to receive more feedback.

Common concerns:

1. The most important point is that the idea here isn’t to apply a rule mechanically (for example, being banned for being one day late). The reviewer would also receive explicit warnings so this wouldn’t come out of the blue. Like every other aspect of peer review, this proposal isn’t without its drawbacks. Nonetheless, it may be that this is an imperfect solution to a much worse (and very common) problem.
2. We are not proposing any punishments for those who simply decline to review in the first place (at least, so long as they actually communicate this rather than leaving the invitation hanging—and declining here is only ever a matter of clicking a link in an email).
3. We are not asking for the intimate details of reviewers’ lives. While it’s not unknown for us to receive tearful emails from authors and reviewers in terrible situations, we do not expect reviewers to bare all; it’s simply not our business. We will take at face value your reasons for any delay or for withdrawing from a review; there’s no need to elaborate or provide ‘proof’ (whatever that might be). We just ask that reviewers make contact!
4. The aim of this proposal is to promote timeliness. It would be only the most exceptional and egregious cases where the content of the report itself might warrant an embargo—and even then only in combination with tardiness.
5. One worry that has been expressed is that people will be more inclined to turn down requests to review. It’s hard to know what to make of this, unless one is really committed to (a) the ability to walk away from a promise to review a paper while (b) not communicating this decision with the editors. But anyway: One of the reasons we’re proposing this is because we actually want people to turn us down if they can’t realistically meet the deadline. And anyone who has reviewed for us in the past and found themselves in need of a reasonable extension will know that we always accommodate this.
6. Another worry is that the proposal is disproportionally harmful to early career researchers. Our internal rating system for reviewers suggests that ECRs are not the problem here.
7. The publishing industry is evil and we oughtn’t cooperate with it: (a) I guess this might eventually harm the publisher, but it will definitely harm the authors and editors (your equally unremunerated colleagues) in the meantime; (b) not all publishers are equal and the BJPS’s income supports the British Society for the Philosophy of Science, who funnel that money right back into the philosophy of science community, via PhD and conference grants; (c) if these reasons don’t motivate you, fine—but then please don’t accept invitations to referee!

Suggested alternatives:

1. We’d love to be able to pay reviewers (and editors!), but that’s not within our gift.
2. In the past, and with their permission, we have thanked our referees by name for their work (e.g. here). We stopped doing this, and maybe we should reconsider, but we assumed reviewers didn’t care because these annual blogposts received very little interest. Also, a hypothesis: those unmoved by the multiple, pleading emails of their (again, also unpaid) colleagues and/or the potentially precarious situation of authors will not be motivated by being credited in this (admittedly very limited) way.
3. Publishing the names of the reviewers alongside published papers creates its own problems. Others have raised examples of this. Here’s another: If I suspect that the author of the paper I’m reviewing is likely to be on a hiring committee I’m soon to face, I might decide to offer a more generous review. Submissions from more senior members of the community would then have a greater chance of publication. And from an editorial perspective, no editor should welcome a move that damages the credibility of reviewers’ reports.
4. Good reviewers could have their own papers ‘fast tracked’ through the peer review system. How? If it was within our ability to fast track papers, we wouldn’t be discussing this proposal.

Finally, there is presumably a reward for referees already in operation: a referee’s work is, in turn, reviewed by their peers (maybe in the same journal—in my original post, I mentioned that six people tend to any one BJPS paper that is not desk rejected—or maybe elsewhere).

Again, we’re very happy to hear more thoughts on this and to answer any questions I’ve left unanswered!