So Now You’re a Referee…
Beth Hannon
It hardly needs saying that referees are essential to the functioning of journals, and the discipline as a whole. Refereeing a paper is a service to the academic community. Those that take this duty seriously don’t just help the editors and the authors; we all benefit from having published papers be as polished as they can be. I’ve written before about the fact that the production of excellent papers is by no means an individualistic endeavour. It takes an academic village to raise a paper! And we all know how busy everyone is, and how refereeing has to be managed alongside all the other teaching, research, and administrative duties that demand attention. All this is to say that we in no way underestimate the hard work done by our referees; on the contrary, we are very grateful indeed.
We do, however, find that some reports are not as helpful as they might be. We have also received queries from early career academics about what constitutes a good report. So, to answer these questions and more—so much more!—we will spill our thoughts on what it takes to be a diligent and helpful referee. More good and helpful advice and discussion from the intertubes can be found here, here, and here.
Perhaps the best way to start is to lay out what one definitely shouldn’t do, and it mostly relates to very simple, practical things that nonetheless can save headaches, heartache, and a world of fretting for authors. Delays in getting decisions to authors are first and foremost caused by referees failing to do one or both of the following:
1. Respond promptly to invitations to referee. Seriously. A paper is left in limbo if you do not let us know whether you will act as a referee. It’s often hard, so very hard, to find an appropriate referee for a paper. This means that we are unlikely to give up and move on to another person. Thus we’re going to hold out for a response from you for as long as possible. No response means the paper just sits around gathering dust and looking at us with accusing eyes. Even if it’s just an email with a ‘yes’ or ‘no’ (this happens), we can keep the paper moving through the system. Even better, these days the BJPS has simple links in our invitations you can click to indicate your willingness or otherwise to referee. You don’t even have to write to us! It couldn’t be easier!
Once upon a time, in the dim and distant past, it was acceptable for papers to linger for unspeakable amounts of time with journals. Those days have passed people. Early career academics in particular cannot afford to have their work benched like this.
2. Relatedly, please try to return your report within the time agreed. We all understand if you go a few days overdue, and emergencies happen from time to time to everyone, so don’t be afraid to let us know if this is what’s going on. The worst thing possible is to disappear off the radar and not respond to our emails. If the report is going to be late, but we know it is coming, that’s something we can work with. On the other hand, if our emails receive no reply, we can’t tell whether the report is in progress or if said report will forever remain notional. Again, as we don’t want to give up on a referee once we’ve found him or her, we will hold out for as long as possible before starting the search for a new referee. Think of editors as limpets and referees as dear life. A worst-case scenario is one where a referee, after much delay, agrees to referee a paper, but doesn’t return the report on time and doesn’t respond to emails, so that we are eventually forced to find another referee. This can add months to how long a paper is with us. Won’t somebody please think of the kids early career academics?!
So now that you have mastered how not to be a referee-jerk, what about the content of your report? The first thing to think about is the function of the report. It’s wonderful when we get reports that include thoughts and advice that really help authors improve their papers; but the first job of any report is not to help the author write a better paper, it is to advise the editors on how to handle the paper. With this in mind, here are some things to consider:
1. The journal approached you because they think you’re ace at what you do, and you are likely far more ace at it than the editor issuing the invite, who will know the area but tend not to specialize in it to the degree that you do. So what is really helpful is if, first, you can summarize what the paper is doing. This involves a synopsis of the argument, but also a bit of contextualization. Does it speak to current debates in the literature? Is it a particularly novel argument? Does it push the debate on in interesting ways? Lots of papers can be broadly right; we’re interested in papers that are right in interesting and original ways.
As a general point, it’s worth considering the journal you are reviewing for. How does this paper fit with what’s gone before in that journal? Do spell out your thinking on this point. The editors are the final arbiters of what is and isn’t appropriate for the journal—they may well have decided to take the journal in another direction or just branch out into other areas—so while you should definitely offer your opinion on the matter, let the editors know that this is the reason for your rejection (for example) so that they can weigh this against their own editorial policies.
2. The most useless report reads as follows:
‘This paper is great! Publish as it is!’
As editors, a report like that makes us very antsy and it rather feels like we are flying blind. We have no sense that the referee has given the paper due consideration. (It doesn’t help that when we’ve had a report like this, it’s usually accompanied by another referee advising major revisions with five pages of detailed comments.) Some papers, of course, are better than others, and some arrive with us in an extremely polished state. But even if a paper really is without fault, I return to the first point: flawless it may be, but is it interesting? If you have no criticisms, at least spell out a positive case for why it ought to be published.
3. Although the report is first for the editors, it hardly needs saying that they can be very important for authors too. Getting the tone right can be tricky: referee reports can seem unduly harsh, and may discourage authors who are post-graduates or in the early stages of their career. This is especially unfortunate as it often takes only a little re-orientating to make a review sound more like constructive criticism than character assassination.
First, it’s best to address your report to the editors. As they are the reason you’re writing the report, this just makes sense. Beyond this, talking about the author in the third person can give you some distance from the frustrations you might have felt in reading the paper, and often takes some of the immediacy and sting out of comments for the author. Second, talk about what is wrong with the paper, not what the author did wrong (e.g. ‘this passage was confusing’ not ‘the author is confused’). That is, the paper should be the subject of your report, not the author. Granted, sometimes as a referee one can feel like pulling out one’s hair. Vent your frustrations in private comments to the editors, not in the report that is to be shared with the author.
4. Journals can differ in how they interpret recommendations such as ‘major revisions’ and ‘minor revisions’. Don’t worry too much about this; your report will usually make clear what you have in mind. If you do decide revisions are necessary, you will need to provide concrete details of the revisions you would like to see. Editors cannot ask authors to revise their papers unless they can also issue authors with details of what needs to be revised and why. Advice such as ‘Section 2 is weak’ or ‘Section 5 is unconvincing’ is not something an author can really work with. Of course, no one expects a referee to re-write the paper for the author, but if there are problems with the paper, they need to be clearly identified if a revised paper is ever likely to see the light of day.
5. These are thankfully not too common, but there are a class of reports that are less about the author and the paper, and more about the referee’s own work and research interests. Reviews that suggest that a paper is interesting, but ought to have cited a selection of the referee’s papers are not much use. There are times when a referee’s work is relevant—this is no surprise given referees are chosen because they work in the area—and if the paper has overlooked a lot of relevant literature then this is clearly a problem. But a report that mostly details why one or a few papers written by the referee ought to have been cited doesn’t help an editor make a decision about whether to publish the paper. Less egregious than this last example, but more common for that, is the report that amounts to ‘this isn’t the paper I would have written’. The paper may not have used your favourite case study nor taken the direction you might have, but these are not prima facie problems with the paper as a piece of philosophical research.
6. A slightly more tricky issue concerns the difference between the sort of problem that ought to be addressed during the peer-review process, and the sort that ought to be thrashed out in print. There are no hard and fast rules here, of course. However, some guidance can be found in the thought that though you may believe a line of thought is completely wrong-headed, if it has some traction in the philosophy of science community, then that’s probably not reason enough to reject the paper.
7. Finally, some people like to sign their reports and the intention behind this is admirable. However, it’s not always practical. If a paper receives a ‘revise and resubmit’ decision, we may ask the referee to look at the paper again. This means we need to keep protected the identities of referees, author(s), and these days at the BJPS, the editors too. (All well and good if you submit your report directly into our online system or attach a word document—we can easily edit those to remove identifying information—but a pdf is a good deal more hassle! If you want to include your own details, at least do so in a way that the editors can easily amend if necessary.)
If you can’t referee a paper, please do suggest other people we might approach instead. Again, you’re ace at this stuff: you know the area and you know the people who work in the area (and who do so well and who do so badly!) better than the person asking you to referee the paper. You, of course, are a busy person with an academic job or PhD research to attend to; but do bear in mind this is usually true of the person emailing you too. Won’t someone please think of the kids early career researchers exhausted and bedraggled editors!
And if you are, like so many of our referees, someone who writes detailed reports, without near-libellous content, and returns them in a timely fashion, then we salute you, unsung heroes of the profession!