A solution to four problems with peer-review?

Matt McKeever (Assistant Editor, Inquiry) has an interesting post on Medium suggesting a simple reform to peer-review that he thinks might solve several problems. McKeever writes:

Here are three [sic] problems with peer review:

(1) It’s a (mild) hassle to submit a paper, requiring the navigation of (mostly) poorly designed online systems, the writing of otiose cover letters, and so on. Since most papers don’t get accepted at the first place they’re sent, this mild hassle is multiplied several times.

(2) Once the editor has received a paper, they are deprived of a very important piece of information about the paper, namely whether and by whom it’s been reviewed before and, if so, what the reviewers thought and how the author responded.

(3) A referee’s work is often in vain: people will submit, a referee will work hard to point out problems with the paper, the paper will get rejected, and the author will submit again elsewhere relatively confident that the referee will not review it again.

(4) It is hard to find referees.

I think a system aimed at solving (1), the least serious problem, could help with the other three problems: in brief, the proposal is a journal-neutral submission system (call it JNSS), a single front-end from which authors can submit and resubmit a given paper to different journals without having to navigate individual sites or redundantly enter the same information numerous times. Onto this front-end an editorial backside could be added, that contains information about the paper’s history and is accessible to only and all journal editors, enabling them to see who has already reviewed the paper and what those reviewers thought of it.

Readers should check out the rest of McKeever's post to see how he thinks this might work. For my part, I think a substantial amount of McKeever's proposal might be a good idea. For instance, I've heard (though I may be wrong) that law journals use something like his journal-neutral submission system--which I think might make things easier on authors, editors, and referees. I also like McKeever's suggestion that authors who submit papers should have to agree to be included in a pool of referees--as a central pool (categorized by topics and author AOS) might make it much easier for editors to find reviewers.

However, I am much less convinced that it would be good to forward past referee reports to editors at new journals, or to give editors at journals inside information about journals a paper has already been reviewed at. My worry is that this might bias editors against papers that have been bouncing around a while--papers that may well be very good. I don't think we should hold a paper's past against it--particularly given that its author may substantially revise and improve it. For these reasons, I'm inclined to think every new paper submission should be treated as a "clean slate."

But these are just my thoughts. What are yours?

Comments

[The Ref]

I do not like the suggestions. First, I do not think we need more centralization. The system - if it is a system at all - works well because there are many different venues to send papers to. Second, I do not think we should streamline the submission process. In fact, we should make it more involved so people have to be thoughtful about what they submit. I remember posting papers to journals. You did not just send in a paper with little thought of what happens next. I just read a news story about a banking app that has helped people with gambling addictions. It basically makes you wait 48 hours before you can place your bet. By then the urge has passed. People have managed to break out of the cycle of gambling addiction. Philosophers need to break out of the cycle of submission addiction.

[anonymous]

I also really don't like this suggestion! Submitting a paper to a journal is really not very much work. It usually takes me about 20 minutes. And, while I don't agree with everything The Ref says, it seems to me that people (by no fault of their own--this is the fault of the publishing arms race) are submitting *way* too many papers to journals. So I'm not really sure why we should make it even easier than it already is to submit a paper.

I also am strongly against editors seeing previous referee reports. Most of my papers have taken 6-7 submissions to get published. I think my papers are good. Many people tell me that they think they are good. (I also have a sense of why it is hard to find a home for them, but that's not important here.) If editors could see previous reports on those papers, they would see a series of referees absolutely trashing my papers, over and over again. I do not think I am alone here.

Journals reward "safe" work that is written in a very specific style. I think that this system would enforce this even more, since it often seems to be the referees who are doing the gatekeeping. It would also be dangerous for people who work interdisciplinarily (either within philosophy, or external to it) who got referees who were unsympathetic to or ignorant of some of what was important about the background of their work.

[Pendaran]

I agree with anonymous. Many of my papers have taken many submissions to get published. Often reviewer comments are not worth considering. No reason to burden editors with comments that the author obviously doesn’t find worth it.

[Captain Obvious]

Suppose I individuate submissions finely enough that S ≠ any referee-provoked revision of S. (What's to stop me? Or anyone else who uploads a submission after previous bouts of peer review?) Well, then whoops there goes the business of alleviating problems (2) and (3).

On the other hand: unlike other proposals being floated at the moment, at least this one isn't blatantly discriminatory, illiberal, and unscholarly.

[Amanda]

Yeah, I do not agree at all about forwarding referee reports. Speaking of that, whenever I review a paper now they ask if they can forward my reports to another journal. I always say "no", for I do not think it is fair to the author, among other reasons. But why do they ask this? Are some journals actually requesting reports from past journals?

So I have a new problem with all this discussion of how to change peer review. People come up with suggestions of various types, and nothing ever changes because the suggestions do not consider whether any changes are feasible. I now think most discussions (there is room for some theory) should take into account that journals are independent entities, and that they are not going to follow any centralized order. It just won't happen. So let's work with that.

[Matthew McKeever]

Thanks, people who commented! I'm glad to get a sense of what (some) people think about the proposal.

One thing: my post has been taken as suggesting we make it easier to submit papers, and this has been taken to be a negative thing. While this is kind of true, it's not the whole story: the envisaged system is also meant to make things harder because after a paper has been rejected, you have to do some reviewing.

As to the undesirability of forwarding referee reports, I certainly take people's concerns on board, and see the value of the 'clean-slate' approach. But I can't help but think that the more perspectives on a paper the better. An editor should feel more confident about a paper with 2 than with 1 report, with 3 than with 2, and so on, but since it isn't feasible to get many reports for each paper, I thought a way to get around that would be to look at previous reports.

In any event, thanks again for the feedback!

[Pendaran]

As things stand now you can get a paper published if you can find 1 or 2 people who like it. This allows for unpopular ideas to make it through the censor. It might take a long time, but it's possible for those who don't give up. I worry more reports will scare editors away from publishing unpopular ideas. The idea that increasing the number of reports will allow editors to better determine quality is suspicious and I'd even say, deeply mistaken. No great idea has ever been accepted, at least not at first, by the majority of one's peers.

[Paul]

I agree with the other comments that due to so many trivial, idiosyncratic, and brief reviews, I am not sure that being able to see the content of those reviews would be helpful. But I am inclined to think that knowing who had already reviewed a given paper would save some time for the editor, and I definitely think that if you submit to a journal you should be willing (and obliged) to review for them (assuming the editor wants to use you). If everyone that submitted agreed to review, then it should not be a problem finding someone with expertise in the area to review the paper that hasn't already done so.

The main problem I see with the editor being able to see the names and number of previous reviewers is that the higher the number (and if the editor knows any of the reviewers) could bias the editor. But first, editors have to know that most articles go through (at least) several rounds of review before being published, so the number of reviewers shouldn't be a problem. And second, since most editors don't blind review, they are already potentially biased against the author. And here is where I agree with Marcus that the more transparency the better...

[Cover letter?]

On #1, I didn't think that anyone actually used the 'upload a cover letter' feature. I've never been sure what to say beyond "hey, this is a paper that I want you to publish, and I prepared it for blind review," but I already check a box (in most online systems) that says all of that.

Do people actually upload cover letters or send additional comments to the editor when submitting a paper?

[re: cover letters]

I mostly do the same and my impression has been that cover letters are a bigger deal in other disciplines (e.g. I recently submitted to a non-philosophy journal that had detailed instructions about what the one-paragraph cover letter should contain, that was sort of a cross between the abstract and the introduction of the paper). But maybe I've been doing it wrong!