Our books






Become a Fan

« A little milestone | Main | Sexual Assault and Citation Practices »

11/14/2018

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Anon

This is an issue that's already been discussed on this blog, of course, but: the biggest problem I'm experiencing right now is time to complete review of papers.

I have one paper that has gone eight months without an initial review (I will probably just get around to withdrawing this one soon, but no one's replied to my inquiries, and I feel a bit worried about some report coming in after I withdraw). I have another paper getting up to six months post submission of revisions.

do I really want to wade into this

Marcus
I am curious to know where you will be speaking about this issue.

Sebastian Lutz

What currently (sometimes) works well

1. Sometimes you get a friendly reviewer who, even when recommending rejection, gives sensible, thought out feedback based on your (actual) arguments.
2. Some journals have established deadlines for reviews. While not all reviewers meet those deadlines, I think this has improved review times a lot.


The currently biggest problems

3. There is little if any incentive to actually do good work as a reviewer besides moral satisfaction. Even publons.com only keeps track of the number of reviews, which is at best a motivation to not be so terrible that you are never asked again. I would love to have an evaluation system for reviewers by the reviewed authors (probably best partitioned into rejected and accepted).
4. Something of a result of the above and the inverse of point 1: Some reviewers seem to read the conclusion first and then write their review based on their opinion of the conclusion. Sometimes this lets weak arguments get published, sometimes this lets strong arguments get rejected.
5. A corollary of the above: The review process is very scattershot, and some journals will never accept specific papers simply because the papers will be send out to the same unfriendly reviewers.
6. Too many half-baked papers are submitted to too many journals. I assume this is the result of publication pressure.
7. A corollary of the above: Editors and reviewers are overworked, which increases false positives and false negatives in journal acceptances.
8. A corollary of the previous two points: The ratio of crap to good papers increases, first, because the number of half-baked submissions increases and, second, because the false positives go up. The increased false negatives won't make as much of a difference.

William Peden

My own experiences have almost all been very positive. I would particularly point to detailed feedback, encouragement, and only one "This author has overlooked the brilliant X on this topic Y [obviously their work]."

From friends, I know that there are major problems with (1) timeliness and (2) pointless negativity by some referees.

If there was just one change I could make, it would be somewhat more detailed information for authors when an article is being delayed, e.g. "Our first choice of referee was unable to do the review." While fundamentally insubstantial, I think this would relieve a bit of easily avoidable stress for authors, especially for when articles take 12+ months to get back.

Also: in extreme cases, when editors don't even acknowledge the delay, maybe we should set a norm that e.g. 12+ months under review is long enough to wait, and the author can start sending the article elsewhere? I'm not sure about this proposal, as it could end up wasting editors' time and they're overloaded already.

Amanda

Problems:

(1) Many journals are double blind, not triple blind. This means the editor can just desk reject a paper for the wrong reasons, or send the paper to a reviewer who they suspect will reject/accept a paper according to the editor's own opinions. The editor also has authority to simply reject papers in spite of positive review reports - this is problematic when the editor is not blind to the author
(2) Even in triple-blind review, many papers are not blind review at all, because almost all experts had already seen the paper before at a conference or read/talked about it online. May reviewers admit to reviewing a paper when they know the author, and editors often approve.I see no solution to this problem, and what bothers me most about all of it is still calling the paper 'blind review.' The more famous and/or connected one is, the less likely the paper is to actually go through blind review.
(3) Everything takes way too long.
(4) Reviewers have *no* incentive to do the review well, and so often they do not.
(5)Journals are intellectually conservative, i.e. controversial papers seem more likely to get rejected. Likewise, the journal process seems to favor careful, boring papers over big, new, and interesting ideas which inevitably have mistakes.
(6) The "top" journals tend to favor traditional metaphysicians and epistemology, and certain kinds of highly abstract ethics. There are lots of great papers in other sub disciplines that have little chance of getting published in the most recognized journals.

Anon

"I have one paper that has gone eight months without an initial review (I will probably just get around to withdrawing this one soon, but no one's replied to my inquiries, and I feel a bit worried about some report coming in after I withdraw). I have another paper getting up to six months post submission of revisions."

Here's a suggestion: wait until reviewers have been found, then withdraw it. And submit it elsewhere in the meantime. That way, the journal gets some small token of punishment (i.e. finding refs for no reason) for their bad behavior.

I had a journal take a year to review. They gave me an R and R but didn't respond to my emails (such as my email asking: hey, if I do this R and R, is it going to be sent to the same lously ref who took a year to review?), so I just declined to revise. They suck.

elisa freschi

1 As an author:
1.1 It teaches one humility.
1.2 At times, I got really good advice (I have to admit, however, that I also got great advise through sessions on Academia.edu).

1 As an editor of peer-reviewed journal issues and a supervisor of PhD students:
1.3 It is a great way to explain to students and younger colleagues that they have to be, e.g., clearer (since the reviewer did not understand X, you just have to explain it better!).

1 As a peer-reviewer:
1.4 At times, I reviewed really horrible papers, papers I would not have thought people would have even dared sending to a journal. One needs a filter and peer review is better than no filter at all.
1.5 It gives a fair chance to many who do not come from prestigious universities etc. I could review, help improving and finally accept articles from young colleagues who did not have English as their mother tongue and/or lived in obscure parts of the world. Which journal would have accepted them without some reviewers vouching for the quality of their contributions?
1.6 It made me aware of colleagues whose work I would not have ever read.

2. As an author:
2.1 The time needed to find referees is at times just too long (I am also among the ones waiting since months and getting no information).
2.2 The fact that at times I get referees asking for contradictory things and the editor is not brave enough to step in and tell me that they understand I cannot do both and that I should rather follow B instead of C.

2. As a peer-reviewer:
2.3 The fact that at times I invest time and energy only to then see the article accepted after minor modifications, so that the process seems at least opaque.
2.4 The fact that at times authors reply in an annoyed way as if they were artists who had not been fully understood in their genius.

Pendaran Roberts

I started trying to publish seriously in 2014. I eventually was able to publish most of my ideas (still waiting on some papers). So, I think, from my experience, that a positive about our system is that if you've got a novel and interesting contribution and don't give up that you can publish it, eventually. The biggest negative about our system is that it's incredibly inefficient. About half of the referee reports I've received over the years were at best borderline incompetent and at worst borderline negligent or dishonest. This slows down the publication of good papers tremendously. Often you end up with that 1 referee that just won't be reasonable or just can't seem to understand the merit of your work.

Some of my papers were accepted almost straight away, but some have taken years. In particular, an experimental philosophy paper of mine, which brings X-Phi to a new area in an interesting way, is apparently too controversial to be easily published. I first submitted it in Jan 2016. It's still going around review land. I've had a total of 5 reviewers accept the paper but haven't lucked out with getting enough reviewers at one journal. At one top 10 journal, 2 out of 3 referees accepted the paper, and it was still rejected because of one overly critical referee. So, the system is delaying the publication of this work for years, while others who wanted to engage with the work are having to wait on the system to function and publish something that should have been published a long time ago.

O.

(1) I'm fortunate to have had really good experiences as an author, with really thoughtful, helpful, interesting reviews that have made my work better. I think I've felt more confident in responding to reviews and in writing reviews because I don't know who the authors or reviewers are. I imagine more than a bit of self-doubt would creep in if I knew that I was responding to some super-famous person (I am not super-famous, to put it mildly).
(2) Many of the problems are connected: too many submissions, too few reviewers, too many delays. Ultimately I think the solution will involve doing something about the unfortunate focus on quantity of publications as the only real measure of scholarly contribution. One small suggestion: encouraging ways to have referee work and various modes of responding/ replying to journal articles and books be counted more in assessing one's scholarly work. These are real, valuable contributions to scholarship; publishing articles is great, but it's not the only way.

Random job market person

Best thing:

1. Some editors are great. I think many editors are sensitive to the plight of both referees and authors and really do their best with what they have got.

For me the biggest problem is shoddy refereeing:

1. Rejection of papers for absolutely absurd/trivial reasons.

2. Failure to even properly read the paper, to the point that papers are rejected for doing things which are very clearly and demonstrably not done (I have had reports where I think it is fair to say the referee outright lied about the paper. More often I think it is simple failure to properly read the paper though).

Peter Furlong

I have had the same experiences as others above. Sometimes I get great feedback, sometimes it is ridiculous. I should note that this does not always correspond to how positive the referee is. I have had someone recommend accepting a paper, but some of their comments made me think they really didn't understand the area I was working in. Of course, some of my rejections I though were for stunning reasons too. Much of the time, though, I think feedback (whether positive or negative) does reveal that the referees carefully read the paper and understood the sub-field it was in.

One thing that I don't think any other comments have mentioned is the phenomenon of papers getting lost somewhere in the process. I have not been submitting papers for that many years, but I have already had two papers lost. In both cases, the editors admitted that they had lost the paper. In one case, after repeatedly contacting the journal, the editor responded noting that the paper (which had been submitted more than a year earlier) had not even been sent out to review. In the other case, I contacted the journal and they responded noting that the referees had given their approval, and the decision was made to accept the paper, and somehow between this stage and informing me of the decision, the paper had fallen through the cracks.

I appreciated the honesty of both editors in admitting the errors that had been made, but it made me wonder whether this is even more common, whether papers get lost and, when this is noticed, editors blame this on slow referees or such.

In any case, I am not quite as pessimistic about the system as some others are. I do think that the system does sometimes operate blindly, does give non-famous authors a chance, and does help filter out some work that isn't quite ready for publication.

I also think that some of the problems with the system can be fixed from the inside, with changes to the policies and cultures of particular journals. Still, I am open to ideas that are a bit more radical.

Chris

One thing I think could be improved is adopting stricter ethical norms for reviewers who seek to violate blind review by (for example) Googling the title of a paper they have received.

There may be no feasible way to police this practice or punish those who do this, but I think we could as a profession make our ethical expectations more clear, that this is not just a type of human weakness to be tolerated, but that it is a serious transgression of academic integrity.

For example, maybe there could be an honor pledge reviewers sign, acknowledging their commitment to blind review and the seriousness of its violation.

Peter Furlong

I think Chris's suggestion is fantastic: I think it would be easy to implement, has no down side, and has the potential to reduce the number of Google reviews. Obviously it would not completely solve the problem, but I think it would help, at least a little.

Matias Slavov

(1) I feel the same way as O.: fortunate to have received many good reports. They have improved the quality of my work. Only a couple of reviews have been incompetent.

(2) Submissions take in general too long to process. Things are changing. Good examples are journals like Ergo and Thought. I submitted a paper to Ergo, and I got very good comments in two days. Although the paper was rejected, I followed the editor's suggestions, submitted it to a next good journal, and got the paper accepted.
There should be a quick screening by the editor (a few weeks at most), and if the paper is likely to be accepted, it should be sent for a review (that should not take more than two months).

Amanda

While Chris's suggestion might help, I want to stress that what I think is the biggest problem to blind review violation (and, at least the least, a significant issue) is not people googling papers but people already knowing who wrote the paper just by looking at it, because they had seen it before. Often people then write to the editor, tell them they know the author, and the editor says to review the paper anyway. I can't think of any real way to address this issue, other than reviewers and editors turning down the option to have a non-blind review. But it already takes sooo long for papers to get reviewed, and eliminating all the times it isn't really blind review would make the process even longer.

Nick

1- Works well: I find it beneficial, as a reviewer, when the journal shares its verdict with me and also sends along the reports from other referees.

2- Works poorly: turn-around time for verdicts.

Brandon Beasley

1. Works well:
When the time-to-decision is reasonable; when you get good, helpful, and charitable comments; when it's clear your reviewers actually read and tried to understand your paper; then the system is pretty good!

2. Works poorly:
The frequency of the lack of some or all of the above. Decision times are too long. Reviewers who clearly did not really read or try to understand your paper rejecting it. You get no comments or perfunctory/worthless ones.

Also, relatedly both good and bad: I think the desk rejection policy of lots of journals should be adopted more frequently. It's so much better to get a "No, sorry, not right for us/not up to snuff" rejection in a week or two then to wait 3 months and get a rejection with worthless comments which show the reviewer(s) didn't really even read the paper enough to engage with its main points.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Current Job-Market Discussion Thread

Job-market reporting thread

Writing Service


Categories