Survey results on peer-review

Several days ago, I put up this informal survey to gauge readers' opinions about various features of peer review. I am going to leave the survey up so that people can still take it. But here are the results so far (n = 91), organized by what I will call the good, the ambivalent, and the bad.

As this is only an informal survey, the results should of course be taken with a grain of salt (and then some). Still, I found the results interesting and will be curious to hear readers' reactions.

The Good

64.1% of respondents agreed that anonymized review works well on balance, giving authors a fair chance (whereas only 23.07% disagreed).

An even larger proportion of respondents 76%(!) agreed with that the peer-review process tends to improve their work.

68.4% agreed that the peer-review process improves their knowledge of other work in their field.

Next, a full 83.3% of respondents either disagreed with or neither agreed nor disagreed with the item that journal editors desk-reject too often. This suggests, somewhat surprisingly (to me, at least), that most people don't think desk-rejection is overused. Let

Respondents also pretty overwhelmingly expressed support for triple-anonymization, with 67.1% of respondents at least somewhat agreeing and 53.16% strongly agreeing.

Finally, respondents overwhelmingly supported journals sharing referee reports with all referees, with 84.8% of respondents at least somewhat agreeing. I find this particularly interesting given how few journals seem to utilize this practice.

In sum, respondents were pretty positive about some features of peer-review. Let us now turn, however, to...

The Ambivalent 

Despite the results reported above, only 37.08% of respondents at least somewhat agreed that they are satisfied with current pee-review practices, whereas 53.9% of respondents either somewhat disagreed or strongly disagreeing. This suggests a bit more dissatisfaction than satisfaction with current peer-review practices across respondents.

However, looking at the chart below, the results as a whole suggest to me some real ambivalence across respondents. For the results appear bi-modal, with roughly equal proportions somewhat agreeing (33.7%) and somewhat disagreeing (38.2%) with them. The results that tugged this item in the "bad" direction was the greater proportion of people strongly disagreeing (15.7%), compared to only 3.37% strongly agreeing.

Given the features of people rated positively above (viz. anonymization giving authors a fair chance), what are the sources of the moderate negativity / ambivalence we see in this item?

First, respondents appeared fairly ambivalent about the typical referee comments they receive. Although 50.6% of respondents at least somewhat agreed that their referee reports tend to be sensible and fair,  a full 49.4% either disagreed or neither agreed nor disagreed.

A similar form of ambivalence was exhibited in response to the question of whether referee reports tend to be too perfunctory. Here 46.8% agreed with the item, but 53.16% either disagreed or neither agreed nor disagreed. In other words, quite a few respondents (just a little less than half) think overly perfunctory reports are a problem, but a little more than half do not.

People also seemed pretty ambivalent about whether too many bad papers are published. Roughly equal proportions at least somewhat agreed (39.7%) or somewhat disagreed (42.3%), with the remaining 18% neither agreeing nor disagreeing.

People were also fairly ambivalent about whether more should be done to ensure anonymity. Although 55.7% at least somewhat agreed, that's just a bit over half of all respondents, the others of whom either disagreed or neither agreed nor disagreed.

Respondents were also fairly ambivalent about revise-and-resubmits, with some agreeing that R&R's needlessly swamp the system but a bit larger proportions disagreeing:

Respondents were also a bit ambivalent over whether editors are too conservative. Although a 51.9% agreed that editors are too conservative (and 24% strongly agreed), the other 48% either somewhat disagreed or neither agreed nor disagreed:

Finally, respondents were fairly ambivalent on the whole to the item on whether a peer-review journal has 'lost' at least one of their submissions. Although a fairly disturbing proportion (30.4%) at least somewhat agreed with this item, nearly 70% disagreed or neither agreed nor disagreed.

Let us now turn to...

The Bad

The issue respondents focused on is not too surprising. A full 93.67% of respondents at least somewhat agreed that 'long and inconsistent turnaround times' are a problem (with 67% strongly agreeing):

A fairly large proportion of respondents (62.5%) also agreed that too many reviewer reports are incompetent, biased, or needlessly aggressive:

People also thought reviewers are too conservative philosophically (60.76% agreeing):

People also thought too many half-baked papers get sent out for review (62.8% agreeing):

Respondents also overwhelmingly agreed that journals could communicate with authors better (87.3% agree, 54.4% strongly agree):

Finally, respondents tended to agree more than disagree that the peer-review process is too opaque and should be more transparent (54.4% agreed, 29.11% strongly agreed, only 13.9% disagreeing)--though 31.6% neither agreed nor disagreed.

Again, these are only informal results. My main aim is to use them as a jumping off point to discuss these issues more in future posts in this series. But what does everyone think? Any reactions?

Comments

[Pendaran Roberts]

I'm happy to see that so many agree with me that 'too many reviewer reports are incompetent, biased, or needlessly aggressive.' I've gotten some real crappy reports over the years. I don't understand why they're not thrown out entirely. Can't editors tell when a report is just ridiculous?

[Malcolm Keating]

I imagine that a survey which looked at AOS would also be helpful, in case there are trends within sub-fields. (One could then perhaps infer what journals are being described, though getting that information directly would be more reliable.)

I have had generally positive experiences on the receiving side of referee reports when submitting papers in Indian philosophy. I have experienced a sense that referees want to improve the work and encourage growth, and not to simply disagree aggressively or self-promote, etc. Perhaps this is because of the relative small community of those of us working in Indian philosophy. It would be interesting to know if this experience is shared among others doing Indian philosophy or if it is a feature of something else.

[elisa freschi]

@Malcolm: As a reviewer in the field of Indian philosophy, I am glad to read that, Malcolm! I wonder whether in the specific case you mention many reviewers may think, like I do, that scholars working in the field needs to be helped and supported, since there is so much to do and so few people doing it.

[Malcolm Keating]

Elisa, that is my suspicion. It's certainly how I have approached reviews myself, when I write them. (Though I can't speak to how people perceive my reports, I usually write at least a page or two starting with my own understanding of the argument.)

I also wonder, having refereed for journals on the linguistics/philosophy of language boundary, about disciplinary norms there (Journal of Pragmatics, for instance, shares reports with all referees, and has a pretty tight turnaround time). Thus I really think talking about "philosophy journals" in general is not going to answer much, unless the tacit assumption is we're discussing the top-ranked generalist journals--in which case we should be focusing on those journals.

[Andy]

I think the point about different subfields having different norms is a good one. I boundary cross quite a lot and, at least in my experience, I have found that there is a reasonable degree of variation in referee comments between subfields of philosophy.

[Marcus Arvan]

Malcolm (and others): Thanks for raising this issue. I too have heard that behavioral norms in different philosophical areas can be very different. And I agree it might be worthwhile trying to learn more about what they are. This could not only better inform early-career philosophers about which AOS they might choose to go into, but also lead to good discussions about which areas have norms to emulate and which areas have norms in need of improvement.

I'll try to put together some kind of poll on this soon!

[Amanda]

I work in very different areas of philosophy, and I do think the norms are different. MandE and main stream ethics seem especially harsh, in my experience. On the other hand certain sub disciplines of ethics seem much better.