Our books






Become a Fan

« Non-standard publications and the market | Main | New series: Unconventional teaching ideas that work »

12/05/2018

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Anon

This thought is also off the cuff, but with the added constraint of trying to think of a particularly simple practice (Because I suspect philosophers would never converge on complex ones, but maybe I'm being too pessimistic).

So here's the thought: In your initial message to the referee, say that a non-response within 24 hours is a rejection (You can even put this information in the subject line of the email; at this point people who act like they aren't aware of the deadline are probably just being willfully ignorant). Immediately move on to the next referee if you must. Moreover, develop a list of likely candidates so that moving on to the next referee doesn't take a lot of thought.

Rationale for this thought: it gets us away from waiting for a long time for a potential referee to say "no" or to never respond, and for the editor to make the judgment call about when they should move on, etc.

Marcus Arvan

Anon: Thanks for weighing in!

I think your proposal may be better than nothing, but I'm not entirely sure. Given how slow people many people are with email, an editor following your suggested policy might often go through reviewer after reviewer--as person after person might just ignore the initial email, leading to a long string of initial emails that don't accomplish anything until the editor finally lucks on someone who *does* say yes within 24 hours. In which case the policy might not solve the problem, and indeed, make things even more frustrating for editors. But who knows, maybe it would work!

In any case, I guess I am more sanguine that a more complex system (such as the one I proposed) might work, particularly one involving philpapers/philpeople.

They have been very proactive over there developing the site as a central resource in the discipline--so my thought is that if a case could be made some central system there (such as the one I propose) had serious advantages for everyone involved (making editors' jobs easier, improving the process for authors, etc.), then perhaps they might actually try to develop something like it.

But who knows - I think it's probably a good idea to brainstorm both simple and more complex solutions, so thanks again for weighing in!

Anon

Right, I definitely think that the scenario you imagine could obtain, but that we lack reason to think it *probably* would, and so it's worth trying. I'm just tempted to see what this process would look like if the slow responders were cut out of it entirely.

But I'm also curious to see what other suggestions there are, too!

Marcus Arvan

Fair enough - me too! :)

Brad

Marcus,
Something like the system you describe already exists. I am an academic editor for PLOS ONE. The journal allows editors to search for reviewers, and the system is set to find matches with the paper under review, matches with respect to expertise. You get a near endless list of matches to choose from. And each person has their e-mail listed, and other relevant information.
Further, when the editor gets the referee report, s/he can rate the reviewer on a scale from 0 to 100. These ratings are averaged, so you can find something about people's track record. Further, there are details about how often the reviewer is late, etc.
This system, though, is designed for a large journal publishing many papers. Indeed, I believe there may be more than 5000 academic editors working for PLOS ONE.

Marcus Arvan

Brad: very cool - I wasn't aware of that! As an editor, do you find the system helpful?

One thing that seems to me suboptimal about the PLOS ONE system is that it leaves the up to the editor to complete at the end of the process. There are two potential problems with this. First, it can open up ratings to bias (such as if the editor knows the referee personally). Another more basic (and perhaps more salient) issue is that it leaves the rating until the editor receives the referee report. By that time, the editor may have forgotten how long it took the referee to respond to the initial referee request--and it doesn't base the referee's score in any way on how often they accept or turn down requests (which is important, as I've heard both contribute substantially to delays in turnaround times).

I'm inclined to think a better system would base an overall Reviewer Score on an agglomeration of several different scores:

(1) Initial response time (in accepting/declining)
(2) Proportion of accepts/declines
(3) Promptness of review time
(4) Quality of referee report

In the model I'm proposing, the central system would keep track of (1)-(3) automatically, and editors would only play a role in evaluating (4).

I'm also inclined to think a better system wouldn't just sort referees by AOS (providing editors a really long list of potential referees on the paper's topic). That still makes an editor's choice difficult, and would seem to me to make it liable for editors to try to always select the best referee they can find. My worry about this is that it doesn't incentivize better reviewer behavior--it might even punish the best reviewers by leading editors to select them first all the time. On the contrary, for reasons given in the OP I'm inclined to think that a better system would populate a potential list of referees for editors to choose from that is not only sorted by AOS, but also prompts editors to select reviewers who have a similar Reviewer Score as the author whose paper has been submitted.

In any case--setting aside whether my version of the proposal is optimal or not--what would you think of something like philpapers functioning as a central "referee recruitment" system like this? Not a compulsory system, mind you--but a system that journal editors (from any journal) could choose to use if it were developed?

Chris

I second Anon's suggestion in the first comment!

It could maybe be 48 hours instead of 24--to help prospective reviewers having a busy day. But beyond that, I think it's fine to move past the reviewer.

It's not a huge tragedy or injustice to miss a chance to review, and if some reviewers do not respond to emails quickly, they will learn over time that they need to do this in order to review papers.

Further, just as it is unethical to never accept requests to review, it would be unethical (on such a system) to never answer request emails within the allotted time.

Reading and replying to short emails quickly would become a basic requirement of the profession, just as many of us now require that of our students.

Brad

Marcus
PLOS ONE gives only 10 days for people to agree to referee a paper.
Further, the editor's assessment is intended to be of the report (and by extension, the referee, because the scores are averaged for specific referees)

Craig

Anon, Chris,

Speaking from personal experience, an email that lets me off the hook by doing nothing is going to make it vastly, vastly more likely that I do nothing. That might be unethical, but insofar as we're concerned about getting referees, and not just in blaming those who fail to referee, setting up a system which is likely to induce a much greater number of failures to referee seems like a bad idea.

Maybe I'm idiosyncratic?

Marcus Arvan

Brad: that’s really interesting. But I just want to clarify: is that a 10 day limit on answering the initial request (accepting the referee assignment), or is it a 10 day deadline to get the referee report in? If it’s the former, that sounds to me like too long. If 3 referees say no in a row, that could be 30 days before a referee is even successfully recruited to review the paper. On the other hand, if it’s a 10 day deadline to get the referee report in, I’d be curious how well it would work in philosophy—as I would worry that referees would tend to turn down most requests, on the grounds that they don’t feel they could meet a 10 day turnaround.

I’m just curious to get a better idea how PLOS One works, to get a better handle of whether it might be a good kind of model to centralize somehow.

Marcus Arvan

Also, I noticed by looking at PLOS One’s Wikipedia page that they have very different review standards (which appear to be unique to the journal). From what I can tell the sole criterion for acceptance is that the paper’s scientific methods are competent—not necessarily good or excellent. Their publication model is then to publish everything that meets this minimal standard, and to let academic audiences evaluate the paper post-review.

This actually seems quite a bit like the ArXiv model I’ve advocated on this blog, as it minimizes the role of referees and adopts a more “crowd sourced” approach to evaluating overall quality and importance of papers. The only real difference is that in the ArXiv model the crowd sourcing comes before peer review at journals, whereas in the PLOS One model the referees get to judge basic competence first and then the “crowd” gets to judge. And actually even this isn’t so different than the ArXiv, the the ArXiv has a minimal system of peer review for evaluating basic competence that must be passed for an article to be posted there.

Whatever the case may be, the kinds of standards referees are supposed to use at PLOS One seem very different than ones standard in philosophy, and so emulating that model would (as far as I can tell) require a pretty radical change of process and standards. In any case, I’d like to learn more, as I’d like to pin down some plausible alternatives to current peer-review processes in philosophy that might improve upon the issues those polled have with our current system!

Anon

Craig, thanks for the comment. My hunch is this: that right now, *already*, there's enough people never responding to the email that requests a response such that significant time would be saved by just moving on after a very brief initial waiting period.

I guess your specific worry could be handled by requesting a response in the initial email (rather than letting someone off the hook in the sense that you describe), but then, after not hearing back for 24-48 hours, explicitly canceling that request and moving on.

(I guess this discussion would be better informed if we had on hand statistics from a range of journals that speak to this issue.)

Wesley Buckwalter

These sound like interesting ideas. But I think a few basic things would help first. One is that authors could be encouraged to suggest reviewers with justification and contact details themselves, as in other fields. Another is that reviewers could be better instructed to write shorter reporters. People often tell me they are writing 1,000, 2,000 or even 5,000-word reports, which is wasting resources that could be better invested in the system.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Current Job-Market Discussion Thread

Job-market reporting thread

Writing Service


Categories