Why referees can spot what's wrong in your paper, but often can't tell you how to fix it

Today, I was watching Neil Gaiman's fiction writing master class, where Gaiman offers advice for how to edit your fiction. A lot of this could just as well apply to non-fiction writing. For example, Gaiman says that your first draft is for your eyes only. If you didn't leave it too late, let the first draft sit for a week and then go back to it. Try to read it like you are reading someone else's work, not too harsh but nevertheless, a disinterested person who might advise you to cut where you might be reluctant to cut. 

Once you've done the second draft, the piece might be ready for some other readers to provide comments. I find it useful to find someone to read my work before it goes out for review. By this, I don't just mean to present the work at a conference or a colloquium, but someone who actually reads the piece as it is written. I've had so many talks that worked beautifully as talks, but their energy and persuasiveness just didn't translate on the page and they got rejected over and over. One useful way to solicit a review is to do a swap.

But it's not always possible to find an external reader, and that's when you'll have to rely on referee reports from the journal you submit to.

Some people tell me they find referee reports completely useless and they don't consider at all what the reports say and just send the paper on to the next journal. I find that discomforting, as someone who does a fair (but not enormous) share of refereeing I put a lot of care in my reports. Maybe I am just more lucky with referee reports than many people are. Even referee reports accompanying a reject decision are often useful for me to further improve the paper. For other papers, the reports are indeed useless. Those are typically the reports where the reviewer wants to see the paper go into a direction I just don't see it going. 

Anyway, Neil Gaiman helped me realize that even in a seemingly not-useful referee report, there is some valuable knowledge you gain (unless the reviewer was just inept, or inattentive, this sometimes happens, but don't assume it too soon!). Namely, if the reviewer says "this didn't work for me" or "I didn't get it", they are always right. They are always right, because indeed, the assertion depends on the reviewer's reception of the work. If they didn't get it, or it didn't work for them, that's valuable information. A lot of redrafting, or excessive familiarity with the topic, or being neglectful to spell out a line of thought can lead a referee to be puzzled. Thus, a referee who says "I'm puzzled here," is giving you valuable information. That referee is saying that there is something puzzling there (to at least one person). It's something you need to fix. 

Obviously, a reviewer might be idiosyncratic but if the report looks otherwise coherent remarks that convey "I didn't get it", "It didn't work for me", or "you need to do more work to establish that p", you need to take this very seriously as an author. Unfortunately, reviewers might be hesitant to tell you that they didn't understand a given line of thought, so look for formulations along those lines, expressing puzzlement and lack of understanding. Read those passages again with a dispassionate mindset, and try to make it clearer.

Gaiman mentions that though the reader is always right if something doesn't work for them, or is unclear, the reader is very often wrong in their advice on how to fix it. In rare cases, the reader has hit upon the right solution, in which case you say "Oh that's a brilliant idea, I'll do that, thank you!" But often they don't. The reader is not as much immersed in the story, its characters, its plotline, as you are, and so their advice is often wrong. Gaiman recommends that you take seriously when a reader flags the story doesn't work for them, or is not understandable, but that you should not necessarily take their advice to heart on how to fix it. Often, completely different solution is better. 

It seems to me this often happens with referee reports too. Referees will point out holes in your argument, places where the argument is unclear, places where the argument goes too quick and where they lose the line of thought. But the solution the reader recommends is not always ideal. This is, I think, where the responses to editors are very important. You can there say something to the effect that you thank the reviewer for spotting a problem with your paper, and then explain why you fixed the problem in a different way than the reviewer recommended. 

Since we don't want to lose the paper in the revise and resubmission process pushing back like this is somewhat scary, and we are tempted to placate reviewers, to go with their solutions even if we have doubts and thank them for what basically made the paper more complicated but not necessarily better.

I think this is unfortunate. As a reviewer, I try to forestall this by saying something like "The following is advice that the author might include if they think it improves the paper", whereas for big problems I do something like "The following are comments that flag problems with the paper I think the author needs to seriously address, or explain to me why they aren't problems in their response to reviewers."

I've been doing this already for a while and now I think the main reason for doing this is to try to avoid authors to become referee-placaters. After all, they know the topic they write the paper on more than I do (in the few cases they don't, it's almost always a very clear reject decision), and they are in the best position to edit the paper to make it better. It's not my job as a reviewer to be a stealth co-author.

For this reason, I'm sometimes tempted to write "The first two full paragraphs on p. 22 are just not working. I'm not entirely sure what point the author is trying to make here," Maybe I don't write things like this, because of some internalized impostor-syndrome that is inherent to reviewing, but I think from now on, I might include something vague like this as it may be useful information for the author. 

I'd be curious to hear people's thoughts on this, especially since now I do more editing and monograph writing than writing articles for peer-reviewed journals. Would something vague like this be helpful, or off-putting? 

Comments

[Tim]

First, a bias admission. I belong to the camp that does not normally value reviewer comments. I almost always read them. And I find a large percentage--maybe 75%?--unhelpful to useless. I do change my papers periodically when I receive comments. But I wanted to flag my bias.

Second, in my own experience, vague admissions of "I don't get this" or "its unclear why..." or "this idea need to be improved..." etc. are not helpful. For me, the problem is straightforward. If I knew there was a defect in what I had written, I would have tried to fix it before I had submitted it. But being told that someone else thinks there is a defect does not tell me what the defect is. (And--in line with my bias--I'm inclined to think on the basis of statistical reasons that there might not be a defect at all.) But if I don't know what he defect is supposed to be, then I don't even know how to begin to fix it. In fact, if anything, I find such comments frustrating. For they tell me that a reviewer is displeased and dislikes a portion of my paper/argument, but I can't figure out how to fix them! It can make revising a paper for a journal feel like shooting in the dark.

But I would be curious if my opinion is in the minority. It has happened before!

[Evan]

I understand Tim’s frustration, but I’ve never published a paper before. However, I once got a ’C’ on a paper without any justification or comments from my professor. Without such justification, I was led to wonder several things: 1) What exactly is problematic about my paper?; 2) does my professor hold any bias against me?; and 3) am I being intellectually gaslighted? The positive side is that I have received an ’A’ on a paper without justification too. But either way, I’ll never truly know *why* my paper was mediocre or good. And if I don’t know, then I can’t improve.

Without justification, I’ve been led to face two dilemmas: Feeling intellectually gaslighted and imposter syndrome/attitude. Without justification or feedback, your students may feel these two things. How are students supposed to know if they’re on the right track or if their efforts are truly worthwhile if they get grades on their papers without justification from their teachers? They probably won’t.

When I got an ’A’ without justification, I ended up asking myself: Did I truly deserve and ’A’?

My situation was on the extreme end. Getting an undesired grade without any justification is a red flag to me. I think a rejection without any comment would be unjust too. It’s a tricky thing when offering feedback. Sometimes, the reviewer may not be a fan of the topic and may just be intellectually gaslighting the author thereby accusing the paper of being ”unclear” and trying to change the trajectory of it. Sometimes, they are genuine and truly don’t understand what’s being written. It’s hard to know since we can’t read minds.

Most people aren’t very good at looking at their papers from a disinterested or objective perspective. At least, until they haven’t read it in a long time.

As well, if the paper was initially written with the intent to just pump-out papers, the author may not have much interest in improving it since he or she may be just publishing it to meet the demands of their department or university.

[anon]

For me, the vague suggestion to look at the two paragraphs would cause me to re-read the two paragraphs, and to briefly reflect whether I think they need more signposting with respect to their purpose. But depending on how that process goes, I might make no changes in response to such a comment.

I take your overall point here, and I do think that in many cases, even when I didn't make quite the fix the referee thought I would make, the referee's comments were still an important occasion for me to reflect and change particular parts of the manuscript.

However, in my experience, "It's something you need to fix" is too strong a statement. I've now published multiple papers that went through multiple R&Rs where me and a referee ultimately ended up in a deadlock - they say I haven't explained why P follows from Q, I say I have, and then I manage to convince the editor that no further changes are necessary. I think I started off too deferential to referees, and that it's important for authors to know that sometimes it makes sense to stand your ground.

[Assistant Professor]

I find Helen's suggestion here really useful both as a writer and a reviewer. As a writer: I appreciate someone being direct about what is not clear, what is unconvincing, or what is downright confusing. They don't have to tell me how to fix it, or even to fix it. I agree that the fact a reader had this reaction means I could improve that sentence/paragraph/section in question. I would also value this kind of feedback, which I think can be done in charitable, not unkind, ways. The reviewer feedback I find least helpful is when a reviewer decides to take issue with one of my claims by denying/negating it without giving arguments for why. In theory, I am an expert on my topic, which is why I am writing about it, and they are an expert on the topic, which is why they are reviewing it, and perhaps we have a reasonable disagreement - but that we disagree is not necessarily cause to reject a paper. If a reviewer instead said: "you didn't convince me here" I would know that I have work to do, not just think that this person is doubling down on their own contrary position from mine.

As a reviewer I really like this idea because I think it helps remedy imposter syndrome about reviewing: if you are being asked to review a paper someone has confidence in your philosophical abilities, and if those abilities lead you to find a part of a paper confusing or unclear as a reader, then that is important feedback to an author (and the journal). While I can see that some people like philosophical conversations to be so specialized and erudite as to exclude any non-specialists from participating in the conversation, I don't accept this as a reasonable goal for philosophy (if it is to have any meaningful use). If I can't understand a part of your paper, then it probably is not clear, not just that I don't "get it."

[P K]

I second Helen's approach. Along the same lines, when I feel like a referee has missed the point or misinterpreted what I'm saying, that's another similar clue that I can take responsibility for conveying my ideas more clearly.

Sometimes, they've wanted me to say something that I have actually said, but they've somehow missed it. In those cases, I've made sure to emphasize that point: if they missed it despite looking for it, I likely wasn't highlighting it enough, and even if I was, it probably doesn't hurt to highlight it even more.