Dealing with undercitation in philosophy? (Guest-post by Ten-Herng Lai)

Ten-Herng Lai (Australian National University) writes in:

I understand that pretty much the same issue was discussed here around 2015, but I think it is important that we bring it up again. I believe that literature review and under-citation is a problem in academic philosophy. Here I can only provide some personal experience as a journal reviewer. It may be that I’m not being a very good reviewer. I’m open to this possibility, and if so, please simply point it out.

I am a very junior scholar, but was fortunate enough to publish a paper in a decent journal on an issue that proved to be quite topical. So I get these invitations to review, several of them. Last time I counted, there are at least 20+ papers published in print or online first on this topic since 2019. A good portion of papers I reviewed cites almost none of them, as if the issue was uncharted territory. Some papers have very good insight, but fail to engage with the literature at all. Some papers just reinvent the wheel. Occasionally there are some papers that make good proposals but would be much better if taken certain points others have made into consideration.

Here’s what I did as a reviewer. I would encourage them to at least say “Archer (2020) Bard (2018) Cleric (2019) Druid (Online First) have all suggested certain solutions, but mine is different and covers an ignored but important point.” Better is that they say why their point is better, or at least distinguish their own point from those made by others. Sometimes I suggest that they can simply cite and rephrase a “respond to objection” someone else already made, instead of writing a separate response to some objection that has already been responded to like 5 or 6 times in different papers. Some more particular points may be that the author(s)’s solution can be combined with someone else’s solution to become even better.

I hope what I did as a reviewer helped to improve the quality of those papers, even if they were not accepted in those particular journals. I sincerely believe that doing some literature review and actually engaging with some previous papers enhances the quality of papers. (And I admit sometimes I see some papers published that fail to do so, and are of very bad quality, e.g., proposing points that have been utterly knocked down by previous publications. I don’t know how they went past the reviewers in those journals...) I also believe that there seems to be some sort of disrespect going on when not engaging in previous publications enough. Probably it’s some sort of arrogance, believing that one has “discovered” something new, just like (though obviously less serious than) how Columbus “discovered” America and how Captain Cook “discovered” Australia.

I agree that this is a problem, and have heard other people say similar things. I also agree think that what Lai did here as a reviewer is right. But this raises an obvious question: what else can/should be done to address under-citation in philosophy? I have two ideas, and am curious what other readers think, including whether readers have other suggestions, as well.

Aside from reviewers like Lai doing good work to uphold scholarly standards in peer-review (which is really just a piecemeal solution), I see two potential ways to address under-citation:

1. The discipline could move (as other disciplines have) to a different peer-review model where it is the norm for authors to upload working drafts to a central repository (in our case, to PhilPapers/PhilArchive), where other authors can provide feedback prior to journal submission. One of the brilliant things about such an alternative model is that it doesn't leave peer-review to a couple of reviewers. Anyone can read a draft paper on an online repository, and not only leave substantive philosophical feedback, but also correct for scholarly errors and omissions.

2. We can work to change how philosophers understand the purpose of citations: On at least a few occasions, I have heard philosophers say that they didn't cite something because it didn't influence their thinking, implying that this is a perfectly fine norm to follow. But, to see what is so odd about this, consider an analogous case in science. Suppose, several years after Einstein published his 1905 papers on special relativity, Brownian motion, etc. (say, in 1910), another scientist submitted papers claiming to be the first to argue that space and time are relative, and that tiny particles (atoms) must be responsible for Brownian motion, not citing Einstein. This seems rather outrageous, no? Part of why citation matters is for the historico-scholarly record to be accurate, conferring proper credit for the development of ideas where it is due.

But these are just my thoughts. What are yours? Do you share Lai's (and Krishnamurhty and Wilson's) concerns? Do you think Lai did the right thing as a reviewer? And what, if anything, do you think could/should be done to improve scholarship norms and citation practices?

Comments

[anonymous tenure track at R1]

Journals need to stop including bibliographies/reference lists in their word count limits if norms are going to change here.

[grymes]

Yep, the first commenter nailed it. My polished presubmission draft always has more comprehensive citations than the draft I submit, because I always have to cut thousands of words and non-argumentatively-crucial citations are the easiest way to cut chunks of words from a polished draft.

[Postdoc tired of bad ivy league advices]

Here is a list of things I have heard in the past couple years in my department:

- "It's fine if you don't cite people with similar views. When journals look for referees, they look for people that you do not cite. If you do not cite competent people that have worked on the same issue, you have a better shot at getting competent referees";
- "Stop reading everything! Start *thinking* instead";
- "If you cite too much, you will give the impression that you are a grad student. Don't give that impression";
- "Don't start highlighting subtle differences between your view and the other ones in print. If you do that, referees will just assume that your view is not original. The referees are not that competent and interested in your project";
- "If you find a paper that defends your thesis with similar arguments, just add a short footnote to the paper saying that X agrees with you. Dn't do more than that. It is *your* paper".

Pretty much all these advices support underciting (the last advice supports "under-engaging", or whatever).

I don't know the extent to which these (bad) advices are common. But if they are common, I can understand that people desperately trying to publish will end up underciting.

[Metaphysics grad]

I agree with anonymous. I often end up cutting citations that seem inessential (e.g. simply citing work done in relevant areas that I don’t meaningfully interact with in the paper) when I am worried about word limits. If the references did not count towards word count I would be more liberal with my citation practices.

[Another ANU Grad]

I'm not sure the problem here is really with not citing enough so much as not reading (seriously) enough.

A lot of people seem to be working with the idea that citation is mostly about giving credit where it is due.

But as Ten points out, citation is in fact much more important than that: it is important for actually doing good philosophy (not reinventing the wheel endlessly, not missing good objections, not missing important connections between ideas).

I suspect that philosophy would look like it made a lot more progress if the default assumption was that you would actually read everything on a particular topic and use it in your work. Rather than, as it seems to be, that you should just have an idea and try as hard as possible to publish it without worrying about how it connects to what has already been said.

All of that said I think getting out of the problem is a collective action problem: no one wants to do the work of really getting to grips with the literature when that doesn't help them get published, and there is such a high pressure on publishing. We need to reduce the pressure on publishing quantity and raise the pressure on publishing quality (where quality includes engaging with the literature) to give people the incentive to do the hard work.

Finally, I have one concrete suggestion for helping with the problem. Especially as mentioned above that word limits can be a big problem: appendices.
Many journals allow you to attach appendices (at least online) that can be of arbitrary length. Your paper might still have to get through review without many citations in it, but if you care about the problem and want to help you can attach a literature review and comments about how your work fits into it as an appendix. And then you can cite basically everyone who is in fact relevant to your work.

Doing things this way has another benefit: part of the collective action problem is that it can be very hard to find all the relevant literature. If a few people start attaching comprehensive bibliographies as appendixes, it makes it much easier for other people to find the relevant literature. And that at least takes away part of the excuse that finding all the relevant literature is just too much work.

[Ten-Herng Lai]

Thanks for the responses. They are very reassuring I totally agree that the "including references" word limit is a huge issue. I also once or twice deleted references just to lower the word count. But relatedly, I do hope that journals with less stringent word limits can be recognised a bit more. Some are great, especially in handling submissions (e.g., Ergo).

And

re Postdoc tired of bad ivy league advices

I totally get the

- "If you cite too much, you will give the impression that you are a grad student. Don't give that impression";

But my advice wasn't from a mentor or more senior academic at my home institution. It was from a journal reviewer at round 2. Round 1 basically asked me to cite a lot more papers. Round two included the comment "it reads, honestly, like a grad student paper for a class where the author needs to show they did all the reading." The reviewer was nice enough just to recommend a revision though.

[Junior Scholar]

When those at the top are among the worst it is hard to get norms to change. Here is a recent experience of mine. A star professor 30 years into his career publishes in a paper in one of the top generalist journals on the topic I specialize in, a topic which he has never previously published on which is periphery to his main area of specialization. The paper is craftily written yet fundamentally flawed in several ways that those familiar with its special topic could easily point out. Its the kind of paper that might come across as deeply insightful to a philosopher who doesn't know the topic well but not to an expert (the frequency with which well-connected people in the profession get these kinds of papers published should lead us to ask what is going on in the reviewing process that allows this).

Anyway, among its flaws is scarce citation of the relevant literature. And on closer examination a stark divide is obvious. The paper cites all the relevant literature that is 30+ years old (including some that, although important 30 years ago is now quite outdated). And it cites NONE of the literature (of which there are least a dozen highly relevant articles and books) published in the last 30 years. It is as if the author had gone back to a term paper he wrote 30 years ago when he did his PhD and taken all his citations from there while not bothering to check if the literature has progressed in the 30 years since. Furthermore, knowledge of this more recent literature is exactly what would lead one to recognize the flaws in this paper.

Anyway, the good news is that there is now a sharp reply to this paper in print that points out the exact flaws that would be obvious to anyone who knows the recent literature well. However, the more general point about how the well-connected in our profession often get weak papers like this into elite journals with extremely lazy citation practices remains.

[Michel]

For what it's worth:

I typically adhere strictly to the word limit in my initial submission. But occasionally that's not possible due to the sheer volume that needs to be cited--this is typically true when I'm working with stuff from other disciplines. In those cases, I make sure my article text is well below the word limit, but I don't worry too much about my citations putting me over. When I submit the paper, I note in my letter to the editors that although the paper is, e.g., 1k words over the limit, 2k ofwords come from the list of works cited.

I figure that way they can decide whether it's a problem. It never has been.

[Jakub]

I'm wondering whether undercitation is a real problem in philosophy. In a way, it is because in philosophy people cite less than in natural sciences. I suspect this is where this the tendency to cite a lot comes from. However, who wants 4-page papers with 50+ authors and 100+ references in philosophy? Did Kant or Heiddeger or Russell cite a lot? I think the real problem is that scholars and their works are evaluated according to the number of citations.

[cite]

I want to express a little dissent here. I study publication norms in science - that is one of my areas. I think people are misunderstanding what is going on. You cite what influences you - that is the proper norm. Clearly, it is a questionable research practice to just cite everything on a topic, if you have not actually read it. This happens all the time in one of the fields I work in. But I do agree that philosophers often do not do enough reading on a topic. I think as you age as a scholar, you master more literature, assuming you keep up with the developments in the field. And, some sub-fields, like history of philosophy, really require you to read a lot.
About over-citing, I have read colleagues' papers, usually younger ones, who excessively cite. The trouble is they lose cite of their own contribution. They no longer know what their own paper adds.
Incidentally, I have published a few papers tracing ideas and arguments through time. These papers have big bibliographies, and they tend to get cited. They are not quite review articles, because I usually have a point I want to make in revisiting the history. I would encourage people to try writing one of these. But you really have to do the research.

[anonymous tenure track at R1]

Michel, I've gotten desk rejections at two different journals for doing that exact thing. Since I can't risk desk rejections at this stage at the journals I need to publish in, I can't do that. But I'm glad it's working for other people (presumably, in other journals).

[Vaughn]

cite - I don't know if this was a typo or intentional: " The trouble is they lose cite of their own contribution." But either way it's brilliant and I love it.

[Assistant Professor]

1. I wish I had more time to read. Yes, sure, I could just "make" time for it, and arguably it would make my work better. But it is hard to figure out where that time would come from, frankly, and I am not evaluated as to whether my papers are intrinsically better, I am evaluated based on how many I publish and where so if I meet the expectations set for me in volume and prestige of publications despite not reading as widely as I could, there is little tangible incentive to reading more widely. I see people who know everything there is to know about X topic but don't churn out enough work to be professionally viable and the professional obligations side of academia weigh against the other kinds of pure intellectual pursuits of it.

2. Despite other articles Marcus references suggesting it is really easy to just go to philpapers and find everything written about a subject, I don't find it that simple at all. Unlike some STEM disciplines, keywords don't work the same way in philosophy to clearly indicate the relevance of a paper to one's topic, at least I don't find this to be the case. Even if one consistently reads everything coming out in the specialist journals in their area the fact of generalist journals that might be publishing articles related to your area (but also might not) makes it hard to stay on top of the literature unless you somehow have time to read every specialist journal in your field and all the generalist journals too (see #1: I don't have this kind of time).

[Ten-Herng Lai]

I totally agree that the keywords system could be better, as I find it extremely frustrating. Just some thoughts of how I keep track of some particular issues. (It is indeed somewhat time-consuming, and I would also like to hear how others do it.)

In addition to google scholars (which I think became pretty bad since last update), academia, research gate, philpapers, I visit this website called the philosophy paperboy every day, even before I start procrastinating.

https://thephilosophypaperboy.com/

The website provides a pretty good update on what papers are published. I try to glance through the titles of the papers, sometimes skipping journals that I know won't publish papers that are relevant to me. I also search relevant terms occasionally just in case I've missed something.

I also update my information based on what is cited in other papers. And that's another reason I think citing properly is important. It's our collective effort to keep everyone updated. (And this is one of the joys of being a reviewer, to learn from what you review.)

[Tim]

A common view is the 'cite what influences you' norm. Several times in this thread people represent the disagreement over the functional/purpose/role of citation. I'd suggest the dispute originates in a deeper dispute: the role of publication.

If you think publication is fundamentally about your ideas and views, then it makes sense that you own cite what influences you. For the publication is a public record of your private thoughts and their history. Alternatively, if you think publication is fundamentally about a group of people thinking through some issue, then it makes sense to cite more widely. For a publication is a contribution to a ongoing discussion.

Regarding practical steps, I greatly agree with the suggestions that journals should not include works cited among word counts. As a reviewer, I think people can also push for more engagement. Many journals ask reviewers to evaluate the degree to which the work cites relevant literature. I think it would be fine to say that a paper doesn't have a glaring omission but because it only cites 20 papers (an illustration) it doesn't engage the literature enough.

[Evan]

In addition to reflecting on the role of publication, it would be helpful to discuss the cost and benefits of the role. If you work in political and legal philosophy, an efficient way to get the most out of research is reading works written by philosophers who have JDs and have written in law journals. These philosophers don’t mess around when it comes to engaging with the literature both philosophical and non-philosophical and so their epistemology is often more reliable and fruitful. But also because law articles tend to be long anyways since a lot of them deal with actual public policy and not just strict theory. These days, I tend to enjoy works written by philosophers who work in other fields besides philosophy. They have philosophical competence while engaging with other real-world problems. I just don’t have time or patience for fruitless articles.

[Assistant Professor]

@Ten-Herng Lai I had no idea about the philosophypaperboy website. THANK YOU! What a useful resource.

[Kenny Pearce]

I really like the appendix idea mentioned above. It seems to me there are three things that lead to lack of engagement with literature. (And I think that’s the real problem, not undercitation. Adding laundry lists of tangentially relevant papers without engaging them isn’t what we need.) The three issues are: laziness/wrong incentives; word limits; and worries about losing the thread of argument and talking about what everyone else has said instead of saying anything new. (It’s the last of these, I think, that leads to people saying you are writing like a student.) Adding appendices that don’t count against word limits in which there is more detailed engagement with existing literature would help with the second and third problems. It would probably only be read by the people discussed in it, but that’s fine. I’ve sometimes seen this done in books (“Appendix: Reply to My Critics”) but never papers.

None of this addresses the first problem: we’re all under pressure to outpublish one another, and it’s currently possible to get published in good journals with little engagement with existing literature.