Evidence that 'weak' publications help candidates

Earlier in my career, both in graduate school and thereafter, I was told by a number of people that I should only try to publish in top-ranked journals, as publications in 'bad journals' are likely to be held against job-candidates. At least anecdotally, I have heard that this type of advice is common, perhaps even the received wisdom among job-candidates and faculty at grad programs. But is it accurate?

A number of years ago (all the way back in 2012), I did some informal data collection on that year's job-market, and what I found seemed to tell a very different story. I found that publications in non-top-20 journals not only didn't appear to hurt candidates for R1 jobs (as roughly half of R1 hires had at least one non-top-2o publication). I also found that non-top-20 publications appeared to help candidates for jobs at 'teaching' institutions (as roughly 2/3 of all hires for these jobs had no top-20 publications and 1/3 had publications in non-top-2o journals).

When I found this, I was still on the job-market myself, with only two minor publications (both 'replies') in a top-20 specialist journal and two publications in lower-ranked journals. So I decided to take a risk: I decided to stop focusing so much on top-ranked journals and instead started sending papers to lower-ranked journals. Over the next two years (2013 and 2014), I published nine articles, all in lower-ranked journals. And what did I find? My interview numbers dramatically increased (mostly at 'teaching' schools, but also the occasional R1 interview).

Now, of course, all of this is anecdotal. I am reporting it again now, however, because a scientific study, 'Do Publications in Low-Impact Journals Help or Hurt a CV?", was just carried out testing the hypothesis, at least in a sample of academics in psychology. And what did they find? Here are some key passages from the study's abstract:

Two experiments revealed that an applicant was rated as stronger when several weak publications were added to several strong ones, and was rated as weaker when the weak publications were removed. A third experiment showed that the additional weak publications were not merely viewed as a signal of additional strong publications in the future; instead, the weak publications themselves appear to be valued. In a fourth and final experiment, we found that adding a greater number of weak publications also strengthened the applicant, but not more so than adding just a few. The study further suggests that the weak publications may signal ability, as applicants with added weak publications were rated as both more hard-working and more likely to generate innovative research ideas. Advice for tenure-track psychology applicants: Don’t hesitate to publish in even the weakest journals, as long as it does not keep you from publishing in strong journals. Implications of the market rewarding publications in low-impact journals are discussed. (my emphasis)

Oh, and the idea that better-ranked journals publish better work? This new study found that studies in top-ranked science journals reproduce less often than studies in lower-ranked journals. To wit:

In which journal a scientist publishes is considered one of the most crucial factors determining their career. The underlying common assumption is that only the best scientists manage to publish in a highly selective tier of the most prestigious journals. However, data from several lines of evidence suggest that the methodological quality of scientific experiments does not increase with increasing rank of the journal. On the contrary, an accumulating body of evidence suggests the inverse: methodological quality and, consequently, reliability of published research works in several fields may be decreasing with increasing journal rank. (my emphasis)

So much for conventional wisdom!

Comments

[Amanda]

I agree with most of this - and would add that this is just another example of showing that there is far more than one way to be a "top" candidate, whether top is research or teaching schools.

In my experience research schools (but not teaching schools) want to see that you have some type of "prestige" ability, typically by having at least some publications in top journals, although a highly ranked PhD program can overcome this need, or even an especially creative /interesting/different profile. Or just a good fit for the particular search. Anyway, once persons pass that bar of having some elite publications, even at research schools top publications don't seem to matter nearly as much as creativity, having a place in your field, respect from scholars, and having work that people genuinely view as quality. As most people know there are all sorts of reasons why more risky papers are less likely to get published in top journals.

I would also add that the area you work in matters a lot. There is this list of top 20 publications that gets mentioned everywhere, but many of these journals are very narrow in the topics they cover. Often each issue contains something like 90% of their publications are in the areas typically considered "core" (epistemology, metaphysics, highly theoretical ethics - and typically papers that tend toward formal styles.) If you don't work in those areas it would be silly to expect most of the publications to be in those journals. Within speciality areas there are speciality journals that might be very well respected by those in the field, even though those out of them might be unfamiliar.

I occasionally write papers in core areas just so I can publish in top journals. But these aren't the publications I am most proud of, nor the ones I think that garner me the most respect.

Teaching schools is an odd thing, because it does seem that on average top publications hurt you, although I know this is not true of all teaching schools. Some teaching schools, even not elite ones, really like to see top publications. Candidates, it seems, should always look at the big picture alongside what type of journals seem the best fit for their own research, lifestyle, and skills.

[anon marketer]

I would be curious to know whether what is true of R1 jobs in general (viz. lower-ranked pubs neither hurt nor help) is true of PhD-granting institutions, PGR-ranked departments, or PGR top-20 departments. Not that it's really rational for anyone to marginally improve their chances at a PGR top-20 department while hurting their chances at teaching jobs, but it seems plausible that the conventional wisdom is true for a narrower class of jobs.

[Amanda]

Although I have not done anything like a systematic study, I have spent a lot of time looking at recent hires. From what I could tell you can’t really separate research schools according to Leiter Rank. There are some research schools that seem to care a lot about or only about elite publications, or elite publications along with elite PhD., But these schools seem to be all over the map as far as Leiter rank or lack of it goes. Seems more about the culture of the department or the beliefs of the search committee members. and I am including elite slacs As research schools.

[Marcus Arvan]

When I collected the data in 2016, two things stood out to me anecdotally about who got hired at Leiter-ranked places: they tended to either (1) have no publications at all but come out of a top-10 PhD program, or (2) have a lot of publications in *both* highly ranked and unranked journals. So, at least anecdotally, what this study reports may hold in philosophy too. Even though people *think* publications in “bad” journals hurts candidates, I saw no indication of that in the data I collected—which suggested to me that (just as this study shows) they may not hurt and may even help.

[anon marketer]

That much sounds about as I would expect. I suppose the case I'm curious about, because it's the one that I think conventional wisdom specifically concerns, is between the top-10 PhD with no pubs and the same candidate with *only* low-prestige pubs. Being in such a department myself, the thing you hear is that as far as the "best" jobs go (i.e. in top-20 PhD programs), you're better off having nothing than having only mediocre ones, though if you already have something in Phil Review a couple more in less-prestigious places won't hurt. Again, withholding from submitting to such journals so as to maximize one's chance at a top-20 job with no pubs at all (at the expense of being a better candidate for teaching jobs) seems like a dumb strategy, but it's what I think the conventional wisdom recommends. Small sample set, non-scientific methods, and other qualifications, but your data actually seem to support that version of the claim, if the people getting those jobs either have nothing or have a mix but never have *only* less-prestigious publications.

[Philosopher]

Fwiw, the advice that I have received and also given, is not to avoid publishing in a bad _journal_, but rather to avoid publishing a bad _paper_, especially not one from too early in your career -- something that might be a fine sophomore effort but that won't be nearly as good as what you will likely be putting out as a later stage dissertizing grad student. If you publish a good paper in a bad place, it'll still get read by search committees (at least if you get on a short list), but the same is true of a bad paper, even if -- especially if -- it managed to get published somewhere decent.

[Marcus Arvan]

Philosopher: Good point - I was given that advice as well. And I think it is good advice, much better than the advice not to publish in 'bad journals' at all.

[Amanda]

anon marketer: In the case you describe, I think it is important to distinguish between non-top 20 publications and "weak" publications. If someone from a top 10 Leiter department publishes in well-respected specialty journals that are not top 20, I think this would help them on average with research jobs (there are a few candidates like this that get hired at research schools every year). On the other hand, if it is a journal that is considered mediocre or weak by most, then I do think having no publications would be better, even if the weak publications wouldn't hurt or might help if the candidate had one paper in Mind, etc.

[Pendaran]

I think weak publications help candidates, as long as they also have some strong publications. You want a nice balance of quality and quantity. So, if all your publications are weak, then another weak publication won't help much probably (at least not for research jobs). But if you have some strong ones, then adding a weak one may help. I think I'm just echoing what others have already said. I guess I'm in agreement.

I wouldn't worry about publishing a bad paper. There is little agreement about what counts as a good paper. I mean avoid publishing a sloppy piece of work I guess, but that's a somewhat different aim. Also, even the best papers have flaws in their arguments or in other aspects of them that can be nitpicked. If you worry too much about whether your paper will be deemed "good" then you'll never publish anything.