What is peer review really? (part 7 – an amusing aside)

by Carl V Phillips

A couple of weeks ago, a funny story came out about a peer-reviewed journal accepting a “paper” which consists of nothing but the message (in the title, repeated in the text for pages, and in two figures), “Get me off your fucking mailing list”, laid out in the format of a research paper. The authors had created this to reply to the plethora of spam that scholars get, inviting them to submit papers, attend conferences, etc., and had been using it for a few years. So imagine their surprise when, after sending it in response to journal spam, they got the reply that it had been accepted for publication after it was favorably “peer-reviewed”. They merely needed to pay the $150 publication fee. (The rest of us were shocked that they apparently decided to not pay the fee and get the paper published. I mean, come on, how could you resist?)

This might baffle you if you are unaware of the phenomenon know as “predatory” journal, which will basically publish any submission for a fee. Hundreds, maybe even thousands, of them have cropped up, usually operating out of India, Nigeria, or China, but often using fake Western addresses. There is a whack-a-mole game afoot to try to catalog them all.

The thing is, the word “predatory” that has been attached to them is a misnomer. “Junk”, yes. But who, exactly, are they preying on? The authors get what they want without having to shop their paper around until a journal accepts it (which will happen). The journal makes a small sum of quick money for doing little more than showing up. I think the better terms is “prostitution” rather than “predation”. Victimless crime and all (setting aside the authors who are tricked into not realizing it is prostitution, but there cannot be all that many of those).

The supposed victims presumably are people who believe that everything that is published in a peer-reviewed journal must be right. But: (a) Whose fault is that??! It is a really stupid thing to believe, even apart from this. (b) It is not like the “predatory” journals are the only ones that take advantage of this naivety. They have just taken the “public health” model to its logical extreme. While it may seem like the predatory journals model is a qualitative deviation, the fact is that it is really just a slightly more extreme version of journal publishing in many fields.

When you introduce rewards for particular behaviors, people respond. Sometimes they respond exactly as you want (e.g., make a carpool lane that moves faster and more people carpool) and sometimes they game the system (e.g., tax capital gains lower than wage income and rich people figure out how to disguise their wage income as capital gains). The latter becomes much more of a problem when you are rewarding only a very rough proxy for what you care about. So, for example, when teachers, schools, and pupils face high-stakes rewards and punishments based on standardized test performance, the schools spend an enormous amount of time teaching how to take the test, taking time away from imparting the knowledge and skills that the test is supposed to measure.

Academic publishing in weak fields has a lot in common with the high-stakes testing fiasco. What society (and universities) wants to reward is the production of knowledge that benefits the field or the world. But, as with good teaching and real learning, it requires quite a lot of skill and effort to judge whether that has occurred; dealing with some simplistic numbers is easier. In a field where there are woefully few people who can make that judgment compared to the number writing papers (recall the post in this series on how that is clearly true for “public health”), the simplistic approach is especially tempting.

So predatory journals have a niche. But so do journals that differ from them only by virtue of randomly rejecting some papers by deferring to the opinion of whoever they happened to manage to get to write a “peer review” for them. This merely creates busy-work for authors, who just send the paper out to another such journal after it is rejected.

Of course, a cleverer way to game the system, if you have a group of people who want to publish junk from a particular organization or perspective, is to just start a journal (e.g., Tobacco Control) or use part of generally legitimate journal (e.g., Pediatrics, partially a research journal about pediatric medicine, but also publishes whatever junk policy and public health research that pleases the cabal that runs it). These differ a bit from full-on “predatory” journals, of course — they only publish junk that supports a particular political agenda, not just any old junk that is submitted.

The journals mentioned in the previous two paragraphs are unlikely to accidentally accept a paper that just consists of seven words repeated over and over, and they tend to fix the worst grammar and formatting errors, but that is hardly a measure of true quality. The basic point is the same: An author wants to get the imprimatur of “peer-reviewed publication” for a paper, and the journal pretty much promises they will oblige so long as conditions are met (i.e., “pay a fee” or “keep submitting until it is accepted” or “toe the party line”, but not “do a good analysis and write conclusions that follow from the data”).

A tempting response to all this is to trust the journal peer review from “good” journals, while recognizing that there will inevitably be journals that will publish everything or at least that will publish anything (i.e., they accept or reject papers fairly arbitrarily, so anything can get published, or they will publish anything of the right political bent). But there are a few problems with that theory. First, as noted, some journals like Pediatrics, which may be legitimately respectable in their technical area, also publish a lot of political junk that takes advantage of their technical reputation. BMJ and Lancet clearly fall into this category also.

Second, in some fields there is a hierarchy of journal quality, and those at the top do indeed make more of an effort to find good reviewers and seek genuine quality. But in the health sciences, papers make it into “good” journals not so much because they are higher-quality work, but because they have sweeping implications or their results are novel. The latter actually means that the results are more likely to be wrong, because if no one else has ever observed a phenomenon, chances are it does not exist. (I often point out that any sentence that starts with “ours is the first paper to show…” should end with “…and therefore it is probably wrong.”)

Third, even to the extent that “good” journals are trying to publish good papers, they simply lack the skills in social science to analyze public health epidemiology papers and the skills in economics etc. to do policy analysis. Thus, papers in those areas tend to be no better than those in any third-rate public health journal — just more sweeping in their conclusions. Indeed, whenever I wanted a recently-published teaching example of bad population epidemiology, I always looked through recent issues of the New England Journal of Medicine. I was never disappointed.

Predatory journals may well be diluting the quality of “peer-reviewed journal publications” in some fields, and thus there is value in pushing back against them. But in “public health” and associated fields, it is not clear that they would actually lower the average quality much because the journals in the field pretty much do the same thing already. There is a good case to be made that pages of repeated “Get me off your fucking mailing list” would be an above-average quality article in tobacco research or “public health” more generally, because the average paper in the field has negative value.