What is peer review really? (part 7 – an amusing aside)

by Carl V Phillips

A couple of weeks ago, a funny story came out about a peer-reviewed journal accepting a “paper” which consists of nothing but the message (in the title, repeated in the text for pages, and in two figures), “Get me off your fucking mailing list”, laid out in the format of a research paper. The authors had created this to reply to the plethora of spam that scholars get, inviting them to submit papers, attend conferences, etc., and had been using it for a few years. So imagine their surprise when, after sending it in response to journal spam, they got the reply that it had been accepted for publication after it was favorably “peer-reviewed”. They merely needed to pay the $150 publication fee. (The rest of us were shocked that they apparently decided to not pay the fee and get the paper published. I mean, come on, how could you resist?)

This might baffle you if you are unaware of the phenomenon know as “predatory” journal, which will basically publish any submission for a fee. Hundreds, maybe even thousands, of them have cropped up, usually operating out of India, Nigeria, or China, but often using fake Western addresses. There is a whack-a-mole game afoot to try to catalog them all.

The thing is, the word “predatory” that has been attached to them is a misnomer. “Junk”, yes. But who, exactly, are they preying on? The authors get what they want without having to shop their paper around until a journal accepts it (which will happen). The journal makes a small sum of quick money for doing little more than showing up. I think the better terms is “prostitution” rather than “predation”. Victimless crime and all (setting aside the authors who are tricked into not realizing it is prostitution, but there cannot be all that many of those).

The supposed victims presumably are people who believe that everything that is published in a peer-reviewed journal must be right. But: (a) Whose fault is that??! It is a really stupid thing to believe, even apart from this. (b) It is not like the “predatory” journals are the only ones that take advantage of this naivety. They have just taken the “public health” model to its logical extreme. While it may seem like the predatory journals model is a qualitative deviation, the fact is that it is really just a slightly more extreme version of journal publishing in many fields.

When you introduce rewards for particular behaviors, people respond. Sometimes they respond exactly as you want (e.g., make a carpool lane that moves faster and more people carpool) and sometimes they game the system (e.g., tax capital gains lower than wage income and rich people figure out how to disguise their wage income as capital gains). The latter becomes much more of a problem when you are rewarding only a very rough proxy for what you care about. So, for example, when teachers, schools, and pupils face high-stakes rewards and punishments based on standardized test performance, the schools spend an enormous amount of time teaching how to take the test, taking time away from imparting the knowledge and skills that the test is supposed to measure.

Academic publishing in weak fields has a lot in common with the high-stakes testing fiasco. What society (and universities) wants to reward is the production of knowledge that benefits the field or the world. But, as with good teaching and real learning, it requires quite a lot of skill and effort to judge whether that has occurred; dealing with some simplistic numbers is easier. In a field where there are woefully few people who can make that judgment compared to the number writing papers (recall the post in this series on how that is clearly true for “public health”), the simplistic approach is especially tempting.

So predatory journals have a niche. But so do journals that differ from them only by virtue of randomly rejecting some papers by deferring to the opinion of whoever they happened to manage to get to write a “peer review” for them. This merely creates busy-work for authors, who just send the paper out to another such journal after it is rejected.

Of course, a cleverer way to game the system, if you have a group of people who want to publish junk from a particular organization or perspective, is to just start a journal (e.g., Tobacco Control) or use part of generally legitimate journal (e.g., Pediatrics, partially a research journal about pediatric medicine, but also publishes whatever junk policy and public health research that pleases the cabal that runs it). These differ a bit from full-on “predatory” journals, of course — they only publish junk that supports a particular political agenda, not just any old junk that is submitted.

The journals mentioned in the previous two paragraphs are unlikely to accidentally accept a paper that just consists of seven words repeated over and over, and they tend to fix the worst grammar and formatting errors, but that is hardly a measure of true quality. The basic point is the same: An author wants to get the imprimatur of “peer-reviewed publication” for a paper, and the journal pretty much promises they will oblige so long as conditions are met (i.e., “pay a fee” or “keep submitting until it is accepted” or “toe the party line”, but not “do a good analysis and write conclusions that follow from the data”).

A tempting response to all this is to trust the journal peer review from “good” journals, while recognizing that there will inevitably be journals that will publish everything or at least that will publish anything (i.e., they accept or reject papers fairly arbitrarily, so anything can get published, or they will publish anything of the right political bent). But there are a few problems with that theory. First, as noted, some journals like Pediatrics, which may be legitimately respectable in their technical area, also publish a lot of political junk that takes advantage of their technical reputation. BMJ and Lancet clearly fall into this category also.

Second, in some fields there is a hierarchy of journal quality, and those at the top do indeed make more of an effort to find good reviewers and seek genuine quality. But in the health sciences, papers make it into “good” journals not so much because they are higher-quality work, but because they have sweeping implications or their results are novel. The latter actually means that the results are more likely to be wrong, because if no one else has ever observed a phenomenon, chances are it does not exist. (I often point out that any sentence that starts with “ours is the first paper to show…” should end with “…and therefore it is probably wrong.”)

Third, even to the extent that “good” journals are trying to publish good papers, they simply lack the skills in social science to analyze public health epidemiology papers and the skills in economics etc. to do policy analysis. Thus, papers in those areas tend to be no better than those in any third-rate public health journal — just more sweeping in their conclusions. Indeed, whenever I wanted a recently-published teaching example of bad population epidemiology, I always looked through recent issues of the New England Journal of Medicine. I was never disappointed.

Predatory journals may well be diluting the quality of “peer-reviewed journal publications” in some fields, and thus there is value in pushing back against them. But in “public health” and associated fields, it is not clear that they would actually lower the average quality much because the journals in the field pretty much do the same thing already. There is a good case to be made that pages of repeated “Get me off your fucking mailing list” would be an above-average quality article in tobacco research or “public health” more generally, because the average paper in the field has negative value.

9 responses to “What is peer review really? (part 7 – an amusing aside)

  1. hahaha good one! Actually, I had often wondered at the sheer junk that gets published in so-called “peer reviewed” journals.

  2. Hmmm. Might be interesting to start a journal to review peer reviews. “Journal of Peer-Peer Reviews” or “Journal of Peer-Reviews Reviews” — a spreadsheet-type thing with one field a link to more info on that journal or reviewer. Leave the authors out of it, it’s not their fault the reviewers failed. JPPR or JPRR? IJPRR (International Journal of Peer Review Reviews)

    Each entry would be either a journal or a reviewer (which means some duplication) — a count of how many of their reviews have been reviewed, grading of those reviews, a symbol for whether or not the reviewer is identified. Start with a random sample of 1-2 issues. Maybe Lancet or BMJ would have 10 reviews, 8 positive (Look, they did a good job on this one), 2 REALLY ripping something bad apart, and an estimate of the ratio of good/bad reviews. Once you have the ratio established, from then on only review the bad ones (and say so.) Re-review the ratio every 2 years for each pub or reviewer. Make these class assignments for students in research fields — get some free labor and teach the kids what good reviews are.

    Or just recruit disaffected Public Health personnel who have been fired for doing something right that was ideologically wrong, per Chris Snowdon’s recent speech in England.

    • Carl V Phillips

      Well, I can’t see making a full-on publication of it. And it is limited by the unconscionable fact that almost all reviews are anonymous and hidden. However, doing a bit of what can be done is kind of what I am working on.

  3. “This might baffle you if you are unaware of the phenomenon know as “predatory” journal, which will basically publish any submission for a fee. ”

    Holy crap! That sounds just like the music business:-)

    • I am going to say “somewhat”. While I certainly don’t know the details of that business so might be missing something, it strikes me that doing that in the music business (or the book publishing business — I am guessing it is basically the same, and I am a bit more knowledgeable about the latter) is more a legitimate business transaction and less a complete gaming of the system. If you are a performer or author and you really want your work to be professionally “packaged”, you can buy that service. Perhaps you are motivated by just wanting to get a few nice copies into the hands of a few people and are willing to pay to make that happen. Or perhaps you believe that you will make money selling it once you get it packaged and out there, even though no major company is signing on to be your partner in that. Either way, you are getting what you are getting.

      Now I realize there is some of the same element in it: If a music or book producer has agreed to partner with an artist/author, that is a stamp of approval that the consumer can see as a vote of quality that does not exist for self-funded publishing. A self-funded publication might create the illusion that this exists. On the other hand, in those areas that imprimatur obviously does not tell the consumer “you are going to like this”, since tastes vary so much, but merely “it has been predicted that enough people are going to like this that it will make money.” So presumably you would not make the mistake of saying “it was ‘published’ so it must be worth my time”, analogous to the “it was in a ‘journal’ so it must be right”.

  4. “So presumably you would not make the mistake of saying “it was ‘published’ so it must be worth my time”, analogous to the “it was in a ‘journal’ so it must be right”.”

    Well, I don’t think that way, but some people do. I was referring to the pay for play scenario that happens so often in the music industry where $$ often trumps talent (hence there is often a facade attached to some acts being “popular” via one gazillion “likes” on Facebook for some band that no one has ever really heard of, for example, or rigging charts, etc…), just as $$ and politics often trump the truth in the tobacco science arena. Of course this is not always the case, but it does happen frequently.

  5. Speaking of journalistic integrity and honest science:

    “The first-ever report on the global use and public health impact of smokeless tobacco finds that more than 300 million people in at least 70 countries use these harmful products.”

    “The serious health effects of smokeless tobacco have been documented.”

    ..and this is quite amusing:

    “Additionally, smokeless tobacco products contain nicotine..”

    What’s even more amusing is the picture of analog cigarettes with the skull and cross-bone pirate flag, which have absolutely nothing to do with smokeless tobacco of course.

    Maybe this is meant to be satire?


  6. Pingback: Science Lesson: on “anecdotes” | Anti-THR Lies and related topics

  7. Pingback: What is peer review really? (part 1) | Anti-THR Lies and related topics

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s