What is peer review really? (part 2)

by Carl V Phillips

In the previous post, I pointed out how the system of journal peer review that dominates in health sciences is a relatively new invention that is already obsolete and has been abandoned by more serious sciences.  In spite of that, there is an idolatry of that system results in a lot of harm, caused by those who mistakenly believe that “in a peer-reviewed journal” means “correct”.  While it genuinely baffles me that anyone needs to have the falsity of that equivalence explained, the reality on the ground shows it is apparently needed.  In this post I will start to make more concrete the observations about the limitations of the system in the form of a series of myths.

I am presenting the myths in no particular order (the numbering is for convenient referencing), though I happen to be starting with the most important one.  Also, unless otherwise specified, I am referring to health science and health science journals.

Myth 1: Peer reviewers have access to more information than any other reader of the paper.

This is the single most important thing to understand about the peer review process.  Even if a peer reviewer is expert (far from always true) and committed to doing a diligent analysis of the submitted paper (almost never), she has access to only the same manuscript that the reader ultimately sees.  Some specific implications of that appear in the submyths to this point follow.  But it is worth pointing out the most important implication of this for those who fetishize peer review:  Anyone who cites whether or not a paper was journal reviewed as if it matters much for its credibility is basically saying, “I am not an expert on this topic.”

Why?  Because that individual, with the paper in hand, has access to exactly the same information that the journal’s editor and other reviewers had or, in the case of papers that are circulating in a different form, would have if it were submitted to such a journal.  If he is genuinely expert in the topic, he will assess it himself — probably far better than the journal’s reviewers did since he is apparently trying to make some use of its content rather than just dashing off a comment on it that he has no investment in (see below, in a subsequent post).  If he is genuinely non-expert and does not plan to ever be so (say, a random reporter who has twenty minutes to learn about a topic) then there is a bit of value in the fact that two or three people who understood the paper better than he did claimed it was not total junk.  But if a supposed expert makes a big deal about something being in a peer-reviewed journal, he is effectively saying “the fact that a few other people, who may or may not be all that expert, supposedly(! — again, read what follows below) signed off on exactly what I am seeing is worth more than my own assessment of it.”  Even worse, if the supposed expert dismisses a paper because it is not in a journal, he is saying he is not qualified to review it himself — a pretty strong claim given the low quality of most reviewers!  (Moreover, as noted in the last post, he is endorsing a system that has already been abandoned by serious sciences.)

So, to put this concretely, when the FDA Center for Tobacco products reviewed the science on e-cigarettes as if only the tiny portion of it the ivory tower journals was worth considering, and treated that portion as if it were all accurate, they were effectively saying that they were so lacking in expertise that they could not evaluate anything for themselves and would defer to the highly-flawed journal process.  Not the behavior we should demand from our supposedly expert regulators.

Of course, none of this is to say that real peer review (as opposed to the fetish of peer review done by health journals) is useless.  Real peer review is what makes science work.  Real peer review includes putting a paper out for comments and updating and improving it based on them before finalizing it.  Real peer review is also what happens after a paper is finalized and put out into the world for everyone to assess.  The fact that many (far from all) of my useful writings appear in journals says almost nothing about their accuracy.  The fact that ANTZ have been attacking me for over a decade with defamation and ad hominems, but have never in all these years — literally not once — scientifically challenged the substance of anything I have written speaks volumes about its fundamental accuracy (which is not to say that everything is right, obviously, but it does provide the strongest possible evidence that it is basically right and that there are no uncorrected[*] major errors).

[*There was one fundamental error in that corpus.  It was caught not by the journal reviewers or hostile ANTZ, but by a non-ANTZ reader a while after publication.  I published an extended correction.  Try finding an analogous example to that in all of ANTZdom.  Just one.]

Part of the problem in public health science — and especially the tobacco field — is that this real peer review system is badly broken.  When gilded idolatry is the norm, the true gods get pushed aside.  People who go into academia in these fields are generally not …um, let’s just say… the best scientific thinkers in the world.  But the problem is made far worse by the journals basically not allowing critical analysis (in any sense of the word “critical”).  If someone does what the journal reviewers should have done — carefully analyzes a paper and identifies fundamental errors or omissions in it — and submits it as a letter to the editor, there is little chance it will be printed.  Someone who writes a whole paper debunking an existing article (it generally takes 10 times as much ink to debunk junk science as it does to write it in the first place) has approximately no chance of getting it published in the same journal, and will have quite a hard time publishing it somewhere else.

Of course, those observations border on concessions to the idolatry.  There is no reason something like that needs to be in a journal.  If public health were really a truth-seeking science, publishing it anywhere that has a high profile in the subfield (say, this blog) would be sufficient.  So long as those who wish to hide the truth can keep others tricked into believing that journal peer review is worth much, even those who seek the truth are unlikely to find it.

In the next post, I will expand upon Myth 1.