by Carl V Phillips
I thought it would be worth taking this series non-linear to follow up on Part 4, which used the recent Popova-Ling “peer-reviewed journal article” as a case-study to illustrate much of what is wrong with journal peer review and the fetishizing thereof in “public health”. Popova and Glantz relied on that paper in the comment to the FDA that I discussed in yesterday’s post about Swedish Match’s MRTP application, which asks permission to remove the false and misleading “warning” labels from their products. This is a great illustration of why the fetishization, “it is in a peer-reviewed journal, so it must be right”, is such a dangerous travesty. (H/t to Brian Carter inspiring some of the observations that appear here.)
You will recall that Popova and Ling did a largely uninformative study, the main result of which is to show that if you expose people to false claims about harms from smoke-free tobacco products, their perception of the risk increases. (What a shock!) To obtain that uninteresting result, they used research methods that were fraught with serious human subjects ethics violations. Of course, they did not really care about the results of the research, as evidenced by the fact that the paper was primarily a statement of their political preferences, with conclusions that in no way followed from the research. There were other serious problems too. The journal’s peer reviewers — being, unfortunately, genuine peers of the authors — were apparently incapable of understanding these problems since they did not identify any of them.
Even though the authors lied and said they had no conflict of interest, they obviously did this study not for the purpose of providing scientific knowledge — there was no way their approach could have contributed to that — but to further their political goals. (Note that details about that observation and what appears in the previous paragraph can be found in the Part 4 post.) They wanted to influence regulators’ choices about “warning” labels on smoke-free tobacco products, and in particular FDA’s response to Swedish Match’s MRTP application. This was obvious from the paper itself, and made concrete in Popova’s comment that I addressed yesterday, which was basically all about the results of that “study”.
The substance of the comment, ignoring the lead-in that reads like the text of a television attack-ad during a political campaign (with about as much scientific sophistication), is based on Popova-Ling. Well, “substance” is too strong a word — they basically just made the three observations, followed by the non sequitur conclusion, that I quoted in yesterday’s post.
What they failed to do in the comment — just like they failed to do in the article — was note that the baseline consumer perception of risk from smokeless tobacco was barely lower than what they found for cigarettes, which means that it was way too high. After consumers got a modicum of accurate information, like they would if shown the proposed new warning, they adjusted their perceptions in the right direction, but they were still way too high. Recall that above I described the study as largely uninformative. This was the one informative bit.
But a funny thing happened on the way to turning the study into a “peer-reviewed journal article”: They dropped the results for cigarettes from the paper. We know those results only because when I researched the paper to do my review of it, I discovered that they had reported them in a previous version before taking them out in the article version. Think about that. The most relevant bit of information from the study for the MRTP application — showing that perceptions of the risk from smokeless tobacco are way too high and remain way too high even after consumers get some corrective information — did not appear in the paper.
So, the paper Popova and Glantz rely on did not include the most relevant result, as Popova obviously knows. Also it has been completely excoriated by the real reviews, including mine, which is published alongside the original paper, and a few others. The problems with the article — and the associated failure of the journal peer review process — were recognized as sufficiently serious that the journal is trying to figure out what to do about it. Currently the following note appears at the top of the paper at the journal:
Editor’s Note: Readers are alerted that concerns have been raised over the study protocol and the reliability of the data presented in this article. Appropriate editorial action will be taken once this matter is resolved.
Before you get too excited about this as evidence that the journal system really works, try to recall a single other occasion when you saw such a reconsideration for anything short of the suspicion that the authors out-and-out fabricated their data. It is not like Popova-Ling was exceptionally bad by the standards of “public health” articles, let alone anti-tobacco “research”. With the possible exception of the gross ethics violation, you can find hundreds of articles a month that are this bad. The difference here is that three independent experts — myself, Clive Bates, and Brian Carter — each put a lot of time into a serious concerted effort to point out the flaws in the research, the irrelevance of the study to the political editorial that the paper really was, and the unethical nature of the human subjects research. Also keep in mind that this is one of the few respectable journals in the public health space.
Such concerted scrutiny is just not going to happen with many papers. We are talking about a lot more than just writing a letter to the editor. Also notice that the note does not really refer to the worst problem in terms of the implications of the study for the world, that the conclusions do not follow from the study results. Perhaps this was just an oversight, but it does mean that readers of the note are not told of the biggest problem.
But does any of this matter for Glantz’s purposes? Probably not. Despite the red flag that appears on the journal page, he did not hesitate to rely on this article. Was this a tactical error? Probably not. He knows that FDA fetishizes journal articles, as evidenced by the series of posts I wrote early this year about their reviews of e-cigarettes, which were summarized in CASAA’s comment on the proposed deeming regulation.
Nothing short of a retraction — and that basically does not happen for anything short of data fabrication or the equivalent — changes the fact that this is a “peer-reviewed journal article”. Nothing. Not the fact that its conclusions are not supported by the paper, not the fact that the research was unethical, not the fact that it was a terrible excuse for research whose results were misreported, and not the fact that the journal felt it appropriate to put a warning label on it.
The only way to deal with this problem is to push for recognition of the fact that just because something is a “peer-reviewed journal article” does not mean that the conclusions, observations, or even the reported study results are accurate. Indeed, in the realm of tobacco research, it does not even suggest they are probably accurate.