Tag Archives: Popova

New Phillips-Burstyn-Carter working paper on the failure of peer review in public health

by Carl V Phillips

Our new working paper is available at EP-ology, “The limited value of journal peer review in public health: a case series of tobacco harm reduction articles”, Carl V Phillips, Igor Burstyn, and Brian L Carter (all of the authors are affiliated with CASAA, for those who may not know).

Abstract:

Background: A widespread belief holds that the journal peer-review process has magical powers to ensure that published claims are correct. While this misperception has limited consequences in many fields, in public health it results in consumer, clinical, and policy decisions being based on blind faith in the accuracy of published claims. At best, the review process is merely a couple of readers — perhaps, but not necessarily, highly expert — reading through a paper to ensure the research and presentation are reasonably sound. In reality, even this is often not accomplished.

Methods: We conducted reviews of 12 articles that focused on tobacco harm reduction published in a mainstream public health journal, BMC Public Health, consecutively during 2012-15. We each wrote a reviewer report of the manuscript version that was sent to the journal reviewers, as if we were writing a review for a journal. We then compared these to the reviews written by the journal reviewers. Additionally, we reviewed the changes made to the papers as a result of the journal reviews.

Results: Almost all the papers in the dataset suffered from major flaws, most of which could have been corrected, but none were corrected by the journal review process. The journal peer reviews were almost all inadequate and many contained no substantive comments. Those that contained substantive observations still did not identify most of the blatant major flaws that we noted. In the single case where a journal reviewer identified many of the major flaws, the comments were basically ignored by the authors and the paper was published with no substantive changes. Other than cosmetic improvements, the journal review process was about as likely to make the published version worse than the submitted manuscript, rather than better. Papers with no apparent value were published by the journal and the potential value of other studies was lost because serious flaws in the paper were ignored. Unreported conflict of interest was common among both authors and reviewers.

Conclusions: Faith in the journal peer-review process is misplaced. Even at best, the process cannot promise that a published claim is correct, but in reality it does not even ensure that patent major flaws are not present. In public health, the phrase “according to a peer-reviewed journal article” seems to mean little more than “I read this somewhere.”

It should be evident from the abstract that the primary study aim is not about THR. However, readers of this blog may be interested for several reasons. Most obviously, the case studies are based on articles about THR. But also, the idolatry of journal peer review is one of the more important causes of the persistence of anti-THR lies. Analyzing the reviews of the papers, not the papers themselves, was the purpose of the research, but that required assessing the papers en passant, which means that readers interested in that aspect will should find a fair bit of the content interesting (particularly delving into a few of the appendices).

Some of the material has already been covered here. The previous post was basically written as a footnote for the paper. The post about the terrible paper by Hughes (“Associations between e-cigarette access and smoking and drinking behaviours in teenagers”, by Karen Hughes, Mark A Bellis, Katherine A Hardcastle, Philip McHale, Andrew Bennett, Robin Ireland, and Kate Pike) was basically an excerpt from that research project. The extensive analysis of the Popova-Ling travesty was incorporated as part of the analysis in the paper.

There are few other papers in the analyzed case series that also fall solidly within the anti-THR lies category. There is this one (“Portrayal of electronic cigarettes on YouTube”, Chuan Luo, Xiaolong Zheng, Daniel Dajun Zeng, Scott Leischow) whose value lies entirely in it being unintentional comedy. Strangely, despite being a useless, silly, and badly conducted study, that was then written up as a political broadside that had nothing to do with the study results, it was probably only the 10th worst of the 12 papers in our case series.

Two of the papers were written to try to vilify snus. This one (“Snus user identity and addiction. A Swedish focus group study on adolescents”, Ingrid Edvardsson, Margareta Troein, Göran Ejlertsson, Lena Lendahls) is mostly just uninteresting in terms of what it claims. This one (“Predictors of smoking among Swedish adolescents” Junia Joffer, Gunilla Burell, Erik Bergström, Hans Stenlund, Linda Sjörs, Lars Jerdén) relates more closely to recurring themes from this blog. In particular, it makes the naive gateway claim that fails to distinguish between association and causation. This is particularly pathetic in this case because the paper is ostensibly about predictors and not causes.

Comments on the working paper are welcome, either in the blog comments at EP-ology or via email.

Advertisements

The failures of peer review do not begin with the journal – more on the Popova-Ling fiasco

by Brian L Carter

[Editor’s note: This is a guest post by CASAA advisor Brian Carter, who is working with me on research on peer review. He took the lead on part of the behind-the-scenes effort to get the Popova and Ling article retracted, an effort whose failure (and the irony of that failure) I covered in the previous post. –CVP]

Those of us who admire the elegance and clarity of thought contained in good scientific reasoning no doubt found Popova and Ling’s report of a study on warning labels severely lacking. The article is here and Carl V Phillips’s and Clive Bates’s devastating critiques of it are hereherehere, and in comments attached to the article at the journal here. People who manage to publish worthless junk out of sheer scientific ignorance are worrisome enough. But special condemnation is called for when people manage to combine their cluelessness with malicious intent. They use the language and trappings of science like a facade, all to support their decidedly unscientific personal policy goals.

At first, it’s difficult to understand how such an ill-conceived, poorly executed, and scientifically vacuous study could have ever been conducted, much less see the light of day in a respected journal like BMC Public Health. The journal peer review process, which we count on to at least identify utter junk science, was a colossal failure from start to finish, as documented here. However, this most basic failure was simply the last in a long line of peer review failures, aided by willful institutionalized ignorance and prejudice.

Beginning with the release of the article, we can work backwards, like crime scene investigators, to trace the various malicious acts back to the original fraud that formed the ideological genesis for this article. Popova and Ling note in the article that the National Cancer Institute (an organ of the National Institutes of Health), funded their study through a grant awarded to Pamela Ling. This means the very ideas behind the study, the background, the logic, the rationale for doing it, had to be blessed by an expert panel of scientific grant reviewers. These reviewers supposedly make their decisions on the basis of good scientific judgment. If you don’t make your case at this stage, you don’t get the money: Ling had to make a compelling argument for doing what she proposed, and she had to do it better than about 95% of the other applicants because there’s usually only enough money to dole out to the very highest scoring grant applications.

The National Institute of Health publishes information about every awarded grant on its Research Portfolio Online Reporting Tools (RePORT). One of the many pieces of information contained on this site is the applicant’s summary description of the proposed research. Ling’s is here.

Although we are not privy to the study details she proposed, this description clearly supports the thinking and methodology she used and subsequently published. The description is a checklist of the standard University of California San Francisco lies, evasions, and fallacies. Here are the lowlights:

 “Tobacco use is responsible for 35% of cancer deaths, and new smokeless tobacco marketing efforts threaten both to increase cancers caused by smokeless use . . .”

“new smokeless products are marketed as line extensions of major cigarette brands (Marlboro and Camel) to promote ‘dual use.’”

“These changes in smokeless tobacco marketing may blunt the effects of smokefree environments and the health benefits of smokers’ cutting down and quitting.”

“test new counter-marketing messages to block initiation of smokeless tobacco use among novices and the dual use of smokeless tobacco and cigarettes as an alternative to smoking cessation.”

“Findings will be relevant to guide development of policies on smokeless marketing and advertising.” 

That this grant was reviewed and given a top score tells us something about the review committee charged with evaluating its scientific rigor. I have sat on several grant review committees, and each grant has three (sometimes two) primary reviewers (just as journal articles do). With as many as 60-80 grants to review, each member is a primary reviewer on 3-4 grants and usually defers to the primary reviewers of the other grants when submitting a vote on quality score. In this way, the grant review system is embarrassingly similar to the journal review system. The only major difference is that with grant review there are 20 or so potential reviewers, and they all are supposed to be highly qualified to sit on the committee. Most have to have been awarded a grant themselves, be highly published and fairly well known in the field related to the RFA, and have the demonstrated expertise to evaluate the grants they are assigned. However, this academic record is obviously no guarantee committee members are blessed with deep critical thinking skills or untainted by strong political bias.

Clearly, the primary reviewers for Ling’s grant were fellow travelers able to ignore (or, through confirmatory bias, simply not see) the clear evidence that smokeless tobacco use has trivial cancer risk, if any, a well researched scientific finding that makes the entire premise of the grant specious. None of the other 15-20 committee members (who have access to all grants under review, and usually have read the summary descriptions) apparently had any serious objections either. Any one of them could have raised the point of smokeless tobacco’s trivial risks and demanded a debate on the topic, a discussion that could have significantly lowered Ling’s score. So Ling’s grant sailed to the top of the score list in much the same way it sailed onto the pages of BMC Public Health.

How does a grant review committee so ignorant of smokeless tobacco pass muster on Ling’s grant? For this clue we have to dig a little further. In the “details” section of Ling’s RePORT page we discover she had submitted her grant under a specific Request For Applications (RFA). The NIH frequently publicizes RFAs when it wants some narrowly focused research applications to address a particular topic area–in this case, an RFA titled, “Measures and Determinants of Smokeless Tobacco Use, Prevention, and Cessation.” The RFA is a special invitation to submit a grant tailored to it. Full text here: http://grants.nih.gov/grants/guide/rfa-files/RFA-CA-08-024.html.

Most of the RFA contains instructions on how to apply, but the most important piece of information is listed in the executive summary under “purpose.” This text describes what the issuers of the RFA want in an application and helps explain the rubber stamp of the review committee.

“The overall goal is to develop an evidence base to inform smokeless tobacco control efforts, and to develop effective ways to limit the spread and promote cessation of smokeless tobacco use.”

There you have it. From the very beginning the federal government, in the form of the National Cancer Institute, deliberately solicited applications for the express purpose of figuring out how to “limit the spread” of smokeless tobacco, especially for smokers who might fall into the fictitious trap of dual use in an attempt to switch. They formed a review committee that would stick to the flawed central premise of the RFA. The premise was right in Popova and Ling’s wheelhouse, and they naturally proposed an experiment going further than merely misleading smokeless tobacco users with false labeling, but adding some graphic and disturbing images to boot. You can fault Popova and Ling’s ignorance of good scientific practice, and their shameless attempt to use their wreck of a study in a shabby attempt to influence FDA policy. But you can’t accuse them of failing to deliver exactly what the government wanted.

Glantz complains about research ethics LOLOLOL

by Carl V Phillips

Yes, the man whose superpower is an inhuman ability to willfully misinterpret study results and lie to the public based on that (and who is completely immune to the effects of evidence, logical argument, authors telling him he is interpreting their studies wrong, etc.) is complaining about research ethics. In particular, he is complaining about the recent Shiffman et al. paper which demonstrated that the prospect of interesting flavors did basically nothing to entice teenage non-users toward wanting to use e-cigarettes.

Glantz wrote an extremely weak letter to the journal that published the paper which, to its credit, rejected it. The study had some definite limitations and I would say that the authors did over-conclude from their results, which Glantz tried to say (a stopped clock is right twice a day, though is seldom so deliciously ironic when it is). But the basic conclusion of the study — that interesting e-cigarette flavors provoke a collective yawn among teenagers who do not use tobacco — is solid. It is this result that flatly contradicts a favorite claim of Glantz and his cabal and that Glantz, completely unsuccessfully, tried to challenge. Continue reading

What is peer review really? (part 4a – case study followup)

by Carl V Phillips

I thought it would be worth taking this series non-linear to follow up on Part 4, which used the recent Popova-Ling “peer-reviewed journal article” as a case-study to illustrate much of what is wrong with journal peer review and the fetishizing thereof in “public health”. Popova and Glantz relied on that paper in the comment to the FDA that I discussed in yesterday’s post about Swedish Match’s MRTP application, which asks permission to remove the false and misleading “warning” labels from their products. This is a great illustration of why the fetishization, “it is in a peer-reviewed journal, so it must be right”, is such a dangerous travesty. (H/t to Brian Carter inspiring some of the observations that appear here.) Continue reading

What is peer review really? (part 4 — a case study)

by Carl V. Phillips

[Update: I have submitted a comment to BMC Public Health that is based on this post. My copy of it can be viewed here.]

[Update: The comment has now been accepted by the journal and appears, attached to the original article, here.]

I interrupt the flow of this series, in which I am currently laying out some common myths about journal peer-review, to provide a motivational case study that makes many points better than any abstract principles can. As I discussed in the previous post, which built on what Clive Bates had already written, a newly published article by Popova and Ling was unethical and misleading, fraught with anti-THR lies. But here is the good news: It was published in a Biomed Central (BMC)  journal. While BMC still basically practices the 20th-century version of peer-review that I have pointed out to be a failure, they do not keep it an anonymous black-box like most journals do. (This is a huge improvement over the standard health science practice — enough so that when I started a journal, I chose to do it at BMC — though still far short of other fields’ real peer review, as I have discussed previously in this series.) Thus, we can review not only the paper, but the “peer-reviews” that caused it to be published. Continue reading

New public health research: lying to people can affect them (as if they didn’t already know)

by Carl V Phillips

A new paper in the normally more-respectable BMC Public Health, by never-respectable ANTZ at the University of California (San Francisco) reports research that mostly showed that, if people were given disinformation claiming (nonexistent) health effects from smokeless tobacco and e-cigarettes, accompanied by gory pictures, then they will be tricked into to thinking the risk was higher. Surprise!

Well, of course, it is no surprise that people can be tricked and no surprise that UCSF “researchers” would conduct such unethical research. It is rather more of a surprise that the non-ANTZ BMC Public Health would publish it and that an ethics committee would allow it to be done. Ok, maybe not the latter — the ethics committees are pretty much in the pocket of public health. That committee at UCSF probably would never allow, say, Farsalinos’s survey of e-cigarette users, and would trump up some claim that it was a threat to the study subjects, whereas they allowed serial liars Lucy Popova and Pamela M Ling a free hand to tell people they might as well smoke.

Anyway, Clive Bates was first off the block in responding to this travesty, and he covered the breadth of it well, so I am not going to reinvent the wheel here.  Go read what he wrote first. Then come back to this, wherein I go deeper into a few specific points. Continue reading

ANTZ telling the truth… about the fact that they are liars

by Carl V Phillips

Hmm, hard to decide whether this counts as truth or lying.  When serial liars actually publish a statement about how to be more effective liars, does that count as rare moment of honesty?  I think, perhaps, that it would be honesty if they admitted that what they were doing was figuring out how to lie better.  Alas, they never actually admit that, so they are even lying in their discussion of how to lie.

The case in point is a relatively new article, published in the journal/comic book, Tobacco Control (which is remarkably difficult to find a copy of  because very few libraries subscribe to it — good for them!).  It examines how to better trick smokers into believing that smokeless tobacco use is far more risky than it really is (the actual risk is, of course, too small to even detect), so that they are not inclined to try THR.  The authors do not really matter (their names, just for search purposes: Lucy Popova, Torsten B Neilands, Pamela M Ling); all you really need to know is that they are minions in the Glantz ANTZ colony at University of California, San Francisco.

So, they probably do not even understand that there is such a concept as lying.  In their world, you say whatever is most useful to get people to do what you want, and the fact that this manipulation might well kill them is of no greater consequence than the act of lying.  Yes, that’s right — this is an article that is openly about teaching people how to be more successful sociopaths.

They performed an experiment on people, showing them snus ads and then various anti-THR lies which they call “antismokeless tobacco ads”, basically admitting — through the use of the word “ads” — that their tobacco control enterprise is every bit as much an industry as the tobacco manufacturing business (though I give them no credit for accidental honesty).  The word they use for “lying” in the paper is “countermarketing”.

They natter on (without saying anything concrete — this is not a real scientific paper, of course) about their wonderful techniques for optimizing their propaganda (indeed, the experiment appears really to just be a window-dressing excuse for talking about their lie-development efforts), talking about effectiveness, focus groups, how they dissected Secret Evil Industry Documents; never once do they make any reference to whether one of their messages is true or not.  It apparently does not occur to them to care about such matters.  They might as well just note in the conflict-of-interest disclosure that they are not capable of conflict of interest because they are sociopaths who do not even understand the concept of ethical obligations.

In describing their “countermarketing”, they report that they:

emphasised similarities between all tobacco products in a straightforward, informational manner

Wait, which was it?  Were they straightforward and informational, or did they claim there all tobacco products are similar in important ways?  That journal could really use a literate editor who could recognize that an extended typo has rendered the sentence internally inconsistent.

They

pointed out tobacco industry attempts to ‘push smokeless gimmicks at smokers’

Yes, damn those tobacco companies for trying to get smokers to embrace the gimmick of switching to near-harmless products.  Gimmicks like that are a serious threat to the tobacco control industry’s business model, which depends on people continuing to smoke.  Keep smoking, dammit!

And then there was:

comparing smokers who used novel smokeless tobacco products to lab rats used by the tobacco industry to test their new products

Snus is so novel that it has been used for merely a few centuries, is the dominant form of tobacco use in only one medium-sized country, and is merely the second-most-studied tobacco product.  Just because its risks have been more precisely estimated than any other tobacco/nicotine product does not mean we know anything.  Geez, anyone exposing people to that might as well be brewing things at random in an organic chemistry lab and passing them out to drunk people at nightclubs, just to see what happens.  I mean, how dare that tobacco industry try to sell smokers something until we know more.  Until we know for sure whether it is 99.5% less harmful than smoking or merely 98% less harmful, it is far better to keep them smoking.

And, of course, the authors lied to their lab rats that snus causes oral cancer and implied that smoking does not, though the evidence actually supports the opposite claim.

What is worse, they performed their experiments on smokers and recently abstinent smokers who they judged to be at the highest risk for starting smoking again — i.e., the people who would benefit from learning the truth about snus, and thus were directly harmed by being in the experiment.  [UPDATE: In thinking about this more, it occurred to me that I had perhaps understated the importance of this observation.  This was not targeted at nonusers who might have considered trying snus, so they cannot even pretend to be “thinking of the children”.  This was a targeted effort specifically intended to figure out how to keep smokers from switching to snus.]

Remember (or be sure to learn, if you do not already know it) that Rodu and I showed that as of 2000 in the US, well before “snus”-branded products increased interest in THR, switching to smokeless tobacco was more likely to be successful than any other smoking cessation method.  Since 2000, the increase in smokeless tobacco use has closely matched the decline in smoking, suggesting that substitution has been the only thing moving smoking rates down.

So what did the authors of this “study” conclude?  Who cares, really?  They are liars to such an extent that they are voluntarily announcing that fact in a paper, so reading what they concluded is likely to only make us less knowledgable?  Besides, those methods for measuring people’s responses to stimuli like the anti-THR lies in this experiment are pretty close to junk science even when deployed by honest researchers.  People’s snap reaction to their first viewing of an advertisement or gory image has little impact on their views a short time later.  The only good news is that when the authors lie to their fellow ANTZ about the definitive implications of their research, they are mostly just wasting their own time (though they are still wasting taxpayers’ money).