New Phillips-Burstyn-Carter working paper on the failure of peer review in public health

by Carl V Phillips

Our new working paper is available at EP-ology, “The limited value of journal peer review in public health: a case series of tobacco harm reduction articles”, Carl V Phillips, Igor Burstyn, and Brian L Carter (all of the authors are affiliated with CASAA, for those who may not know).

Abstract:

Background: A widespread belief holds that the journal peer-review process has magical powers to ensure that published claims are correct. While this misperception has limited consequences in many fields, in public health it results in consumer, clinical, and policy decisions being based on blind faith in the accuracy of published claims. At best, the review process is merely a couple of readers — perhaps, but not necessarily, highly expert — reading through a paper to ensure the research and presentation are reasonably sound. In reality, even this is often not accomplished.

Methods: We conducted reviews of 12 articles that focused on tobacco harm reduction published in a mainstream public health journal, BMC Public Health, consecutively during 2012-15. We each wrote a reviewer report of the manuscript version that was sent to the journal reviewers, as if we were writing a review for a journal. We then compared these to the reviews written by the journal reviewers. Additionally, we reviewed the changes made to the papers as a result of the journal reviews.

Results: Almost all the papers in the dataset suffered from major flaws, most of which could have been corrected, but none were corrected by the journal review process. The journal peer reviews were almost all inadequate and many contained no substantive comments. Those that contained substantive observations still did not identify most of the blatant major flaws that we noted. In the single case where a journal reviewer identified many of the major flaws, the comments were basically ignored by the authors and the paper was published with no substantive changes. Other than cosmetic improvements, the journal review process was about as likely to make the published version worse than the submitted manuscript, rather than better. Papers with no apparent value were published by the journal and the potential value of other studies was lost because serious flaws in the paper were ignored. Unreported conflict of interest was common among both authors and reviewers.

Conclusions: Faith in the journal peer-review process is misplaced. Even at best, the process cannot promise that a published claim is correct, but in reality it does not even ensure that patent major flaws are not present. In public health, the phrase “according to a peer-reviewed journal article” seems to mean little more than “I read this somewhere.”

It should be evident from the abstract that the primary study aim is not about THR. However, readers of this blog may be interested for several reasons. Most obviously, the case studies are based on articles about THR. But also, the idolatry of journal peer review is one of the more important causes of the persistence of anti-THR lies. Analyzing the reviews of the papers, not the papers themselves, was the purpose of the research, but that required assessing the papers en passant, which means that readers interested in that aspect will should find a fair bit of the content interesting (particularly delving into a few of the appendices).

Some of the material has already been covered here. The previous post was basically written as a footnote for the paper. The post about the terrible paper by Hughes (“Associations between e-cigarette access and smoking and drinking behaviours in teenagers”, by Karen Hughes, Mark A Bellis, Katherine A Hardcastle, Philip McHale, Andrew Bennett, Robin Ireland, and Kate Pike) was basically an excerpt from that research project. The extensive analysis of the Popova-Ling travesty was incorporated as part of the analysis in the paper.

There are few other papers in the analyzed case series that also fall solidly within the anti-THR lies category. There is this one (“Portrayal of electronic cigarettes on YouTube”, Chuan Luo, Xiaolong Zheng, Daniel Dajun Zeng, Scott Leischow) whose value lies entirely in it being unintentional comedy. Strangely, despite being a useless, silly, and badly conducted study, that was then written up as a political broadside that had nothing to do with the study results, it was probably only the 10th worst of the 12 papers in our case series.

Two of the papers were written to try to vilify snus. This one (“Snus user identity and addiction. A Swedish focus group study on adolescents”, Ingrid Edvardsson, Margareta Troein, Göran Ejlertsson, Lena Lendahls) is mostly just uninteresting in terms of what it claims. This one (“Predictors of smoking among Swedish adolescents” Junia Joffer, Gunilla Burell, Erik Bergström, Hans Stenlund, Linda Sjörs, Lars Jerdén) relates more closely to recurring themes from this blog. In particular, it makes the naive gateway claim that fails to distinguish between association and causation. This is particularly pathetic in this case because the paper is ostensibly about predictors and not causes.

Comments on the working paper are welcome, either in the blog comments at EP-ology or via email.

Post publication peer-review: Correction to Burstyn (2014) and related matters

by Carl V Phillips and Igor Burstyn

[Igor Burstyn is an Associate Professor at Drexel University School of Public Health and a member of the CASAA Board of Directors. His research that is described here was sponsored by CASAA.]

Burstyn (2014), “Peering through the mist: systematic review of what the chemistry of contaminants in electronic cigarettes tells us about health risks”, BMC Public Health, is probably the most read scientific paper on e-cigarettes and among the most read in the history of tobacco harm reduction. It is often described as the most important paper on e-cigarettes, being the first to point out that there is ample existing evidence that the non-novel chemical exposures from vaping – which are used to concoct alarmist propaganda — are inconsequential. So imagine our surprise when, after well over 100,000 people had viewed the paper at the journal’s website and countless more via other means (the announcement of the publication of the working paper version remains the most read post on this blog), it went through journal peer-review, and each of us poured over many revisions, one astute reader caught a bright-line error in the results. It is recounted in the following text by IB: Continue reading

“Second-hand vapor”: what tobacco controllers would say if they really believed it was a problem

by Carl V Phillips

None of the people nattering on about the environmental effects of vaping are actually concerned about what they claim.  This has occurred to me before, but it was made particularly clear at the recent FDA workshop that I recounted here and here. It is a very simple point: If there were anyone out there who was actually worried about the effects of “second-hand vapor” or (giggle) “third-hand vapor”, rather than just using claims about them to rationalize attacks on tobacco harm reduction (THR), they would be encouraging the use of smokeless tobacco (ST) instead. Continue reading

More on the FDA ecig workshop

by Carl V Phillips

Those of you who watched my contribution to the workshop (which you can do by following the link in yesterday’s post) probably found the most memorable observation to be the one about San Francisco. But I am rather prouder of not missing a beat regarding a later question. Leading into that, there was a rambling multi-part question to the panel, which a couple of others responded to bits of. I took the mic last to respond to the phrase “renormalizing smoking” in the question. Continue reading

Notes from the FDA e-cigarette workshop

by Carl V Phillips

I have several posts worth of material to cover on the themes presented here. I’ll see if I can get to them. In the meantime, some random thoughts about the third FDA Center for Tobacco Products workshop on e-cigarettes that took place on Monday and Tuesday of this week. As with the previous workshops, it was a mixed bag of good science from industry and a few others, sneaking in amidst the utter junk from the “public health” types. At this one, the mix was perhaps almost up to 50-50, which is a lot better than the previous ones.  Continue reading

Gateway effects and snus taxes in Alberta

by Catherine M Nissen

[Editor’s Note: This is a guest post by CASAA Research Associate, Catherine Nissen. –CVP]

As most readers of this blog know, the gateway effect argument has been used aggressively by anti-THR advocates against smokeless tobacco and electronic cigarettes. They claim that using the low-risk alternatives will cause people to start smoking. Recently I have been working with Carl V Phillips on his paper about detecting gateway effects. He explains the logic of the claim and why it would be unlikely to occur in natural circumstances. But, as he explains, policy actions could manage to create gateway effects, either intentionally or inadvertently. Even then, he was able to offer only a few examples of where it might have happened. I am living in the middle of one of those examples, in western Canada.

As Carl has noted a few times (example), several nonsmoker experts on low-risk tobacco products have started using snus as adults because they learned about the minimal health risk and wanted to reap the benefits of the cognitive enhancement and risk reduction in neurodegenerative diseases that tobacco offers. While I may not be among the esteemed senior scientists he is referring to, and I admit I use snus a bit more for the pure pleasure of it rather than the cognitive enhancement, I am among those who took up snus use after learning about the low costs and substantial benefits. My grandmother suffers from Parkinson’s, and the risk reduction from it that tobacco offers is definitely among my motivations for continued use. I am also among the unfortunate few who have been pushed from low-risk product use into smoking some of the time. I am not sure whether this counts as a gateway case. It certainly is not an example of what the tobacco controllers have in mind when they make the gateway claim, since they are the ones responsible for it.

I have only smoked very occasionally throughout my life, yet I enjoy using snus (smokeless tobacco). I have a firmly established preference for nicotine, but I also have firmly established boundaries on how much I am willing to pay to enjoy it. I am a nonsmoker because the cost of smoking, health-wise, was always too much for me.

After Carl established the tobacco harm reduction research and education efforts at the University of Alberta School of Public Health in the 2000s (a group that would later include me, Karyn Heavner, and several others who have contributed to the field, along with Igor Burstyn sitting in), he, Paul Bergen, and some of Carl’s students launched the first concerted attempt to widely popularize the THR message. In response to this, the anti-tobacco people in Alberta shifted their effort away from trying to discourage smoking to trying to discourage THR. It worked. Today the taxes here on smokeless tobacco are exorbitant. I pay almost $25 for a tin of snus. A pack of cigarettes is about $12, less than half the cost. The price of a tin of snus in the U.S. or overseas is about $4. Under those economic circumstances, only people who are both wealthy enough and highly motivated would choose smokeless tobacco it in lieu of cigarettes. For a person of average means, this price difference is a strong incentive not to try THR, even if they are aware of its benefits.

I have a friend who was a very avid smoker, but armed with the knowledge of THR (simply because he knew me, and despite “public health” efforts to sow disinformation) and the desire to quit smoking, he switched successfully to snus. It wasn’t easy at first, but he became a regular user. As the prices kept rising, though, he moved back to cigarettes. The balance of cost and benefit in his case gave the edge to cigarettes. Eventually, however, he moved back to snus, despite the cost, because of his desire to be smoke-free. He is being punished for that choice, at a rate of about almost $5000 per year more than he would pay for smoking. The difference is because the government is charging him over $7000 per year in taxes on snus, a price he pays for not smoking.

It was much the same with me. I found myself strapped for cash and not willing to pay the cost of snus. My initial brief thoughts on buying cigarettes instead (as I talked about here) became much more prevalent. I caved to them and did buy cigarettes for a time. However, I found I could not smoke more than a pack a week because the health effects from them were so immediate and unpleasant. At the end of a week I would cough throughout my day and my taste and smell were affected. I ended up giving up on cigarettes and now just use snus infrequently. This lowers my welfare substantially from what it would be, and provides no benefit for me or anyone else.

Still, I am luckier than many others. There are countless stories around here of people who prefer smokeless tobacco and have a history of using it instead of smoking sometimes, and being tempted to switch because of the absurd anti-THR price differential. Many of them are also victims of the disinformation about smokeless tobacco and so think there is little or no health difference between the products. For me, the punitive tax on snus made smoking a more attractive option in the short-term, but near-abstinence was more attractive because I understand the huge increase in risk with smoking and because I don’t gain as much from tobacco use as some others do. But others will just choose to smoke. I don’t personally know any smokeless tobacco users who switched to being exclusive smokers because of this tax, but I’m sure there are some because there are many smokeless tobacco users in the region.

So we have a demonstration of policy actions causing a gateway effect from smokeless products to cigarettes (for both a nonsmoker and an ex-smoker). For people uninformed about the health benefits of THR, high taxes on smokeless products (and lower taxes on the more harmful alternative) discourage it. People respond to price in the near term, especially when they are strapped for cash. This is a clear argument for making sure that THR products remain at lower prices than cigarettes, not higher.

As far as I can tell, my story is the only published testimonial of someone who was never a smoker (and clearly never would have become a smoker) who started using smokeless tobacco, and as a result of that became a smoker (though only temporarily). The proponents of the gateway claim never give any examples of people who seem to be gateway cases, which is one of the many reasons that their theory does not hold up to scientific scrutiny. I doubt that my story is what they have in mind, however, since anti-THR efforts were the ultimate cause of this. Without those “public health” policies, I would not have become a smoker.

I am sure a much more common story is like that of my friend who was encouraged to return to smoking even though he wanted to switch, which you might or might not call a gateway effect. This same effect seems to have happened with electronic cigarettes in Spain. It may be happening with electronic cigarettes elsewhere (the UK, California), because disinformation about their risk is tricking many people into believing they might as well smoke. It could certainly happen if all or most e-cigarettes are banned, as the U.S. FDA has proposed.

There is a high prevalence of “natural” smokeless tobacco use in Alberta, probably more than anywhere else outside of Scandinavia and a few rural subpopulations in the USA, because so many people work in jobs where smoking would be difficult. Most likely they don’t know they are engaging in THR (thanks to the disinformation campaigns that are ongoing here). Some may simply enjoy the fact that they are coughing less and experiencing other health benefits from reducing or quitting their smoking. In any case, giving these people a good reason to smoke instead is potentially creating a gateway effect to smoking, and yet it is created by the very people who claim to be worried about gateway effects.

 

The failures of peer review do not begin with the journal – more on the Popova-Ling fiasco

by Brian L Carter

[Editor’s note: This is a guest post by CASAA advisor Brian Carter, who is working with me on research on peer review. He took the lead on part of the behind-the-scenes effort to get the Popova and Ling article retracted, an effort whose failure (and the irony of that failure) I covered in the previous post. –CVP]

Those of us who admire the elegance and clarity of thought contained in good scientific reasoning no doubt found Popova and Ling’s report of a study on warning labels severely lacking. The article is here and Carl V Phillips’s and Clive Bates’s devastating critiques of it are hereherehere, and in comments attached to the article at the journal here. People who manage to publish worthless junk out of sheer scientific ignorance are worrisome enough. But special condemnation is called for when people manage to combine their cluelessness with malicious intent. They use the language and trappings of science like a facade, all to support their decidedly unscientific personal policy goals.

At first, it’s difficult to understand how such an ill-conceived, poorly executed, and scientifically vacuous study could have ever been conducted, much less see the light of day in a respected journal like BMC Public Health. The journal peer review process, which we count on to at least identify utter junk science, was a colossal failure from start to finish, as documented here. However, this most basic failure was simply the last in a long line of peer review failures, aided by willful institutionalized ignorance and prejudice.

Beginning with the release of the article, we can work backwards, like crime scene investigators, to trace the various malicious acts back to the original fraud that formed the ideological genesis for this article. Popova and Ling note in the article that the National Cancer Institute (an organ of the National Institutes of Health), funded their study through a grant awarded to Pamela Ling. This means the very ideas behind the study, the background, the logic, the rationale for doing it, had to be blessed by an expert panel of scientific grant reviewers. These reviewers supposedly make their decisions on the basis of good scientific judgment. If you don’t make your case at this stage, you don’t get the money: Ling had to make a compelling argument for doing what she proposed, and she had to do it better than about 95% of the other applicants because there’s usually only enough money to dole out to the very highest scoring grant applications.

The National Institute of Health publishes information about every awarded grant on its Research Portfolio Online Reporting Tools (RePORT). One of the many pieces of information contained on this site is the applicant’s summary description of the proposed research. Ling’s is here.

Although we are not privy to the study details she proposed, this description clearly supports the thinking and methodology she used and subsequently published. The description is a checklist of the standard University of California San Francisco lies, evasions, and fallacies. Here are the lowlights:

 “Tobacco use is responsible for 35% of cancer deaths, and new smokeless tobacco marketing efforts threaten both to increase cancers caused by smokeless use . . .”

“new smokeless products are marketed as line extensions of major cigarette brands (Marlboro and Camel) to promote ‘dual use.’”

“These changes in smokeless tobacco marketing may blunt the effects of smokefree environments and the health benefits of smokers’ cutting down and quitting.”

“test new counter-marketing messages to block initiation of smokeless tobacco use among novices and the dual use of smokeless tobacco and cigarettes as an alternative to smoking cessation.”

“Findings will be relevant to guide development of policies on smokeless marketing and advertising.” 

That this grant was reviewed and given a top score tells us something about the review committee charged with evaluating its scientific rigor. I have sat on several grant review committees, and each grant has three (sometimes two) primary reviewers (just as journal articles do). With as many as 60-80 grants to review, each member is a primary reviewer on 3-4 grants and usually defers to the primary reviewers of the other grants when submitting a vote on quality score. In this way, the grant review system is embarrassingly similar to the journal review system. The only major difference is that with grant review there are 20 or so potential reviewers, and they all are supposed to be highly qualified to sit on the committee. Most have to have been awarded a grant themselves, be highly published and fairly well known in the field related to the RFA, and have the demonstrated expertise to evaluate the grants they are assigned. However, this academic record is obviously no guarantee committee members are blessed with deep critical thinking skills or untainted by strong political bias.

Clearly, the primary reviewers for Ling’s grant were fellow travelers able to ignore (or, through confirmatory bias, simply not see) the clear evidence that smokeless tobacco use has trivial cancer risk, if any, a well researched scientific finding that makes the entire premise of the grant specious. None of the other 15-20 committee members (who have access to all grants under review, and usually have read the summary descriptions) apparently had any serious objections either. Any one of them could have raised the point of smokeless tobacco’s trivial risks and demanded a debate on the topic, a discussion that could have significantly lowered Ling’s score. So Ling’s grant sailed to the top of the score list in much the same way it sailed onto the pages of BMC Public Health.

How does a grant review committee so ignorant of smokeless tobacco pass muster on Ling’s grant? For this clue we have to dig a little further. In the “details” section of Ling’s RePORT page we discover she had submitted her grant under a specific Request For Applications (RFA). The NIH frequently publicizes RFAs when it wants some narrowly focused research applications to address a particular topic area–in this case, an RFA titled, “Measures and Determinants of Smokeless Tobacco Use, Prevention, and Cessation.” The RFA is a special invitation to submit a grant tailored to it. Full text here: http://grants.nih.gov/grants/guide/rfa-files/RFA-CA-08-024.html.

Most of the RFA contains instructions on how to apply, but the most important piece of information is listed in the executive summary under “purpose.” This text describes what the issuers of the RFA want in an application and helps explain the rubber stamp of the review committee.

“The overall goal is to develop an evidence base to inform smokeless tobacco control efforts, and to develop effective ways to limit the spread and promote cessation of smokeless tobacco use.”

There you have it. From the very beginning the federal government, in the form of the National Cancer Institute, deliberately solicited applications for the express purpose of figuring out how to “limit the spread” of smokeless tobacco, especially for smokers who might fall into the fictitious trap of dual use in an attempt to switch. They formed a review committee that would stick to the flawed central premise of the RFA. The premise was right in Popova and Ling’s wheelhouse, and they naturally proposed an experiment going further than merely misleading smokeless tobacco users with false labeling, but adding some graphic and disturbing images to boot. You can fault Popova and Ling’s ignorance of good scientific practice, and their shameless attempt to use their wreck of a study in a shabby attempt to influence FDA policy. But you can’t accuse them of failing to deliver exactly what the government wanted.

Peer review – are they really even trying anymore?

by Carl V Phillips

As part of our research on journal peer review in public health — the practice and interpretation of which is a dire threat to THR and other policies based on good science — my colleagues and I found ourselves contemplating this report at Retraction Watch from March, about BioMed Central (BMC) retracting 43 published articles for improprieties in the peer review process. We were bitterly reminded of BMC’s lack of retraction for the travesty of an article by Lucy Popova and Pamela M Ling in BMC Public Health. Those authors claimed that they had demonstrated that harsher warning labels about smokeless tobacco and e-cigarettes were warranted, when actually their results — such as they were — supported the opposite conclusion. You may recall from my previous posts that the Popova-Ling paper was excoriated as unethical, utterly useless, and misleading by me, Brian Carter, Clive Bates, and others. The journal considered retracting it after our complaints, but ultimately decided that was all fine by them. Continue reading