The failures of peer review do not begin with the journal – more on the Popova-Ling fiasco

by Brian L Carter

[Editor’s note: This is a guest post by CASAA advisor Brian Carter, who is working with me on research on peer review. He took the lead on part of the behind-the-scenes effort to get the Popova and Ling article retracted, an effort whose failure (and the irony of that failure) I covered in the previous post. –CVP]

Those of us who admire the elegance and clarity of thought contained in good scientific reasoning no doubt found Popova and Ling’s report of a study on warning labels severely lacking. The article is here and Carl V Phillips’s and Clive Bates’s devastating critiques of it are hereherehere, and in comments attached to the article at the journal here. People who manage to publish worthless junk out of sheer scientific ignorance are worrisome enough. But special condemnation is called for when people manage to combine their cluelessness with malicious intent. They use the language and trappings of science like a facade, all to support their decidedly unscientific personal policy goals.

At first, it’s difficult to understand how such an ill-conceived, poorly executed, and scientifically vacuous study could have ever been conducted, much less see the light of day in a respected journal like BMC Public Health. The journal peer review process, which we count on to at least identify utter junk science, was a colossal failure from start to finish, as documented here. However, this most basic failure was simply the last in a long line of peer review failures, aided by willful institutionalized ignorance and prejudice.

Beginning with the release of the article, we can work backwards, like crime scene investigators, to trace the various malicious acts back to the original fraud that formed the ideological genesis for this article. Popova and Ling note in the article that the National Cancer Institute (an organ of the National Institutes of Health), funded their study through a grant awarded to Pamela Ling. This means the very ideas behind the study, the background, the logic, the rationale for doing it, had to be blessed by an expert panel of scientific grant reviewers. These reviewers supposedly make their decisions on the basis of good scientific judgment. If you don’t make your case at this stage, you don’t get the money: Ling had to make a compelling argument for doing what she proposed, and she had to do it better than about 95% of the other applicants because there’s usually only enough money to dole out to the very highest scoring grant applications.

The National Institute of Health publishes information about every awarded grant on its Research Portfolio Online Reporting Tools (RePORT). One of the many pieces of information contained on this site is the applicant’s summary description of the proposed research. Ling’s is here.

Although we are not privy to the study details she proposed, this description clearly supports the thinking and methodology she used and subsequently published. The description is a checklist of the standard University of California San Francisco lies, evasions, and fallacies. Here are the lowlights:

 “Tobacco use is responsible for 35% of cancer deaths, and new smokeless tobacco marketing efforts threaten both to increase cancers caused by smokeless use . . .”

“new smokeless products are marketed as line extensions of major cigarette brands (Marlboro and Camel) to promote ‘dual use.’”

“These changes in smokeless tobacco marketing may blunt the effects of smokefree environments and the health benefits of smokers’ cutting down and quitting.”

“test new counter-marketing messages to block initiation of smokeless tobacco use among novices and the dual use of smokeless tobacco and cigarettes as an alternative to smoking cessation.”

“Findings will be relevant to guide development of policies on smokeless marketing and advertising.” 

That this grant was reviewed and given a top score tells us something about the review committee charged with evaluating its scientific rigor. I have sat on several grant review committees, and each grant has three (sometimes two) primary reviewers (just as journal articles do). With as many as 60-80 grants to review, each member is a primary reviewer on 3-4 grants and usually defers to the primary reviewers of the other grants when submitting a vote on quality score. In this way, the grant review system is embarrassingly similar to the journal review system. The only major difference is that with grant review there are 20 or so potential reviewers, and they all are supposed to be highly qualified to sit on the committee. Most have to have been awarded a grant themselves, be highly published and fairly well known in the field related to the RFA, and have the demonstrated expertise to evaluate the grants they are assigned. However, this academic record is obviously no guarantee committee members are blessed with deep critical thinking skills or untainted by strong political bias.

Clearly, the primary reviewers for Ling’s grant were fellow travelers able to ignore (or, through confirmatory bias, simply not see) the clear evidence that smokeless tobacco use has trivial cancer risk, if any, a well researched scientific finding that makes the entire premise of the grant specious. None of the other 15-20 committee members (who have access to all grants under review, and usually have read the summary descriptions) apparently had any serious objections either. Any one of them could have raised the point of smokeless tobacco’s trivial risks and demanded a debate on the topic, a discussion that could have significantly lowered Ling’s score. So Ling’s grant sailed to the top of the score list in much the same way it sailed onto the pages of BMC Public Health.

How does a grant review committee so ignorant of smokeless tobacco pass muster on Ling’s grant? For this clue we have to dig a little further. In the “details” section of Ling’s RePORT page we discover she had submitted her grant under a specific Request For Applications (RFA). The NIH frequently publicizes RFAs when it wants some narrowly focused research applications to address a particular topic area–in this case, an RFA titled, “Measures and Determinants of Smokeless Tobacco Use, Prevention, and Cessation.” The RFA is a special invitation to submit a grant tailored to it. Full text here:

Most of the RFA contains instructions on how to apply, but the most important piece of information is listed in the executive summary under “purpose.” This text describes what the issuers of the RFA want in an application and helps explain the rubber stamp of the review committee.

“The overall goal is to develop an evidence base to inform smokeless tobacco control efforts, and to develop effective ways to limit the spread and promote cessation of smokeless tobacco use.”

There you have it. From the very beginning the federal government, in the form of the National Cancer Institute, deliberately solicited applications for the express purpose of figuring out how to “limit the spread” of smokeless tobacco, especially for smokers who might fall into the fictitious trap of dual use in an attempt to switch. They formed a review committee that would stick to the flawed central premise of the RFA. The premise was right in Popova and Ling’s wheelhouse, and they naturally proposed an experiment going further than merely misleading smokeless tobacco users with false labeling, but adding some graphic and disturbing images to boot. You can fault Popova and Ling’s ignorance of good scientific practice, and their shameless attempt to use their wreck of a study in a shabby attempt to influence FDA policy. But you can’t accuse them of failing to deliver exactly what the government wanted.


15 responses to “The failures of peer review do not begin with the journal – more on the Popova-Ling fiasco

  1. Outstandingly funny, right along the lines of the paper (draft) I sent CVP today. Cheers!

  2. What do you want to bet that grant group just blindly takes the description from the grantor’s end assuming it’s already been reviewed elsewhere?

    • I don’t think anyone assumed it was good science and had been through some previous review. Grant applications are like trade secrets. You don’t share your ideas outside your closest colleagues. The more likely assumption that influenced the grant review committee is, as a member of the $13 million per year Glantz grant empire, Ling was a reliable tobacco control soldier. They hardly cared if her science was any good, only that her results could be used to help support the policy goals.

  3. Pingback: The failures of peer review do not begin with t...

  4. We need to find out how much this farce cost us taxpayers. Where can I get that information?

  5. Brian Carter

    Since 2009 UCSF has received $2,612,538 for six Ling “projects” No telling when this particular study was bought and paid for, but it was likely in 2013 when UCSF got $479,752. To be accurate, in this context a single “project” probably involves more than on study. The infamous one with Popova doesn’t strike me as being that expensive or time consuming to do. But it likely cost the tax payers at least one or two hundred grand.

  6. The abstract for another 2013-14 Popova paper : “Scaring the snus out of smokers: testing effects of fear, threat, and efficacy on smokers’ acceptance of novel smokeless tobacco products.” Is pretty clear about intention, no?
    This is gobsmaking :
    “desired outcomes” = “Less favorable attitudes toward snus and lower behavioral intentions to try snus in the future” .

    • Indeed. This paper is based on the same prohibitionist big lie that runs through all Popova and Ling’s work: all tobacco products are equally dangerous to health and smokers have no real options but to quit.

      What’s interesting here is that Popova is the sole author of this paper despite the research behind it clearly being funded by Ling’s grant listed above. Seems impossible that Ling had so little input she didn’t warrant at least second authorship. At the time this paper was submitted Popova was a post doctoral fellow at UCSF. No way Popova cooked up this idea, executed the study, and wrote up the paper all on her own. Ling obviously took one for the team by declining authorship she deserved to boost the stature of Popova. (UCSF probably has one of those idiotic policies that require someone have a sole authored paper to be considered for promotion.) It seems to have worked. Soon after, Popova was awarded a career development grant (K99CA187460) to help establish her independence as a tobacco control researcher.

      Taking a look at the RFA for Popova’s K99 grant we find this under “purpose.”

      “Research projects must address the research priorities related to the regulatory authority of the Food and Drug Administration (FDA) Center for Tobacco Products (CTP) as mandated by the Family Smoking Prevention and Tobacco Control Act (FSPTCA), Public Law 111-31.”

      Is it any wonder why Popova (and Ling) think their work has the utmost importance for regulators? Their sole purpose is to cook up the research results needed to support existing FDA policy as mandated by the FSPTCA.

  7. Pingback: What is peer review really? (part 1) | Anti-THR Lies and related topics

  8. Pingback: Real implications of the RSPH “sting” of ecig vendors | Anti-THR Lies and related topics

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s