by Carl V Phillips
Yesterday, I explained what was fundamentally wrong with Stanton Glantz’s new “meta-analysis” paper, beginning with parody and ending with a lament about the approach of his critics who are within public health. Glantz posted a rebuttal to the press release from those critics on his blog, which does a really nice job of helping me make some of my points. I look forward to his attempt to rebut my critique (hahaha — like he would dare), which would undoubtedly help me even more.
Glantz pretty well sums it up with:
The methods and interpretations in our paper follow standard statistical methods for analyzing and interpreting data.
He offers this as a non sequitur response to the quote from Peter Hajek that I highlighted yesterday. Hajek’s analysis was the best thing in that press release, and it involved actual scientific reasoning, so it is little wonder that Glantz did not try to really respond to it.
But consider the ramifications of what Glantz said without pretending it is a response to Hajek. He is not wrong. Misinterpreting what studies actually show and doing inappropriate synthetic meta-analysis are indeed fairly standard practice in public health research. Notice that he did not say they are good or valid methods. This handily makes my point from yesterday: This battle against Glantz cannot be won by playing by the (awful) rules of public health academic discourse, because Glantz is playing by those rules. And winning — see his list he added to that post, of media outlets that reported his lies.
Glantz begins this reply to those critics with:
The most remarkable thing about these criticisms is their generality rather than engaging the specific substance of the paper.
This is simply false, fodder for Glantz readers who — as evidenced by the fact that they are Glantz readers — will believe anything they read. Half of the points in the critiques drill into the substance of the paper. The reason he can get away with this, however, is that the other half indeed did not engage substantively. As I noted, effective critiques need to cut to the heart of what is wrong. But of the three easily-identifiable and utterly fatal problems that I identified (and I have read the other critiques and stand by my assessment that it is really about those three), only one is effectively included in the press release. That is the second one I covered, about selection bias, and it was presented deftly only by Hajek. But this is the only one of the three where the details matter and can be debated, and thus the hardest to argue (though it is still indisputably a fatal flaw). The other two are 100% slam-dunks, but they require that public health people leave their comfortable zone [sic] to make them.
Robert West offered a weak version of my third point, that the results contradict reality:
“If use of e-cigarettes caused fewer smokers to quit, the quit rate in England would have decreased as use of e-cigarettes has increased. Data from The Smoking Toolkit Study – a large ongoing population survey of smoking in England – shows, if anything, the opposite.
Cigarette use rates have been declining since the 1970s in the UK, prior to the introduction of e-cigarettes. Simply comparing rates of cigarette and e-cigarette use does not allow for evaluation of the association between e-cigarette use and quitting (www.smokinginengland.info).
Credit where due (unlike many of those in this story, I do not refuse to recognize the validity of an argument based on who wrote it): Glantz is right. I could not have said it better. Well actually I could, and did: I noted that West’s claim is only valid if nothing else is affecting the rate of smoking cessation, which is really the point Glantz is trying to make. But he basically got it.
You will recall that I ran some numbers about the real world that show that Glantz’s conclusion cannot be right, that is utterly implausible. That is the real reason why reality contradicts the study. Playing the “I am citing a study that contradicts your study” game is not a winning strategy in general, and even Glantz can rip through it when the study does not, in fact, contradict him. Actually writing out a simple quantitative model (mine took me only a few minutes) is a great way to cut through absurdities. But if we start doing that, it becomes a threat to business-as-usual in public health research.
Circling back to Glantz’s introduction:
Despite these limitations, it is possible to draw important conclusions based on the available studies. In particular, we conducted a formal sensitivity analysis of the factors (and other factors) that the experts raise, which appears in Table 2 of the paper.
This is a type of claim that is hard to pre-rubut. That is, even though it is easy to show that it is wrong, it is hard to anticipate every absurd defense and address it specifically until after you see it. But in this case, I did address that claim, indirectly, in a couple of ways (the press release did not). In my parody version of the abstract, I pointed out that his sensitivity analysis consisted of “weak statistical stuff that we are not going to really explain or justify”. It really does nothing. Later I noted the value of comparative meta-analysis, a method for trying to explain why different studies produced different measures of supposedly the same phenomenon. There is no reason that analysis needs to be done with fancy statistics — it can be done via simple inspection.
Glantz’s Table 2 is an obvious fail because it does not identify the features of studies that produced extreme results. In fairness to him, it does actually address several of the supposed problems that the press release critics brought up, like whether the subjects were actively interested in quitting smoking. What it does not do is identify the studies that have fatal selection bias due to the “immortal person-time” problem I explained. If your “sensitivity analysis” does not actually figure out what the results were sensitive to (when it was clear that they were sensitive to something, given that they were widely scattered) it was a failure. But here we are back at the fundamental problems of public health research: Mock pro forma sensitivity analyses, that just give the illusion of robustness while avoiding the real scientific question, are the norm.
In response to most of Linda Bauld’s longish comment in the press release (19 sentences if you replace semicolons with periods), Glantz writes:
The studies of more intensive users of e-cigarettes and users of higher generation devices were specifically discussed in the Discussion as populations who might have increased rates of smoking cessation. It is important to emphasize that these people represent a small fraction of all e-cigarette users.
Setting aside the accuracy of Glantz’s statement, the real lesson here is don’t bring an essay to a soundbite fight. Bauld makes several good points. My quick count says 11 of her sentences were basically right, though not truly cutting, criticisms of the paper (6 were tangential; 2 were wrong). Glantz uses the oldest trick in the flame wars book, cherrypicking a single point to reply to, and implying it is a complete response. I made clear that I do not think highly of the soundbite fight game, but if you are going to play it, bring a soundbite. If you want to write a more complete analysis as a rebuttal, by all means write that instead. Or better still, just cite mine. :-) But pick one.
The real problem there was that Bauld, coming from a card-carrying tobacco controller perspective, suggests that the utter junk science approach is basically fine, and the problems lie in the details. For the one detail that Glantz cherrypicked to respond to, he is basically right: It is theoretically possible that some optimal users of better devices are helped to quit smoking, but most users of most devices are harmed. It turns out to be false, but it is not necessarily so. If you do not deny the validity of Glantz’s whole approach, you still lose on that point. Bauld did allude to my point 2, though she garbles it and misidentifies the problem as confounding rather than selection bias.
She also alluded to my point 1, noting the huge heterogeneity of the studies, but fails to point out that there is no possible way studies in this space could ever be homogenous enough to justify synthetic meta-analysis. Perhaps she does not even realize that is true. Or just wants to reserve the option of employing the same junk methodology when its results suit her personal politics. Indeed, earlier today a colleague messaged me with a rumor that this group of critics now wants to do their own meta-analysis of the “good” studies in the mix as a response to Glantz. It was punctuated with a headbang emoji.
I responded with two headbangs. Just what we need: an implicit endorsement of the junk methodology. I will go ahead and start writing Glantz’s post about it for him:
Several e-cigarette advocates who criticized my original study the moment the embargo lifted have now conducted a similar study. They employ the same methodology, which is the best way to answer this question, but they have purposely excluded informative studies. They have chosen to include only the studies most favorable to their political views, to bias the results in their favor. Proper research methods require including all studies. Our meta-analysis remains the most complete summary of all the available information on the topic, and thus its results are most reliable.
Ok, I write much better than Glantz, but he can dumb it down and add a random number of extra spaces between sentences.
Glantz responds to two specific points from Bauld with this (about popularity):
The fact that e-cigarettes are popular as quit aids does not make them effective. The overall conclusion from all the available data suggests that e-cigarette use is depressing quitting.
and this (about future studies that are planned):
We look forward to reading these studies when they are published. We hope that such studies are not simply limited to e-cigarettes used under “ideal” conditions, but rather are studied as actually promoted and used in the real world.
And, once again, Glantz is right. In keeping with my above observation, it is possible that e-cigarettes overall cause net harm to smoking cessation even though an identifiable subset of use is beneficial. Again, it is not true. But the best response to that is “this cannot possible be true, because….” (an example being the back-of-the envelope quantification I did). Bauld never actually does that. Glantz is not honest and has no apparent scientific skills, but he is plenty clever — he can see through a tangential observation that does not actually contradict him.
In response to another comment from the press release, Glantz writes:
Our paper is based on all the available data as of the time we wrote the paper.
I am actually at a loss to figure out why he thinks this is a response to the comment he quotes. But skip that and just look at the assertion. It is obviously not true. The paper considers only the tiny subset of the available data that takes a particular form and was published in a particular way. It is obvious to anyone who is aware of the real body of knowledge — much of which is far more relevant to assessing real-world impacts than anything Glantz considered — that Glantz’s conclusion is wrong. But once again, his critics gave him this one. They quibbled about Glantz’s game on its own terms, trying to argue about which of those studies represents reality, rather than simply pointing out that the conclusion must be wrong because it contradicts actual reality which we know a lot about. West does try to say that, but he still ties it back to a particular “standard” study result. None of them were willing to admit that most of the relevant information comes from outside the realm of their and their colleagues’ standard studies.
Finally, Glantz fails to respond to Ann McNeill’s contribution to the press release. I like what she wrote. She start off with, “This review is not scientific.”, which is a nice (and accurate) soundbite. But she does not really build on it, and so it is easy for Glantz to ignore it, and he does. The rest of her comment is about Glantz’s misinterpretation of two particular studies that she coauthored. He responds:
In the Adkison study, the measurement of cigarette use was longitudinal (from one wave to another), while the assessment of e-cigarette use was at follow-up. For the longitudinal studies, the paper was clear about the time of e-cigarette assessment, i.e. whether it was at baseline or follow-up (Table 1 and the discussion of individual studies in the appendix). The results in the Hitchman study, like any such study, are based on the behavior reported during the time of the study. In any event, the sensitivity analysis did not show an effect of the timing of e-cigarette measurement on results.
For the first of these, I actually am not really sure what McNeill’s point was, so it is hard to identify fault with the response. The second was one of the studies that had huge “immortal person-time” selection bias when interpreted the way that Glantz did. (As I noted yesterday, this is no fault of the original authors. Their study was what it was and the bias is introduced by trying to interpret the results as a measure of something the authors did not purport to be able to measure. McNeill notes they had already pointed this out to Glantz.)
So though McNeill does not cut to the heart of what is wrong with Glantz’s paper, she employs the very nice tactic of pointing out that one particular element of it was clearly wrong, and Glantz had been told that, so the reader ought to infer that the whole project is suspect. Unfortunately, she does not finish the thought, and effectively explain why you cannot use the data from Hitchmann (i.e., the selection bias). That created the opening for Glantz to write a sentence that ignored the real problem and imply it was a rebuttal. Of course, if you scroll up and read what Hajek wrote, it explains the why. But it is easy to pretend that was not there when responding to a scattershot collection of criticisms.
Overall, my assessment of the intra-public-health discussion here is Glantz won handily. Anyone who is somewhat expert or actively truth-seeking could find their way to my analysis, or Clive’s/Legacy’s, and really understand the fundamental criticisms. But trying to put myself in the minds of the press release’s actual targets (reporters and random commentators — largely people who simply cannot understand analysis that is more sophisticated than what appears in public health journals), I can’t see that many points were scored.
There was definitely some value. Reporters reading the critical press release in addition to Glantz’s, rather than just the latter, were at least somewhat immunized against just reporting the lies. As I have chronicled before, CASAA has succeeded a few times in discouraging widespread coverage in the US media of some junk science that got lots of coverage in the UK. But the critique was just too unfocused to be good counter-soundbite material and fell far short of being a solid scientific critique. (Note to UK critics of anti-ecig junk science: You might want to bring us in next time. Some of us are quite good at soundbites and we can extend your reach. I will take the liberty of volunteering our services. Oh, also you need to send it out as a press release in advance of the embargo.)
However, if a scientifically unsophisticated reader (i.e., a typical public health person, medic, or health reporter) read Glantz’s original press release or abstract, and then the criticisms, and then Glantz’s response, I would guess he would feel Glantz won the debate. He did the fancy statistical thingy, after all, and it was all peer-reviewed and published and all that. None of the criticisms clearly denied the validity of the overall methods or conclusion. And then Glantz made the criticisms appear to be all debatable little technical issues that he properly considered.
I am not saying that battles on these terms can never be won. But I seriously doubt they can ever be won by people who are trying to preserve the reputation of the their public health “brand”, pretending it is not the case that health journals regularly publish utter junk and that many common methods and behaviors in public health research produce nonsense.
[Update: I should have finished this with my tribute to Hans Snape, with a minor rewrite of a quote from one of his lesser roles. In response to “Why don’t I like you?”: “Because you think I’m an asshole, and I’m not really, I’m just a scientist”]