Category Archives: truths

discussions of how to best present the truth

The unfortunate case of the Cochrane Review of vaping-based smoking cessation trials

As many of you are aware, there was a recent major update to the old Cochrane Review of smoking cessation intervention studies (trials) that gave some or all participants e-cigarettes. This report is an unfortunate turn of events. I foresee yet another highly publicized vaping “success” statistic that so hugely underestimates the benefits of vaping that it is really a perfect anti-vaping talking point.

For those not familiar, Cochrane reviews are complicated-seeming simplistic analyses where a bunch of study results are averaged together, using a technique that is typically called “meta-analysis” though is properly described as “synthetic meta-analysis” (as in, synthesizing the results; it is not the only kind of meta-analysis). For those not familiar with that methodology, it is basically junk science if a set of fairly strong conditions is not met, conditions which are far from being met in the present case.

For more on that general observation, see this previous post. I am not going to go into that level of detail again here, but I will summarize. First, just because you can declare a bunch of numbers to be part of the same category and average them together doesn’t mean it makes any sense to do so. A bunch of studies with different interventions, different populations, and other different methods cannot be treated as if they were just one big study of a single phenomenon, even if they can all be described with the same imprecise phrase, like “studies of whether vaping helped people quit smoking.” The analogy I thought of while working on this was asking “what is the average mass of house pets?” Yes, “pets” is a category you can create, and average mass is something you can calculate. But why would you want to know that average? It is a meaningless amalgamation of several clearly different collections of observations. Why would you want to know the average smoking abstinence rate, at a given future moment, of people who were handed some e-cigarettes, with some degree of flexibility in their choice, with some level of information and assistance, for some people, at some place and time over the last ten years. Yes, you can calculate that number, but why would you?

Well, you can calculate it in theory, but in reality you are stuck doing something that is weak proxy for it. The Cochranoids only pretend to be calculating that number because, of course, measures of all those different combinations of “some” do not exist. Instead what they have is whatever nonsystematic combination of “some”s that someone decided to study and write down in a journal article. It is like trying to assess the average mass of pets by looking at the records of one veterinary practice. Do they specialize in dogs or cats? Whichever types of animals they happen to see is going to be what you measure, not the population average. What’s worse, there is no attempt in Cochrane or the typical synthetic meta-analysis to figure out a population representative weighting (not that you could even do it in this case, but they never even try). By this I mean that you could bring in an estimate of the relative number of dogs and cats in a population and use that to weight (no pun intended) the average of the data you have for dog and cat averages to get a reasonable estimate for the average of the set of all {dogs, cats}. But no, the Cochrane methodology just weights the average by however many observations happened to be in the studies (analogy: averaging cats and dogs based on how many were in the vet practice’s database, even if they see ten dogs for every one cat).

As I noted, this correction does not work for smoking cessation studies (since they do not represent any real-world practice at all, so there is no real-world weighting to use), but it is still a problem. If the collection of studies included one huge study that used a particularly ineffective vaping intervention, it would drag the average way down. If that same study instead had a low sample size, the estimated average would go up. Just think about it. Can a method that has this property possibly be considered valid science? Consider the analogy again: If the vet practice also sees twenty horses, the average mass shoots up. If it sees only one, the average is pulled up, but not that much.

But even worse, the most common (by head count) pets never visit a vet. The modal pet category, in terms of individuals, is caged critters like fish, rodents, and the occasional lizards and hermit crabs. So the vet practice selection methodology is not representative of “pets”, the originally defined category. The analogy is that the Cochrane paper purports to be looking at the effects of vaping-based interventions on smokers, but really it is only looking at the effect of (a few particular) interventions on people who volunteer for smoking cessation trials. Yes, you can redefine the categories to be “pets who see vets” or “people who volunteer for smoking cessation trials”, but altering your scientific question to better fit your data is another pretty good sign that you are doing junk science. Although that is far better than pretending you are pursuing the original question, while actually analyzing data in a way that could only answer the redefined question, which is what usually happens (including in this case).

And then there is the related problem that clinical interventions are not how most smokers are introduced to vaping. So this was never about measuring the effect of vaping on smoking cessation, but the effect of being told to try vaping in a clinical setting on smoking cessation. Those are very different concepts, but the results are interpreted as if they are the former when they are obviously the latter. If you wanted to assess how much vaping reduces smoking in the actual real world, rather than in barely-existent clinical interventions, you would use an entirely different body of evidence. These trials do approximately nothing to help answer that question.

It turns out that a systematic review of vaping-based smoking cessation trials could legitimately help answer some interesting and useful questions. It is just that this paper does not do that. Most notably: What characteristics of an intervention seems to cause the highest rate of smoking abstinence — which e-cigarettes, what advice and support, which types of people, and whatever else the methods reporting from the studies lets you figure out? With that information, you could design better vaping-based clinical interventions (which are not unheard of, though the are too rare to really affect the question of how much vaping reduces smoking). You could also add useful assessments like what future trials should do for best practices (based on current knowledge) and what characteristics they should test to see what seems to work better.

This potential value of the review only serves to reinforce the fundamental failing of what was done. Why, oh why, would you want to take the success rates from the better-practice interventions and average them together with the rates from other interventions? And weight the result based on how many people happen to have been studied using the various methods? And then report that number as if it meant something? My mind just boggles that anyone ever would think this is a useful question to ask.

So I trust we have established that the number they reported is meaningless junk, even for what it purports to be. By the way, that number is a four percentage point increase in successful medium-term smoking abstinence, compared to null or near-null interventions. I buried this because mentioning a scalar in a headline or early in a piece tends to cause the reader to fixate on that number and consider it the main takeaway. It is not. It is meaningless. I urge you to never repeat it.

The reason I mention it at all is to comment on how low it is. If this were really the measure of the smoking cessation benefits of vaping, it would not make a very good case for vaping. Yes, you can spin it as “the prestigious definitive Cochrane Review [cough cough cough] finds vaping is better for smoking cessation than ‘officially recommended’ methods like NRT.” But the magnitude of “better” is so low that it is easy for someone to convincingly make the case that it is not good enough to justify the scourge of teen vaping, or whatever. Or that it is so low that we can just develop some improvement to the ‘officially recommended’ methods that would be even better.

So far, I have only hinted at the main reasons why that number is not a valid measure of how much smoking cessation is caused by vaping. Even if that statistic were a valid measure of what it could measure — “what happens if you use clinical methods to encourage vaping for smokers who are seeking aid to quit” — and not just some bizarrely weighted average of a random collection of often terrible ways of going about that, it would still be a huge underestimate.

There are three main pathways via which vaping causes less smoking: 1) For some people who are actively attempting smoking cessation, it increases their chance of success. 2) For some people who would not otherwise be attempting cessation, it inspires them to try or just do it. 3) It displaces some smoking initiation, replacing it with vaping instead. I am highly cognizant of the failure to understand this distinction because a colleague and I recently finished a review of those “population model” papers about the effects of vaping on future smoking (which hopefully will see the light of day soon). We discovered that almost every one of those papers just ignored 2) and only looked at 1) as a measure of how much cessation would increase. Some of them did this rather overtly (though they never admitted — or apparently even realized — they were accidentally making this assumption), while for others it was implicit. (Some, but not all, also considered 3) separately, but that is not immediately relevant.)

People described by 2) include “accidental quitters” as well as people who decide vaping is tempting and decide to try to quit smoking (switch) because of that. It seems safe to make the educated guess (for that is all we can really do with the data we have) that this has greater total effect than 1). In addition to creating new cessation attempts (which those “population model” papers mostly assume do not happen), vaping gets “full credit” for any resulting cessation, not just credit for the increase in the success rate (another error in the population model papers). That is, even if someone would have quit smoking even without vaping had they given it a try — and thus the the fact that switching to vaping is a particularly effective way to quit did not even matter — that case of cessation was still caused by vaping.

Like those problematic population models, the Cochrane approach only looks at 1). Everyone studied is doing something to attempt to quit smoking, or at least is going through the motions and signed up for some guided quitting attempt. So half or more of the cessation effect of vaping is being assumed away.

And it gets worse still, for both the whole meta-analysis method and this particular exercise. Behavior is not biology. The Cochrane method is sometimes valid if what is being studied is a biological effect (see the above link for more details of other conditions that must be met), but is hopeless for assessing personal behavior. Why? Because people know themselves and make choices, of course. So in a population (place, time) where vaping is reasonably well known, a smoker who finds it an appealing option is likely to try it, and if she was correct that it was indeed just what she needed, then she is going to quit smoking. She is a category 2) success story, or perhaps category 1) if she was already dedicated to quitting. And then what happens? She doesn’t volunteer for a smoking cessation trial!

That is, the people who are accurately self-aware that they are a particularly good candidate for quitting via vaping just do it, and so do not contribute to the study-measured success rate. It is like the fish in the “average mass of pets” analogy — they never show up to the vet to get weighed into the average. This cuts both ways, of course: Anyone who is self-aware that they just need some nicotine gum also quits and is not in the study to give due credit to nicotine gum. The difference, of course, is that basically no one accurately thinks that.

We also know that people who pursue formal smoking cessation interventions are more likely to be “unable” to quit on their own, which could bias the results either direction. I.e., it could be that people who just decide to quit are more likely to be helped by vaping, as compared to someone seeking aid, because their baseline success rate is higher and vaping multiplies that. Or vaping might matter less for them because they would be successful even without vaping. But it is almost certainly not exactly the same.

In fairness to the Cochranoids, it was not their task to review category 2) quitters, or selection bias, or other evidence. It was not their job to provide useful information. Their job is to just mechanically average whatever numbers someone hands them. However, that is being rather too fair to them, since they pretended they were measuring something useful. They conclude “There is moderate‐certainty evidence that [vapes] with nicotine increase quit rates….” This claims implies that they are measuring how much quitting is caused by vaping, full stop, not merely how much more likely clinical study volunteers are to be abstinent if they are in the vaping trial arm.

To summarize: If clinical assignment to try vaping really only increases successful smoking cessation (or, more precisely, medium-term abstinence) by four percentage points, it is really not very impressive. But we are pretty sure that is not the case because it is based on a population where many of those most likely to switch have already exited, and it is based on randomly averaging together best practices and poorly designed interventions. Moreover, even if it were right this would only be one of the many pathways from vaping to smoking abstinence, and one of the least important, so who cares that it is low?

On the bright side, most of the headlines and pull quotes I have seen about this fake science say something like “vaping shown to be better for quitting than NRT” or “new study shows vaping helps people quit smoking.” While these stories seem to be all written by someone without a clue about the Cochrane Report, this is a case where three clueless wrongs make a right: At least those vague unquantified messages are correct. (The third wrong is that people have been indoctrinated into thinking that NRT has measurable benefits, so they interpret “better than NRT” as meaning “good” when it really only means “not quite zero”.)

The problem is that after a spate of “this just in!” headlines this month, which will affect almost no one’s beliefs, we can look forward to a few years of this paper being cited as evidence that vaping has a trivial effect on reducing smoking. The four percent number will be successfully portrayed as definitive and the entirety of the effect of vaping on smoking prevalence. And everyone who is currently suggesting that it is not total junk, because they like the headlines of the day, is helping make that happen.

Can smoking protect you against COVID-19?

by Carl V Phillips

Many of you will have already seen or heard about a paper by Farsalinos et al., in which they review some case series data from China and observe that for hospitalized COVID-19 patients, the recorded smoking prevalence is far lower than would be expected given the population prevalence. The US CDC also released data a couple of days ago that shows the same pattern. If the data is representative and accurate (but note that there are compelling reasons to question whether either of those is true), this strongly suggests that smoking is hugely protective against COVID-19 inflection and/or the resulting disease progressing to the point that hospitalization is required.

We are not talking at the level of “well I guess smokers get a bit of compensation this year for all the health costs of smoking.” This is at the level of “everyone should take up smoking for a few months until the pandemic abates.” The protective effect implied by the data is absolutely huge. Continue reading

New Glover-Phillips paper: “Potential effects of using non-combustible tobacco and nicotine products during pregnancy: a systematic review”

by Carl V Phillips

This new paper, by Marewa Glover and me, is just out in Harm Reduction Journal. In it, we review the available epidemiology evidence about the effects of nicotine-sans-smoke (NRT, snus, vape) on pregnancy outcomes. It was a bit of a challenge to get it published because we wrote the paper we needed to, rather than a “typical review”. As you might know, the journal publication process is rather …well, let’s just say conservative.

(I should note that we finalized the review before this new contribution to the literature by Igor Burstyn et. co., “Smoking and use of electronic cigarettes (vaping) in relation to preterm birth and small-for-gestational-age in a 2016 U.S. national sample” (note that this link bypasses the paywall, but does not seem to work in all browser configurations). Igor’s paper is higher quality than anything we reviewed.)

A typical review of epidemiology looks at the results that are reported in journal articles and then just naively believes them, suggesting that What We Know consists of a vague summary of whatever results the previous authors chose to publish. Or even worse — so much much worse — suggesting that a calculated average of those results is the best estimate. That is never a legitimate assessment of existing knowledge, and less so in our case. Continue reading

“Dependence” and the danger of adopting the language of your oppressors

by Carl V Phillips

Vaping, smoking, and other tobacco product use are routinely described as “addictive”. As I have pointed out repeatedly, this is a very misleading characterization. (You might recall my major essay – years in the making – on this topic from earlier this year. If you missed it and are reading this, you will definitely want to go read it.) The two sentence summary of the headline point is: All ‘official’ definitions of “addiction” hinge on the behavior being highly disruptive to someone’s functioning – work, social, etc. But tobacco product use has no such effects, at least not for more than a minuscule fraction of consumers.

So the fallback position, in the event that someone recognizes the problem with that word, is to say that tobacco product use produces dependence. But this is barely more accurate and is equally misleading. For those who use these products or advocate for their acceptance, to use of either of these words is to make the rhetorically and psychologically dangerous mistake of adopting the language of one’s oppressors. Continue reading

Essay on bans by Marewa Glover

by Carl V Phillips

I wanted to call attention to a new essay about bans by Marewa Glover, probably the most thoughtful and clear-reasoning person who still self-identifies as a tobacco controller. The piece is entitled “Do We Really Need Another Law? The cost to New Zealand of banning smoking in cars,” which describes its titular focus and the motivating policy proposal. But it is more a wide-ranging examination various implications of bans and tobacco control more generally. It is a long read at 16K words (which she calls a book, even though I would call it two long blog posts :-), but easy going and worth the time.

You can find it at her institute’s website, here. (Needless to say, by recommending it I am not endorsing every word of it. I have some disagreements.) Continue reading

Tobacco Wars collateral damage: feature, not bug

by Carl V Phillips

A single-observation post, inspired by the great consternation I am seeing this week about proposed FDA retail restrictions on vapes, ostensibly for the purpose of reducing vaping by minors. There are quite a few reasons this is a terrible policy (see coverage by Clive Bates here), but the theme of the typical criticism is that it will hurt legal “proper” vape consumers (primarily by denying them flavors they like or convenient purchase venues) more than it will “help” teenagers (by denying them something they want to do). The criticism is presented as if this supposedly odd perverse effect might persuade tobacco controllers to change the policy.

Here’s the thing: Hurting people who continue to use a tobacco product is considered a feature, not a bug. Despite the endless chatter about trivial policies, there is basically only one category of tobacco control policies that matter (by any measure), other than the bans that exist for some products in some places: the high punitive taxes on cigarettes and other products. These policies is lauded by many of the same people who condemn blunt-instrument anti-teen-vaping policies. Yet they are almost exactly the same from an ethical perspective. Continue reading

What is a lie?, revisited.

by Carl V Phillips

As regular readers know, I have written a fair bit about the nature of lies. I make a serious study of it as part of the mission of this blog and my larger approach to the politics of harm reduction and real public health. I do this with as much scientific rigor as is possible for such a question. Recently a confluence of events — the ongoing attempts of the press to deal with Trump’s claims, dealing with my ex’s lawyer, and most importantly the “vaping causes seizures” controversy — reminded me that I have not updated my thinking on this for a while. So here goes. Continue reading

New statistics about vape risk misperception (and a subtle extra-bad implication)

by Carl V Phillips

A new paper in JAMA Network Open by Jidong Huang et al. from Georgia State University provides some new statistics about just how effective the war on vaping is, in terms of the average American’s perceptions of risk. Despite working for one of FDA’s pet research shops, the authors make clear their opinion that it is bad that so many people think that vaping is as harmful as smoking or worse. Continue reading