Category Archives: truths

As many of you are aware, there was a recent major update to the old Cochrane Review of smoking cessation intervention studies (trials) that gave some or all participants e-cigarettes. This report is an unfortunate turn of events. I foresee yet another highly publicized vaping “success” statistic that so hugely underestimates the benefits of vaping that it is really a perfect anti-vaping talking point.

For those not familiar, Cochrane reviews are complicated-seeming simplistic analyses where a bunch of study results are averaged together, using a technique that is typically called “meta-analysis” though is properly described as “synthetic meta-analysis” (as in, synthesizing the results; it is not the only kind of meta-analysis). For those not familiar with that methodology, it is basically junk science if a set of fairly strong conditions is not met, conditions which are far from being met in the present case.

For more on that general observation, see this previous post. I am not going to go into that level of detail again here, but I will summarize. First, just because you can declare a bunch of numbers to be part of the same category and average them together doesn’t mean it makes any sense to do so. A bunch of studies with different interventions, different populations, and other different methods cannot be treated as if they were just one big study of a single phenomenon, even if they can all be described with the same imprecise phrase, like “studies of whether vaping helped people quit smoking.” The analogy I thought of while working on this was asking “what is the average mass of house pets?” Yes, “pets” is a category you can create, and average mass is something you can calculate. But why would you want to know that average? It is a meaningless amalgamation of several clearly different collections of observations. Why would you want to know the average smoking abstinence rate, at a given future moment, of people who were handed some e-cigarettes, with some degree of flexibility in their choice, with some level of information and assistance, for some people, at some place and time over the last ten years. Yes, you can calculate that number, but why would you?

Well, you can calculate it in theory, but in reality you are stuck doing something that is weak proxy for it. The Cochranoids only pretend to be calculating that number because, of course, measures of all those different combinations of “some” do not exist. Instead what they have is whatever nonsystematic combination of “some”s that someone decided to study and write down in a journal article. It is like trying to assess the average mass of pets by looking at the records of one veterinary practice. Do they specialize in dogs or cats? Whichever types of animals they happen to see is going to be what you measure, not the population average. What’s worse, there is no attempt in Cochrane or the typical synthetic meta-analysis to figure out a population representative weighting (not that you could even do it in this case, but they never even try). By this I mean that you could bring in an estimate of the relative number of dogs and cats in a population and use that to weight (no pun intended) the average of the data you have for dog and cat averages to get a reasonable estimate for the average of the set of all {dogs, cats}. But no, the Cochrane methodology just weights the average by however many observations happened to be in the studies (analogy: averaging cats and dogs based on how many were in the vet practice’s database, even if they see ten dogs for every one cat).

As I noted, this correction does not work for smoking cessation studies (since they do not represent any real-world practice at all, so there is no real-world weighting to use), but it is still a problem. If the collection of studies included one huge study that used a particularly ineffective vaping intervention, it would drag the average way down. If that same study instead had a low sample size, the estimated average would go up. Just think about it. Can a method that has this property possibly be considered valid science? Consider the analogy again: If the vet practice also sees twenty horses, the average mass shoots up. If it sees only one, the average is pulled up, but not that much.

But even worse, the most common (by head count) pets never visit a vet. The modal pet category, in terms of individuals, is caged critters like fish, rodents, and the occasional lizards and hermit crabs. So the vet practice selection methodology is not representative of “pets”, the originally defined category. The analogy is that the Cochrane paper purports to be looking at the effects of vaping-based interventions on smokers, but really it is only looking at the effect of (a few particular) interventions on people who volunteer for smoking cessation trials. Yes, you can redefine the categories to be “pets who see vets” or “people who volunteer for smoking cessation trials”, but altering your scientific question to better fit your data is another pretty good sign that you are doing junk science. Although that is far better than pretending you are pursuing the original question, while actually analyzing data in a way that could only answer the redefined question, which is what usually happens (including in this case).

And then there is the related problem that clinical interventions are not how most smokers are introduced to vaping. So this was never about measuring the effect of vaping on smoking cessation, but the effect of being told to try vaping in a clinical setting on smoking cessation. Those are very different concepts, but the results are interpreted as if they are the former when they are obviously the latter. If you wanted to assess how much vaping reduces smoking in the actual real world, rather than in barely-existent clinical interventions, you would use an entirely different body of evidence. These trials do approximately nothing to help answer that question.

It turns out that a systematic review of vaping-based smoking cessation trials could legitimately help answer some interesting and useful questions. It is just that this paper does not do that. Most notably: What characteristics of an intervention seems to cause the highest rate of smoking abstinence — which e-cigarettes, what advice and support, which types of people, and whatever else the methods reporting from the studies lets you figure out? With that information, you could design better vaping-based clinical interventions (which are not unheard of, though the are too rare to really affect the question of how much vaping reduces smoking). You could also add useful assessments like what future trials should do for best practices (based on current knowledge) and what characteristics they should test to see what seems to work better.

This potential value of the review only serves to reinforce the fundamental failing of what was done. Why, oh why, would you want to take the success rates from the better-practice interventions and average them together with the rates from other interventions? And weight the result based on how many people happen to have been studied using the various methods? And then report that number as if it meant something? My mind just boggles that anyone ever would think this is a useful question to ask.

So I trust we have established that the number they reported is meaningless junk, even for what it purports to be. By the way, that number is a four percentage point increase in successful medium-term smoking abstinence, compared to null or near-null interventions. I buried this because mentioning a scalar in a headline or early in a piece tends to cause the reader to fixate on that number and consider it the main takeaway. It is not. It is meaningless. I urge you to never repeat it.

The reason I mention it at all is to comment on how low it is. If this were really the measure of the smoking cessation benefits of vaping, it would not make a very good case for vaping. Yes, you can spin it as “the prestigious definitive Cochrane Review [cough cough cough] finds vaping is better for smoking cessation than ‘officially recommended’ methods like NRT.” But the magnitude of “better” is so low that it is easy for someone to convincingly make the case that it is not good enough to justify the scourge of teen vaping, or whatever. Or that it is so low that we can just develop some improvement to the ‘officially recommended’ methods that would be even better.

So far, I have only hinted at the main reasons why that number is not a valid measure of how much smoking cessation is caused by vaping. Even if that statistic were a valid measure of what it could measure — “what happens if you use clinical methods to encourage vaping for smokers who are seeking aid to quit” — and not just some bizarrely weighted average of a random collection of often terrible ways of going about that, it would still be a huge underestimate.

There are three main pathways via which vaping causes less smoking: 1) For some people who are actively attempting smoking cessation, it increases their chance of success. 2) For some people who would not otherwise be attempting cessation, it inspires them to try or just do it. 3) It displaces some smoking initiation, replacing it with vaping instead. I am highly cognizant of the failure to understand this distinction because a colleague and I recently finished a review of those “population model” papers about the effects of vaping on future smoking (which hopefully will see the light of day soon). We discovered that almost every one of those papers just ignored 2) and only looked at 1) as a measure of how much cessation would increase. Some of them did this rather overtly (though they never admitted — or apparently even realized — they were accidentally making this assumption), while for others it was implicit. (Some, but not all, also considered 3) separately, but that is not immediately relevant.)

People described by 2) include “accidental quitters” as well as people who decide vaping is tempting and decide to try to quit smoking (switch) because of that. It seems safe to make the educated guess (for that is all we can really do with the data we have) that this has greater total effect than 1). In addition to creating new cessation attempts (which those “population model” papers mostly assume do not happen), vaping gets “full credit” for any resulting cessation, not just credit for the increase in the success rate (another error in the population model papers). That is, even if someone would have quit smoking even without vaping had they given it a try — and thus the the fact that switching to vaping is a particularly effective way to quit did not even matter — that case of cessation was still caused by vaping.

Like those problematic population models, the Cochrane approach only looks at 1). Everyone studied is doing something to attempt to quit smoking, or at least is going through the motions and signed up for some guided quitting attempt. So half or more of the cessation effect of vaping is being assumed away.

And it gets worse still, for both the whole meta-analysis method and this particular exercise. Behavior is not biology. The Cochrane method is sometimes valid if what is being studied is a biological effect (see the above link for more details of other conditions that must be met), but is hopeless for assessing personal behavior. Why? Because people know themselves and make choices, of course. So in a population (place, time) where vaping is reasonably well known, a smoker who finds it an appealing option is likely to try it, and if she was correct that it was indeed just what she needed, then she is going to quit smoking. She is a category 2) success story, or perhaps category 1) if she was already dedicated to quitting. And then what happens? She doesn’t volunteer for a smoking cessation trial!

That is, the people who are accurately self-aware that they are a particularly good candidate for quitting via vaping just do it, and so do not contribute to the study-measured success rate. It is like the fish in the “average mass of pets” analogy — they never show up to the vet to get weighed into the average. This cuts both ways, of course: Anyone who is self-aware that they just need some nicotine gum also quits and is not in the study to give due credit to nicotine gum. The difference, of course, is that basically no one accurately thinks that.

We also know that people who pursue formal smoking cessation interventions are more likely to be “unable” to quit on their own, which could bias the results either direction. I.e., it could be that people who just decide to quit are more likely to be helped by vaping, as compared to someone seeking aid, because their baseline success rate is higher and vaping multiplies that. Or vaping might matter less for them because they would be successful even without vaping. But it is almost certainly not exactly the same.

In fairness to the Cochranoids, it was not their task to review category 2) quitters, or selection bias, or other evidence. It was not their job to provide useful information. Their job is to just mechanically average whatever numbers someone hands them. However, that is being rather too fair to them, since they pretended they were measuring something useful. They conclude “There is moderate‐certainty evidence that [vapes] with nicotine increase quit rates….” This claims implies that they are measuring how much quitting is caused by vaping, full stop, not merely how much more likely clinical study volunteers are to be abstinent if they are in the vaping trial arm.

To summarize: If clinical assignment to try vaping really only increases successful smoking cessation (or, more precisely, medium-term abstinence) by four percentage points, it is really not very impressive. But we are pretty sure that is not the case because it is based on a population where many of those most likely to switch have already exited, and it is based on randomly averaging together best practices and poorly designed interventions. Moreover, even if it were right this would only be one of the many pathways from vaping to smoking abstinence, and one of the least important, so who cares that it is low?

On the bright side, most of the headlines and pull quotes I have seen about this fake science say something like “vaping shown to be better for quitting than NRT” or “new study shows vaping helps people quit smoking.” While these stories seem to be all written by someone without a clue about the Cochrane Report, this is a case where three clueless wrongs make a right: At least those vague unquantified messages are correct. (The third wrong is that people have been indoctrinated into thinking that NRT has measurable benefits, so they interpret “better than NRT” as meaning “good” when it really only means “not quite zero”.)

The problem is that after a spate of “this just in!” headlines this month, which will affect almost no one’s beliefs, we can look forward to a few years of this paper being cited as evidence that vaping has a trivial effect on reducing smoking. The four percent number will be successfully portrayed as definitive and the entirety of the effect of vaping on smoking prevalence. And everyone who is currently suggesting that it is not total junk, because they like the headlines of the day, is helping make that happen.