by Carl V Phillips
I have spent a lot of my career pointing out how the choice of research methods and statistical models — in particular, the choice of which to report among the many that were tried — create bias in the epidemiology literature. It is easy to create a study that is designed to get a particular result, especially if the desired result is to fail to observe a phenomenon (in the days before e-cigarettes, I was baffled that the anti-ST people never ran an intended-to-fail intervention to “show” that THR did not work).
It seems that the American Legacy Foundation has taken this one step further. In a comment on the ECLAT study, which found that many smokers who were forced to try e-cigarettes for a while (as study participants) decided to switch to them, they basically described the designed-to-fail methodology they would have used and criticized the honest researchers for not using it. Mike Siegel summed it up (emphasis in original):
According to the press release: “The researchers reported that e-cigarettes decreased some smokers’ cigarette consumption and that 8.7% quit smoking 40 weeks after the intervention ended. Unfortunately, they also found that smokers quit rates were not statistically different whether given e-cigarettes with or without nicotine –thereby causing a placebo effect. … We cannot conclude from this study that e-cigarettes promote cessation. While the study showed that some smokers quit, it does not show that the product itself had any role in the behavior change. In fact, the results merely show that sucking on an empty cigarette holder (a placebo) would likely accomplish the same thing.”
This press release misses the whole point. And in doing so, it ends up misleading the public.
There is no true “placebo” effect involved with electronic cigarettes because the mimicking of smoking with the use of a cigarette-like device is the main point of the product. We do not want research to control for this effect. We want research to measure this effect.
Obviously Legacy is wrong about a cigarette holder being the same experience as an e-cigarette. But they did figure out that if you want to design your study to show a null result — to minimize the apparent effect of e-cigarettes on smoking cessation — you should compare nicotine e-cigarettes to non-nicotine and claim that this is the contrast of interest. Based on that insight, they went back and pretended that this existing study had such a design flaw and reinterpreted the results accordingly.
Part of the problem is the entire notion of using clinical trials to study complicated learning- and socially-influenced processes like THR. You might be able to argue that testing medicalized nicotine products (the way NRTs are marketed) can be done reasonably in a clinic because they are used in a very clinical way. But that is not true of most ways of quitting smoking. In fairness to the ECLAT authors, that methodology has some advantages, and they were not actually doing a standard cessation trial. But the RCT fetish that is common among medics who only half-understand scientific research makes it very easy to design a study to fail and claim that it must be a good study because it is an RCT.
RCTs usually have net advantages compared to observational studies when (a) the assigned protocol is a realistic version of what someone would experience in real life and (b) the mere act of having people in a clinical setting and assigning them something does not affect the outcome. This makes them nice for examining medical procedures or treatment drugs, where these conditions are pretty much met. But they are quite bad for studying behavioral phenomena, especially those where, in real life, people fiddle with the details of the methods and act on their own without the artificial pressure of being in a study.
A further complication is what Siegel alluded to: RCTs tend to work better only if (c) it is obvious what to compare the intervention to. Comparing nicotine to non-nicotine e-cigarettes is not an interesting comparison. In any case, despite the rhetoric you hear about placebos, most proper RCTs do not compare a treatment of interest to a placebo, but to the realistic alternative. To see if a new method for performing an appendectomy produces fewer complications, you do not assign half of the subjects to the placebo of being anesthetized but their appendix left in. That would be insane. You compare the new method to the best available old method.
This further emphasizes the importance of point (b). Who do you compare the group assigned to use e-cigarettes to? Should they be given a placebo treatment of just being handed a quit smoking pamphlet that is known to have no effect? If so, you are still looking at people who agreed to participate in the trial (not representative of the population) and are comparing people who were asked to take a major step to those who just throw away a piece of paper and forget the whole thing. To merely control for the entire Hawthorne effect (the effect of feeling like you are being studied) the alternative may need to be more aggressive than that. To control for any placebo effect it would be necessary to give people pills that are inert but described as being a satisfying substitute for smoking (not a “cure” for it), because everyone knows that e-cigarettes are about substitution. That fiction is unlikely to hold up very long.
Basically, the more thought you give to trying to do the science right, the more clear it becomes that there is no particularly good way to do the RCT. Thus, the advantages of observational research over RCTs start to predominate.
—–
As an aside for those who click through and read that Siegel post: You will notice that the thesis of the post is about Legacy failing to disclose that they receive funding from the pharmaceutical industry, which stands to lose sales as a result of e-cigarettes. I have to say that it seems like rather a stretch to demand that a large corporation disclose their relatively modest pharma funding on everything they write. It is kind of like asking FDA to do the same. (Perhaps the more relevant disclosure would be that Legacy was created and funded by a sales tax on cigarettes, the MSA.)
The impact of pharma funding on the anti-THR attitudes of Legacy and other pseudo-health corporations is somewhere between zero and trivial. Part of the reason is that there are much stronger self-interested motives for being anti-THR. More important still, people do not adopt these semi-religious beliefs because of funding. Many gravitate to where there is funding that supports the mission they have adopted, but that is causation in the other direction. Finally, the amount of money at stake is trivial to the pharmaceutical industry. They give grants to keep a hand in things and get inside information, certainly. But it is very difficult to believe that they are so concerned about relatively modest erosion of the tiny tiny corner of their business that is smoking cessation that they would exert pressure on those they fund to attack e-cigarettes.
Recall that yesterday I pointed out that holding an unrealistic view of your enemies’ motives is a recipe for adopting bad tactics. While this case is not quite as dramatic as the one I was discussing, it is another example. It is a mistake to think that pharma cares so much about THR that they are throwing around bribes to try to discourage it (even if you are willing to assume they would be willing to take such actions), and also a mistake to think that those funds play a major role in the decisions of anti-THR actors. It is probably safe to say that the impact is not exactly zero, but there are much more important forces afoot. If we focus on the red herring of donations rather than the major social forces and other base interests, we are likely to be rather less effective.
Tobacco harm reduction, anti-THR lies, and related topics 
I don’t know why certain people lie about the e-cig, I don’t understand the deliberate ignorance and attacks on a device that does what it says it does. All I know is I quit smoking three years ago after a 40+ year, 3 pack a day habit and all I want is to be free to keep using my vaporizer as an alternative to smoking. I’m 60 years old and don’t need ‘permission’ to make my own choice in this matter. To me, those telling these lies are criminals guilty of murder for profit, not people concerned with health issues. Even my grandkids can tell by sense of smell that e-cigs are better than smoking. Anyone pretending to be educated and not factually reporting e-cigs as at least 98.99% safer than smoking is corrupted and probably being rewarded for their lies. How do they sleep? Psychopaths sleep pretty well I hear.
Please make sure to get your story into our testimonials collection! You can just copy and paste this if you do not want to take the time to give more detail.
Be careful about making overly precise comparative risk claims. My estimate puts smokeless tobacco in the order of 1/100th the risk from smoking, but it is impossible to be much more precise (indeed, a better way to say it is “1% of the risk, plus or minus 1%” — i.e., it could be zero). The best guess would put e-cigarettes in the same range, though most likely slightly more hazardous than ST. Even saying “99% less harmful” claims a bit more precision than we actually have, though it is clearly intended to be rough. Adding a decimal place or two to that definitely cannot be defended.
Carl, this post just shows my frustration with those who will not admit the truth about e-cigs or at least leave us be to make an adult informed choice. I will make a thoughtfully worded, less angry testimonial for you.
I recently ran across the phrase ‘intention to cheat’ analysis…
Delightful. I will try to remember that one.
Quick comment about the aside rather than the main post (I am more optimistic about the value of controlled studies than you but don’t have time to go into a detailed response right now!)
Re: ‘opinions bought & sold for pharma funding’ – yes, I agree completely and, more so, think this is an actively unhelpful route to go down. To some of the people who might otherwise be won over, it conceptually lumps HR/e-cig advocates along with groups like anti-vaccination campaigners and quack remedy people who adopt similar simplistic dichotomising rhetoric (‘big pharma bad, dolphin reiki good therefore everybody that disagrees with dolphin reiki is a big pharma shill!)
You get tarred with the same brush as these types, and it becomes much easier to dismiss your arguments as non-credible fringe views. So it’s really tactically unwise & I wish people would stop doing it (as tempting as it might be).
I had to look up “dolphin reiki”. Pretty funny. So long, and thanks for all the research grants.
You left out the bit where it also lumps in with ANTZ who object to funding from industry (except for themselves, of course). In every case, it is pretty much the classic example of anti-liberalism: “whatever I do is ok; whatever I do not choose to do is bad” or even “…should be forbidden”. And, to rephrase your point, it is typically employed by those who know they have no valid argument, like magical dolphins, which is all the more reason to avoid it when you do have a valid argument.
E-cigs are not necessarily tobacco products even though they are regulated as such. The only “tobacco” in an electronic cigarette is the flavor, and thats only if you choose a tobacco flavor in the first place.
The real research needs to be done on PG and VG via inhalation to the human body.
And before I hear: “PG IS USED IN ANTIFREEZE”
Yes, It was added to antifreeze to replace Ethylene glycol, as ethylene glycol is toxic to both humans and animals. The FDA allows propylene glycol to be added to a large number of processed foods, including ice cream, frozen custard, and baked goods.
Considering that e-liquid consists of flavor, nicotine, and over 80-90% PG or VG (which breaks down into sugar and water btw), the next time someone decides to talk about how “toxic electronic cigarettes MIGHT be” they should stop and look in the fridge….
SOOO
By the logic of this study, electronic cigarettes do not promote cessation, and are a placebo effect.
A simple search will include a number of definitions of placebo, including one definition that indicates it “must be harmless”. But one thing is shared amongst all definitions, a placebo has no effect.
And this is where it falls apart…. ANY non-smoker who takes a few drags off of a 24/mg e-cig is going to get a decent nicotine buzz (I still get one after smoking for 10 years)… which, by definition, indicates that electronic cigarettes cannot be a placebo.
If it has an effect AT ALL it is not a placebo, and therefore the argument is invalid.
As I noted, exactly what constitutes a valid comparison in a trial is a bit of a tricky question. One strategy is to try to provide something that eliminates the Hawthorne effect and placebo effect (the effect of knowing you are being studied and the effect of believing you are receiving a useful treatment, respectively), so that any additional effect is “real”. But what constitutes a “useful treatment” for a consumption choice? Thus my suggestion about a magic pill and the observation that the fiction would not hold up long.
To do a trial of e-cigarette-based smoking cessation, the obvious strategy would be to compare to a study arm that received a standard “state of the art” NRT-based intervention. I assume we will see some such studies soon. They still have some serious epistemic flaws, contrary to the RCT fetish myth, but it will be pretty hard to design them to fail (which is why we will probably not see them coming from the tobacco control industry).
As for studying the effects of inhaling PG etc., yes, we could use more information about that. It is good to know that ingestion is safe and that short term inhalation exposure seldom has negative effects, but those are not quite good enough. Don’t get too excited about those “90%” type thoughts, though. Cigarette smoke is in the order of 99% air, after all. (There is good reason to believe that the other exposures pose trivial risks based on absolute quantities, but it is not because they are a small percentage of the whole.)
And as for “tobacco products”, there are actually surprisingly few jurisdiction that explicitly regulate e-cigarettes as a tobacco product. My use of the collective term is based on e-cigarettes existing in the tobacco products space in common thinking: They serve the same basic purpose as other tobacco products and fill the same consumer and market niche; if you are concerned about form rather than function, it is also the case that the key ingredient comes from tobacco and is its defining characteristic. Moreover, on the tactical side, I think it is a lot more useful to try to reclaim the “t word” from those who use it as a negative epithet rather than fight the hopeless battle to pretend the characterization does not exists — you might recall some previous effective uses of that tactic for other “isms”.
What someone really needs to do are real life effects of Electronic Cigarettes. I can’t be the only one whose blood pressure has gone from 140/90 to 115/65 after switching solely to E-cigs. Not to mention my heart rate going from the mid 80’s to the 60’s on a regular basis. Not to mention that I don’t cough anymore and I can take deep breaths again. Real life medical studies would be nice. I have been a non smoker of traditional cigarettes for over two years now. I have reduced my nicotine input from 18 mg to 6 mg and many juices that I have have no nicotine at all in them. How can they dispute the fact that Electronic cigarettes are safer than traditional cigarettes? JMHO
That is the type of comparison where RCTs would be valid since most of those outcomes (perhaps not including self-assessed breathing) are biological and largely unaffected by knowing you are being studied (though they are highly affected by knowing they are being measured at a given moment, which is a problem for any study design). But since we are unlikely to have RCTs that involve sufficiently long-term switching, and observational studies are perfectly good for measuring this, it is likely that such evidence will be observational.
Of course, since the ANTZ favor their own personal version of a perfect world, they are going to want to compare the results of switching to the results of quitting entirely. This would be interesting (it would help address the question of the risk from e-cigarettes compared to nothing). Unfortunately, observational results are likely to have more noise (from confounding in particular) than signal and doing the RCT would be quite difficult (though not impossible) and is unlikely to happen. There will probably be very short term RCTs looking at this, but that is not really what is of interest.
Carl wrote:
“It is a mistake to think that pharma cares so much about THR that they are throwing around bribes to try to discourage it (even if you are willing to assume they would be willing to take such actions), and also a mistake to think that those funds play a major role in the decisions of anti-THR actors. It is probably safe to say that the impact is not exactly zero, but there are much more important forces afoot. If we focus on the red herring of donations rather than the major social forces and other base interests, we are likely to be rather less effective.”
While Legacy’s e-cigarette prohibition policies weren’t due to drug industry funding (as Legacy has billions of dollars of MSA funds, and Cheryl Healton supports only NRT products for THR), its obvious to me and many others who have actively advocated for THR products and policies that the several hundred million dollars drug companies have given to CTFK, ACS, AHA, ALA, Pinney Associates, ATTUD and others has had enormous influence on (and more than paid for) the policies, policy activism and propaganda those groups have done opposing e-cigarettes, dissolvables and smokeless tobacco products.
And while some tobacco control recipients of drug industry funds have NOT publicly opposed THR products or policies, its obvious that the reciept of drug industry funds (and the desire for future funds) has influenced virtually all of those folks to remain silent about THR and to remain silent as many of their drug industry funded TC colleagues make and repeat false fear mongering claims about THR products and policies.
At the WCTOH in 2000, ACS executive director John Seffrin praised GSK for its secret exclusive endorsement contract with ACS for Nicorette, Nicoderm and Commit. Like all other exclusive endorsement contracts, GSK almost certainly inserted a clause in that contract prohibiting ACS from endorsing any competitive products including other NRT brands and other smokefree tobacco/nicotine alternatives.
But that contract was never made public, and ACS never disclosed that huge financial conflict of interest. And if it weren’t for drug industry funding, CTFK wouldn’t even exist. Please note that CTFK has set/coordinated nearly all tobacco policies for ACS, AHA, ALA during the past decade.
Drug companies only gave CTFK, ACS, AHA, ALA that money so that those groups would endorse their drug industry products as the only safe and effective way to quit smoking, and to oppose all other competitive products and manufacturers.