Analysis of the new ASH ecig survey

by Carl V Phillips

One of the nice things about having a scholarly blog is being able to write what are basically extended footnotes for papers. Writing anything other than simple lab reports in the health sciences is extremely difficult for real scholars because of the length constraints: pressure to keep papers too short to do serious analysis and the lack of footnotes. These result from a combination of atavism (to fit more papers into 20th century paper journals), a warped sense of how science works (hint: pronouncements from authority), and a fear that if people knew that actual thoughtful analysis was an option the whole enterprise of churning out thought-free claims would collapse. Since I need to reference the ASH results for a forthcoming paper, and because someone asked me about them, here is my assessment.

ASH’s report can be found here. The reporting is typical for a public health article, which is to say that it is woefully inadequate for real scientific assessment. I have to guess at most of their methodology.

As far as I can determine from the sloppy reporting, the survey sample was kinda sorta representative of the UK population. (They refer to “Great Britain” without clarifying whether they are using the definition that is all of the UK or just the main island — more sloppy reporting. I will just use “UK” as close enough. Maybe someone can clarify in the comments.) There are a few clues that it was not really quite representative, but not enough information is reported to say anything more. Thus, we should assume there is nontrivial sampling bias, but it is a wild guess as to its nature.

But assuming it is basically representative, it offers some interesting estimates of population statistics which are not possible from the convenience samples of vapers (i.e., every other survey result you are familiar with other than some population-representative surveys that just ask about usage). The downside is that because representative surveys are expensive and ASH was wanting to cover a lot of topics, there is a modest sample of vapers and the questions to them are limited. Anyway, assuming (and we have to just assume due to the paltry methodology information) they got this right, we have this, based on their new and previous surveys:

Who do they count as an e-cigarette user? They don’t tell us. If you scroll down to Figure 14 you see that it allows for “less than once a month”, so it is a pretty sweeping definition. Perhaps it is was something like CDC’s “one puff in the last 30 days” or perhaps it was self-identification. Perhaps someone among those who are hyping value of these statistics could clear that up in the comments.

Figure 14 does show that a large majority of the ex-smoking vapers used e-cigarettes “everyday” [sic — hey, who am I to complain about typos], and that two-thirds of the dual users used them every day or “three or four times a week”. What about those using them ten times a week but not every day? We don’t know. Their options for usage frequency were flawed. At least we can hope that the definition was consistent across years, and that the distribution across usage levels was similar, and look at the trend.

I was asked how the time trend compares to the predicted pattern of adoption from my modeling efforts (see these posts for more). I would say it fits pretty well. I predicted a logistic growth curve — slow growth for a while, followed by a rapid climb, and then slow growth again, perhaps toward an asymptote — and this seems to fit. The rapid climb started about five years ago for the UK and now we are down to slow and slowing growth. Roughly speaking, we may be detecting the “peak ecig” level, at about 6% of the population maybe on its way to 7%, based on whatever definition of “user” ASH used.

Of course, that is a sloppy shorthand when we are talking about a social phenomena. There are no constants, nor even long-term equilibria, in social science. Cohort replacement alone means consumption prevalence statistics are likely to change over time, to say nothing of changing social factors within the population. I played with some of those in my modeling. But those do not change the model’s basic, almost-inevitable prediction that rapid growth will occur once and then end, and after that changes in prevalence will be much smaller. Notice in particular that while the absolute growth rate has slowed, the relative growth rate has plummeted (because the installed base of users has grown). This suggests that the “contagion” effect — of each vaper “transmitting” the idea of vaping, creating more vapers and thus positive feedback in the prevalence — is nearing saturation of the “susceptible” population.

Of course, the UK is not a homogeneous interacting population. There are presumably geographic and demographic subpopulations that are nearing their peak and others that are far from it. So prevalence in some of those latter groups may still be in the rapid growth phase, with the change obscured by the dilution into the whole population. Thus the quasi-equilibrium level of usage might be higher than what the overall slowing curve suggests, continuing to be pulled up by pockets of “susceptible” smokers who have not been adequately “exposed” yet.

Moving on, ASH observes that e-cigarettes use is “negligible” (no number reported) for never-smokers and then reports this:

This has been badly misinterpreted by some commentators, who do not seem to understand the nature of simple dynamic systems. The number of “dual users” has not dropped; it has been flat for three years. What we are seeing here is a stock-flow effect. Overstating things for clarity, ex-smoking vapers accumulate (a growing stock), while current-smoking vapers are in a disequilibrium state on their way to giving up one product or the other (and thus are the current flow of new vapers). As the stock accumulates the portion of it who are part of the recent inflow necessarily drops. Yawn.

Of course, exclusive vaping is not actually an absorbing state; some quit entirely and some return to smoking. And dual using is not necessarily a disequilibrium — some people quite like that choice. But there is some tendency toward that exaggerated stock-vs-flow pattern, which makes this trend inevitable. (If this is still not clear, look at ASH’s Figure 4, which directly illustrates that most new adopters of e-cigarettes still smoke, while most who have been using a year or more do not. Their analysis gives no sign of understanding how these figures relate to each other.) What this does mean is that the observation about disequilibrium is supported by this data, or put another way, it supports the claim that dual users are often “in transition”.

All of the above observations are further supported by ASH’s Figure 2 that shows the portion of current smokers who also use e-cigarettes growing only slowly toward 20%. The portion of smokers who have tried them, which obviously is going to slow over time, is flattening toward about two-thirds. It is, of course, possible that many among the remaining third would want to adopt e-cigarettes if they tried them, but it is a safer bet that most of those who were more likely candidates for adopting them have given them a try.

ASH went on to ask a vapers in their sample who were smokers (n=330) and ex-smokers (n=329) why their “reasons for using” e-cigarettes (Figure 5). The sample sizes means that random error is nontrivial (they did not report confidence intervals, but to give you an idea, the 95% CI around 30% would be (.25,.35)), in addition to whatever sampling bias (they did not use their population weighting corrections for these) and measurement error.

We do not know exactly what they asked, only how they characterized the answers. We can surmise from the answers (Figure 5) that the question was actually some version of “why did you start using e-cigarettes” and that they were somehow instructed to identify all that applied. The details of both of these matter for interpreting the results, but are not reported. Because of the huge overlap inherent in some of the answers  (e.g., “to help me stop smoking entirely” and “because I felt like I was addicted to smoking tobacco and could not stop using it even though I wanted to”), the structure of the questions matters a lot: If it was just a list with a “which of these describe you” instruction, some subjects for whom both of those were true might pick only one, while if it was a series of separate questions of the form “Do you agree with the following statement: I started using e-cigarettes to help….”, then there would be less risk of that.

Also a couple of the reported answers are a mess, and so we can surmise that the question was a comparable mess, including the second one above and “because I had made an attempt to quit smoking already and I wanted an aid to help me keep off tobacco”. Does that mean someone was already smoking abstinent and now seeking a substitute, or just made a failed quit attempt (as most everyone would have) and was looking for a new method to try?

I can only find a few seemingly meaningful observations in these results. About half of those who quit smoking and a third of those still smoking identified saving money as a motivation. Several results support the observation that there is “accidental quitting” though at a fairly low rate: Of those who quit smoking, 7% said they intended only to cut down on smoking but not quit completely, 8% said they wanted to continue to smoke but wanted to vape in smoking-prohibited venues, and 10% said “just to give it a try” (no idea how that was asked). Presumably those are mostly the same people, but it suggests a small but nontrivial rate of e-cigarettes inspiring accidental quitting.

ASH goes on to show a trend among all survey subjects toward believing that e-cigarettes are at least as harmful as smoking, rather than less harmful, a lot less, or harmless. The toxic effect of anti-ecig propaganda on this perception has been widely discussed and is not my interest at the moment, so I will leave it at that. I will just note that the survey question was a great demonstration of both bad survey design and/or Dunning-Kruger: “don’t know” was an option, but relatively few people chose it. If the question was “what is your perception” then “don’t know” should not be an option (perhaps something along the lines of “I have never even thought about it so have no perception” would work, but “don’t know” is not an answer to “what is your perception”). If the question was “which of these is true” then close to everyone should choose “don’t know” (they don’t know, after all — but, hey, the first rule of Dunning-Kruger Club is that you do not know you are in Dunning-Kruger Club). The decision about whether to choose “don’t know” will be idiosyncratic, rendering the answers fairly meaningless: some subjects will pick their best guess even if they would not bet even money on it, while some would still say “don’t know” if they would bet 10-to-1 on their best guess. Bad. Survey. Design.

Moving on, they asked the smokers who had tried e-cigarettes but did not use them (n=703) the main reason they “stopped” (Figure 8). I put that in scare quotes because “tried” could mean one puff or one session, and it is really odd to describe not trying something a second time as stopping (indeed, 16% indicated they had merely tried them to see what they were like). ASH focused on the 9% of the answers that relate to the previous point, an unrealistic worry about the health risk. But about half gave an answer that they were simply not a satisfying substitute (which ASH notes) and that feeling was probably also true for many who identified some other main reason (which they did not mention and did not measure because they unwisely asked the question in terms of “main” reason). Answers to the next question (Figure 9), what might prompt those subjects to try e-cigarettes again, showed a similar pattern.

There are two general ways to interpret (or spin, depending on your relationship with science) this result. One is that better e-cigarettes that directly address whatever inadequacies the subjects experienced would attract more switching. The other, in keeping with “peak ecig” concept, is that most of those who would rather vape than smoke (all costs and benefits considered) are already vaping. Parsing the answers against some time data (either a secular trend or recency of trying e-cigarettes) would go a long way to sorting out which of those was dominant. If the portion of those who reject switching is going down over time (as the products improve), that would argue for the “better e-cigarettes” interpretation. If it is going up, it would suggest that most of those who might really want to switch (after experiencing e-cigarettes) have already done so.

In Figure 10, ASH reports the reasons given by smokers for not having tried an e-cigarette. Further to the above point they are trying to spin up, they emphasize that the plurality response was concern about safety. But I am inclined to dismiss responses to this question entirely. People make decisions for myriad conscious and subconscious reasons, but will backfill a seemingly good and rational reason if asked. But often that answer bears little resemblance to the real motives. At least with an affirmative decision, like why someone tried e-cigarettes, there was probably a clear moment of conscious thought we can ask about. But for the decision to not take a particular affirmative action there is not, and the backfilled response is meaningless.

ASH goes on to drill a little deeper noting (Figure 11) that smokers who have not tried e-cigarettes overestimated the contribution of nicotine to the risk from smoking. But then again, so did the vapers, so not much to see here. (And they have the “don’t know” problem again in this question.)

Figures 15 and 16 show a time trend toward increasing use of open systems, with 81% of ex-smokers and 63% of current smokers using them in the recent survey. This is seemingly unique information because we do not have good disappearance data that would let us estimate that, and surveys of self-selected vapers cannot estimate it for obvious reasons. Of course, it cannot be extrapolated beyond the UK.

Figure 17 reports an uninteresting time trend toward current vapers considering e-cigarettes more satisfying as compared to cigarettes. I say “uninteresting” because this represents an unknown combination of e-cigarette technology getting better, the mix of products chosen tending toward more satisfying options, an increase in the average period of e-cigarette use resulting in users becoming more accustomed to them or better at using them, and the accumulation of dedicated users meaning that a larger portion of all current users really like e-cigarettes. This result is inevitable. An interesting question would be what is the mix of reasons, and also why the change was surprisingly small, but ASH offers no insight about those.

ASH finishes with some questions about what e-cigarette liquid varieties people use, which resulted in their notorious statement that the E.U. Tobacco Products Directive’s arbitrary restrictions on nicotine density was no problem because only 9% of vapers use levels that will be banned. That, in turn, resulted in a great deal of “with friends like these…”-themed responses. I will leave that fight to others. I will, however, note that ASH’s data does not actually support their claim: 12% of respondents said they did not know what nicotine density they used. These are presumably users of disposable and cartridge systems (anyone buying vials of liquid would know what they are buying), many of which have higher nicotine densities. That seems to have been overlooked by those criticizing ASH’s pro-TPD message.

The final result is potentially interesting, identifying which flavors subjects use most often, but the question is badly flawed, making it hard to interpret. Rank-based questions (i.e., what is used most, rather than ever) about categories are always bad survey design, and generally bad communication (if you are trying to do science rather than propaganda, that is). It is always possible to divide or combine categories differently to alter the ranks. It turns out that tobacco flavors topped the list at 33% and menthol/mint tied for second at 22%. But what were they up against? Other categories included “fruit” (which tied mint), “chocolate, desserts, sweet, or candy” (wait, aren’t a lot of candies fruit flavored? and don’t Brits call desserts “pudding”?), “vanilla” (hold on, now, who thinks of vanilla as a separate category from desserts?), and “energy drink or soft drink” (which are different from “sweet” because…?).

It is too bad they screwed up this question so badly because it would have been interesting to know how many vapers preferred the cigarette-like flavors compared to very non-cigarette-like flavors (and also how that cross-tabs with how long someone has been vaping). But because they chopped up the latter, we cannot tell. Consider: someone who uses tobacco flavor 21% of the time, and each of vanilla, chocolate, cola, and cherry flavors just under 20% of the time would correctly answer that they use tobacco flavor the most. We know that vaping enthusiasts mostly prefer non-cigarette flavors (e.g.) but our information about the average vaper is limited to what we can surmise from the limited disappearance data. And thanks to ASH not getting someone more skilled to write their survey, it is still mostly limited to that, though the results tend to confirm the (obvious) point that vaping enthusiasts are more into a variety of flavors than the average vaper.