by Carl V Phillips
There have been a spate of claims about the comparative risk of e-cigarettes compared to smoking, triggered by the recent Public Health England (PHE) report which was published under a headline claiming that e-cigarettes are 95% less hazardous than smoking. My previous mentions of this are here and here. The 95% claim has led to criticisms and attempts at clarification, every single one of which, as far as I can tell, is wrong. Everyone in sight is trafficking in non-science at a minimum, and nonsense quite often. However, it turns out that a reviled tobacco controller commentary on the matter comes closest to getting it right.
Two simple observations:
Observation 1. To derive a quantitative estimate of comparative risk, you need (a) research that provides the relevant epidemiologic estimates and (b) arithmetic, or (c) a close analog where there is already a quantitative estimate and (d) a quantified analysis of what differences might exist.
Want to report the ratio of people killed by guns versus cars in a particular country? It’s easy, so long as you are talking about any place with decent record keeping. Look up how many people are killed by each. Divide. Done. You can also come pretty close to doing the comparative risk calculation for smokeless tobacco, because we have the data. [Note: if that link does not work for you, you can download a copy here.]
Want to figure out the risk from a new generic drug? Look at the estimated risks for the name-brand version for a pretty good approximation. If you want to improve on that, look at any chemical differences between the products and assess whether those seem like they might matter; if so, do what you can to adjust the estimate based on this. Done? Not really, because actual observations of the effects would be better, but you have an evidence-based estimate which can be improved later if it is worth the effort. Something similar could be done for e-cigarettes.
If someone is claiming a quantification and there is only vague hand-waving toward (a) with none of (b), or a hint of (c) with none of (d), then you can be sure it is not a scientific estimate.
Observation 2. E-cigarettes are not all the same and vaping behavior is not all the same. I would like to think that anyone writing about them knows this. But apparently many such authors do not realize it matters. Unlike many other consumer product categories, the variety is so great that the resulting range of health risks is huge.
Can you make a particular e-cigarette and use it in a particular way that causes more health risk than from smoking? Yes, obviously. I have no doubt that out there somewhere is someone whose choice of vaping products and behavior makes his vaping more harmful than if he were smoking instead. (We do not know enough to say which products and behaviors pose such risk. But it is sufficiently plausible and there is sufficient variety and quantity out there that it is safe to predict it is true.)
Can you use a particular e-cigarette in a particular way so that it is 10% as harmful as smoking? 1%? 0.1%? Yes to all of those, obviously. How about -0.1% (i.e., it is somewhat beneficial for one’s health on net)? Quite likely.
No, I am not playing the “take refuge in complexity” epistemic nihilism game the ANTZ do, saying that since there is not an easy answer so we know nothing. You know me better than that, don’t you?
Notice that I referred to smoking as if it were homogenous, when obviously it is also not. Products vary enough that risks from them inevitably vary. The differences could be in the range of twenty percent. (Tobacco controllers like to recite a mantra that all cigarettes are exactly equally harmful; this is so obviously false it serves as a shibboleth of scientific innumeracy.) Behavioral variations matter more. But despite the complexity, it is still meaningful to refer to a generic quantity, “the” risk from smoking — averaging across the relatively stable mix of products and behaviors that happen to exist (ideally excluding the lowest risk versions of the behavior, like smoking only one cigarette a week, that distract from the measure of interest).
But there is no such summary construct available for e-cigarettes. The harms from different products and usage patterns — even restricting it to regular users and relatively normal products and behaviors — almost certainly varies by full multiples around its median, not merely tens of percent. The range is far wider still if we consider the outlier products and behaviors. Product technology continues to change rapidly, probably in both directions for risk (i.e., most of the more conservative products are probably becoming lower risk; some of the new wilder products are probably offering higher risk options).
So, put these observations together and we immediately see that declaring e-cigarettes to be 95% less harmful than smoking (stated as a precise point estimate, with no caveats) is nonsense. Given Observation 2, it simply makes no sense to imply — as the PHE report did — that the risk from e-cigarettes is a single well-defined number, let alone a constant over time. Of course the precision implicit in the stated estimate is absurd in the first place, but it is made far worse by them not defining what particular mix of e-cigarette products and behavior the claim applies to.
The PHE authors fail to do this because they do not know; they pulled the number from a source whose problems are so enormous that issues like “you did not ever say what mix of products are you talking about” does not come close to making the top ten list. This is a case of working that magic that turns vague junk science claims into concrete “facts” (if you follow the tobacco control literature, you are familiar with that process).
The claim is attributed to Nutt et al. (2014), which was not a scientific study, but rather a group of people getting together and making up numbers. There is absolutely no indication they engaged in any of the scientific approaches described in Observation 1. Did they even consider the tiny bits of epidemiology we have about e-cigarettes? Probably not. Did they do any of the necessary calculations? Apparently not. Did they reason by analogy? Quite clearly not, since what they claimed about the analogous product that we do have evidence about (smokeless tobacco) was clearly wrong. Did they analyze the difference between e-cigarettes and smokeless tobacco to try to get a measure of how they might differ? Yeah, right. All you need to know to understand how little this resembled science is that a committee of eleven people made up about 200 numbers over the course of two days. That allows for less than ten minutes for each made-up number, not even considering the time to type up the numbers and taking time to change the keg. They did not have time for any scientific analysis.
Oh, but it gets worse than even that. Not only did they just make up the numbers, but they did not actually even report the made-up number that the PHE authors claim. The PHE claim specifically refers to “95% safer”, but the numbers reported in the paper are an arbitrary amalgamation of a list of 14 effects, including not just health risks (as the PHE report claims), but also personal financial costs, social spillovers, and strange constructs like “dependence”. Some of the numbers measure the effect of using the product on a particular user, while others are the utterly incommensurate aggregate effect on society of all the use of the product. Yet they just add up their made-up numbers for each of 12 different products into a single histogram, their only reporting of their “results”. What do those sums even mean? Nothing really. What is their conversation rate between one unit of an actual cost and a given amount of dependence, crime, or environmental damage? No one knows — not even them, since the latter cannot even be meaningfully quantified. What is their tradeoff between X money spent and Y mortality risk created, or among their four different measures of health effects? Those could be meaningfully quantified, but they do not do it. How could they? Solidifying and defending any one of these conversion rates would take a whole paper in itself.
But even if this amalgamation had been done in a remotely valid way, it would still not be a measure of “safer” as the PHE authors reported. Most of the inputs have nothing to do with “safer”. To see this you need only consider one of the many problems with Nutt et al.: a large portion of the cost attributed to smoking is the financial harm caused to smokers by cigarette taxes, not by smoking. (But did they count the taxes collected as a positive in the social impacts categories? Apparently not. Did it occur to them that pure transfers like taxes should not be counted as net costs or benefits when doing a social-level analysis? When you only have ten minutes to make up each number, there is really no time to stop and discuss even basic principles of economics.)
I suppose it is possible that the PHE authors were basing their 95% claim on just the made-up numbers about health effects only, though even this would still be an arbitrary amalgam of at least four different made-up numbers for each product. But if so, they did not get them from the cited paper because the authors never reported their numbers, just that histogram, and the details are obscured by the graphics resolution. (Presumably their failure to provide the obvious and simple table of all their input numbers was to reduce the chance that readers would notice (a) they were just making up a boatload of numbers they should have been attempting to estimate based on a well-defined methodology and actual evidence and (b) many of the numbers they made up that can be discerned from the histogram are pretty absurd.) However, if the authors were working from such inputs, they ought to have cited to their actual source (“Reference 10: Personal communication with the authors who made up these numbers.”) rather than to the paper, so presumably they were just using the histogram.
The Lancet published an editorial disputing the PHE report, particularly the 95% figure. This has been feverishly condemned by pro-ecig activists since it came out yesterday. Indeed, there is plenty to deride about it, including the usual ad hominem attack on some of the cited authors, which the dim-witted people who believe in the “public health” movement seem to consistently mistake for an argument. But the criticism of the source of the number comes remarkably close to being right if you leave out a few words:
But neither PHE nor McNeill and Hajek [two of the PHE authors who wrote a subsequent defense of it] report the caveats that Nutt and colleagues themselves emphasised in their paper. First, there was a “lack of hard evidence for the harms of most products on most of the criteria”. Second, “there was no formal criterion…”. In other words, the opinions of a small group of individuals with no prespecified expertise … were based on an almost total absence of evidence of harm. It is on this extraordinarily flimsy foundation that PHE based the major conclusion and message of its report.
This is basically right. Indeed, it understates the case against using Nutt et al., presumably because it was written by a Lancet editor — i.e., someone who is not actually very good at critically analyzing science. In the first quoted observation, the original authors present a facile excuse for their junk science. In reality, their potentially well-defined numbers like health and financial costs could be defensibly estimated based on scientific knowledge. The real problem is that someone would have to do a serious study to figure each one out, which would get in the way of the authors’ preferred approach of getting together for a bull session and making up a couple of hundred numbers in two days. The ellipses in the second quoted observation omit, “for the recruitment of the experts“. While it is true that the selected participants were an arbitrary group (notably lacking experts in several fields that would have been needed to do this exercise correctly, even if they took the year it would take to do the work), “no formal criteria” appears to describe everything they did.
Alas, the second ellipses doom the Lancet comment to being typical “public health” garbage despite them coming rather close to getting it right. The omitted phrase is “in tobacco control“. Apparently the Lancet editors would have been just fine with people having a bull session to make up numbers if the fiction authors in question were the right people. No, not experts in the relevant epidemiology or economics, nor experts in the specifically relevant toxicology or in how to do cost-benefit type tradeoffs. No, they should not have been scientists at all, but rather people with expertise in tobacco control (which is branch of marketing and lobbying, not of science). Oh, and there is also the problem that Lancet entitled their editorial with the phrase “evidence-based confusion”, whereas their analysis almost managed to accurately point out that the real problem is that the PHE claim was not evidence-based.
The other problem with the editorial was the subtext. The Lancet is trying to imply that the 95% number is too optimistic about e-cigarettes, while the reality is that — for the average e-cigarette out there and the average consumer behavior — it is far too pessimistic as a point estimate. As I noted before, if e-cigarettes are really 5% as harmful to one’s health as smoking, I would be recommending that people seriously consider quitting e-cigarettes as soon as that would not result in them resuming smoking. After all, 0.05*{the harm from smoking} is still a pretty big number.
In response to this, some pro-ecig commentators have tried to rewrite the PHE claim as saying that e-cigarettes are at least 95% less harmful, or that 5% as harmful is an upper bound. But this is wrong too. Those authors generally offer no more scientific basis for their claim than the PHE authors did for theirs. Observation 2 tells us that such a claim is clearly not true for all e-cigarette use, but these commentators fail to include the necessary caveats to make clear they are referring to some particular devices and behaviors. Moreover, we cannot rule out that current e-cigarettes are worse than that on average. Yes the reasonable point estimate is is well below that, but it is genuinely true that we do not know everything, and in particular we lack observations of long-term effects, and so cannot rule out a risk greater than 5% of that from smoking. Of course, for many important questions there are long thin tails of uncertainty around our best estimates. The fact that we should still make decisions according to the best current estimate (with some complications that do not apply in this case), does not change this.
Bottom line: Pro-ecig commentators and supporters of THR are used to responding to tobacco controllers’ innumerate, dishonest, anti-scientific attacks on accurate science. But in this case, the critics have — despite being prevented from getting it quite right due to their serious limitations as thinkers and honest human beings — come much closer to being right than the original claimants. Unless the plan is to descend into mimicking the tobacco control disdain for science, this would be a very good time for e-cigarette proponents to learn that not every criticism of an ostensibly pro-ecig claim is wrong.
Tobacco harm reduction, anti-THR lies, and related topics 
Carl,
Is there a link to the comparative study on smokeless tobacco that regular folks like me can have access to?
Thanks:-)
I am going to answer multiple possible interpretations at once:
1. Does that link not work for you? Or do you have to be signed into Researchgate (which is just a matter of a free registration, I believe). If that does not work, you can download the file at this old archive: http://www.tobaccoharmreduction.org/calculating.htm
2. If you are looking for more than the poster, it no longer exists. We used to have a web tool up that allowed the reader to learn more, play with the numbers, etc. But it is sufficiently out of date that we let that disappear when we left Alberta.
3. Yes, it is long overdue to revisit this. Igor Burstyn and I are thinking about doing something shortly.
It is free, but you have to be signed in as a member of an official scientific organization &/or as a student (of which I am in theory…lol) or product developer.
I did not know that. I would suggest registering in any case — it is not like they check. Increasingly, serious researchers break the evil journal paywalls there, so it is a good source of downloads.
I will add the other link to the main text, though.
Carl, I’d like to know how you came to your conclusion in Observation 2:”I have no doubt that out there somewhere is someone whose choice of vaping products and behaviour makes his vaping more harmful than if he were smoking instead. ” particularly as you later state “As I noted before, if e-cigarettes are really 5% as harmful to one’s health as smoking, I would be recommending that people seriously consider quitting e-cigarettes as soon as that would not result in them resuming smoking. “? As you know the formaldehyde scare lead to DrF doing some experimentation that I believe and trust from my own experience, that vapers are very capable of detecting the onset of dry hit and high aldehyde production (it tastes crap) so how else could this be possible? (unless you are assuming an inexperienced vaper might continue vaping something that tasted really really toxic and bad?)
Carl, also, could you suggest a method for coming up with a definitive, accurate measure for the relative harm of ecigs v cigs that doesn’t involve unethical experimentation? Do you think you are being a little pedantic about numbers when the issue is really combating the outright lies of the antz before it is too late? (I believe PHE had good intention)
Working backward through your two comments: I agree that the PHE authors and the agency had good intentions; this is not about intentions. Bad science is still bad science if the intentions are good. And the road to hell, and all.
The measurement method (definitive is not on the table) I envision is: consider a reasonably clean ecig and an “average” usage pattern; assume that any hazards are non-interactive; estimate the hazard from nicotine based on what we know about smokeless tobacco (which would be worth updating from the last time it was calculated); take a guess at the hazards for the other known exposures based on toxicology or occupational epidemiology about them (necessarily rough though it may turn out so trivial it does not matter); add in some quantified point estimate for unknown effects; explain all your methods and report all your numbers so that others can identify any problems or counter with variants; repeat for worst reasonably plausible case and best to create a range.
I am not quite sure the link between the two quotes from me. Addressing the basis for the first, it is a simple “if it is not impossible, given enough opportunities it is inevitable” principle. It is obviously possible. It is not about one single possible failure point for this complicated system. Identifying a single possible point of failure and arguing that it mostly does not happen is not the same as ruling out all points of failure or saying it never happens.
“..consider a reasonably clean ecig and an “average” usage pattern..”
Carl,
What constitutes average usage?
Most/many experienced vapers (from my own observations) prefer advanced systems with more power and lower resistance tanks/RDAs/RBAs.
…Or are you referring to inhalation technique….or time/duration of use? Varying constituents in flavorings?…Or all of the above?
When I wrote that I was referring to inhalation technique and the like. But as for the device itself, it would probably make the most sense to anchor the whole thing in a well-studied closed system (a major manufacturer’s product where they have run lots of labs) and then work from there into how it might change with less clean devices. I am not suggesting the latter step would be at all easy, and it might require changing scales (i.e., in units of “how bad smoking is for you” which we tend to use, the estimated delta among normal products will almost certainly be so small that it is clearly not a natural unit to use), but it is probably the way to go. It certainly offers the most practical advice, along the lines of “if you choose to vape rather than be abstinent, you increase your health risk by roughly the same as…; if you choose to vape with a more creative system rather than the cleanest cigalike, you increase your health risk by roughly the same as…; you should consider these when making your decision.”
My cousin, a trained lawyer (and a good one), told me I should not try to expect science to sway any lawmakers…what they need to see is evidence. I asked her what “evidence” is and she replied, “someone prominent and trusted in the medical field would have to make a strong pro-ecig statement.” When dealing with legislators, Glantz has credibility because he sounds credible. That is how the Challenger was launched …the engineers saying it would be safe were more eloquent than those saying it would not be safe, hence more “credible.” Our legal system does not call for decisions to be made by people who know the subject matter, in fact knowing anything even remotely relevant about a topic gets one kicked off the jury — in other words, off the decision-making body. So…how should we present evidence if we don’t want to get blown up?
I suppose the one word answer you are inviting is: Feynman. The longer non-answer is that I am not talking about persuading people today. This post is about the legitimacy of scientific claims.
Not sure of the exact thing you mean by Feynman being a one-word answer but wow am I glad you got me to look him up. Fascinating scientist and scientific philosopher. And clear speaker. I might need to buy some more books.
Well, you did bring up the science surrounding the Challenger. That kinda screams Feynman.
You are correct Karyyl, and this is obvious when you look at the prosecution of a small business man in Western Australia, Vincent Van Heerdon. http://www.tobaccocontrollaws.org/litigation/decisions/au-20140624-hawkins-v.-van-heerden
Actual evidence of safety or otherwise doesn’t matter, when laws and policies are made on the grounds of hypothetical confusion or risks.
I think Carl does a wonderful job of posting about the scientific validity or not, of claims such as “95% safer than smoking”, but in reality this type of number is useless when fighting the lies and misinformation pumped out by the tobacco control industry.
I know that this number will be used by the nutters in tobacco control to bog down any discussion, which is why it is better for advocates to focus on the attack against the 3 scientists, out of 11, in one study of the 185 studies reviewed by PHE, and the accusations of conflict of interest, which the ANTZ seem to think is a big woopydoo problem, and are attempting to discredit the whole review and report based on this accusation.
Please try to stay on topic. There are plenty of posts where I talk about tactics, but this is not one of them. Indeed, I even talked about why this 5% as bad as smoking claim is bad tactics. The comments are still open on the other posts, so feel free to have this conversation there.
But they said it was a best estimate
Got what you are trying to say, however it was obvious that Lancet editorial wasn’t contesting the science behind 95% but just a slur to discredit, in general.
The Lancet piece was mostly the usual crap. Some version of it would have happened even under the counterfactual where the PHE report stuck to the evidence and valid science. But that does not change the fact that in reality they called it out and were (mostly, basically) right when they did.
I believie them being basically right as inadverent to their actual intent
I must admit that I cannot understand the problem.
In the actual report, in the Forward, there is this statement:
“In a nutshell, best estimates show e-cigarettes are 95% less harmful to your health than normal cigarettes…..”
And in the first chapter, “Key messages” appears this:
6. There has been an overall shift towards the inaccurate perception of EC being as harmful as cigarettes over the last year in contrast to the current expert estimate that using EC is around 95% safer than smoking.”
[My bold]
It seems clear enough to me that the figure if 95% was not intended to be accurate.
But there is another problem – in the context, what does the word ‘safer’ signify? How accurate is it intended to be? Suppose that I was to use the two words ‘harmful’ and ‘dangerous’? Generally, a scratch is harmful but not dangerous, whereas a bullet to the head is extremely dangerous (though there have been occasions where people have survived such trauma). So does ‘safer’ mean ‘less harmful’ or ‘less dangerous’?
I’m just wondering if we are looking for a piano concerto where Chopin’s “Chopsticks” would suffice (though ‘Chopsticks’ is complex enough!).
Quoting from the Lancet article:
“First, there was a “lack of hard evidence for the harms of most products on most of the criteria”. Second, “there was no formal criterion…”. In other words, the opinions of a small group of individuals with no prespecified expertise … were based on an almost total absence of evidence of harm. It is on this extraordinarily flimsy foundation that PHE based the major conclusion and message of its report.”
What that means, as far as I can tell, is that, for whatever reason, there is insufficient data to to provide positive evidence that ecigs are ‘safer’ and that lack of evidence of harm is insufficient and ‘flimsy’. Is that the part that you agree with, Carl?
What I cannot understand about that argument is that such positive evidence is impossible to find at this time and never will be if ecigs are de facto banned by regulation. The substitute, as appears in the report, is evidence adduced by determining the toxicity of the vapour. Why is that not sufficient?
Or am I barking up entirely the wrong tree?
Well, let’s see. I am not sure I fully understand what you are not understanding, but let me try. The foremost problem is that 95% is not a best estimate and it did not come from experts on doing such estimates. I suppose someone in the group that did it maybe has such expertise (though I am not aware of any such demonstrations), but he did not get a chance to exercise it in that process.
Don’t get hung up on the word “estimate”. Some people are trying to use it as an excuse. But “estimate” does not mean “we are reporting something that is wrong but, hey, don’t worry about it”. It is the scientific term for referring to any empirical numerical measure. We have an estimate of the speed of light in a vacuum.
Chopsticks would indeed do in this case. That is the problem. The PHE authors, the Nutt et al. authors, and countless others are trying to make up scientific claims out of whole cloth and pretend they are a concerto when actually they are nothing. But a simple rendition of Chopsticks would be adequate. So why are they lying? It is ridiculously self-defeating (unless defeat is really the goal, as has been suggested).
Re the quoted material, the quotes within it are from Nutt. As I pointed out, the first of these is simply not true. The authors are trying to convince the world that pulling numbers out of their asses is the only option as an excuse for doing it. But actual science-based estimates are possible. There is plenty of evidence the PHE authors could have used to claim that the risk is much lower, but instead of trying to deal with it, they just cited a made-up number.
On ST you take as given the CDC attributable percentages. How robust are they?
On the ‘95% Claim”, I seem to agree with your assessment.
Nutt et al 2014 used a MCDA model but would presumably have used base figures in a similar manner. It was also restricted to Nicotine thus leaving out much of the context. It seems the question was, “if we need to do something about nicotine products what should we concentrate resources on?” This is a management rather than medical assessment. It doesn’t even attempt to determine whether we should be doing anything at all. By simply changing the criteria, smoking (tobacco) can be 100% safer than vaping. How so? Well if only concentrating on harm from batteries, smoking has a distinct advantage. Maybe we need solar powered e-cigs to reduce the risk?
Another group with other priorities may well choose different criteria and weightings leading to different results. Since Nutt et all may have been the basis for the “95% claim” this might explain why some PHE members have been criticized by the Lancet.
Nutt et al 2014 also refers to harms rather than safety and only suggests one thing is less harmful than another (based on their criteria). There seems to be no claims to either Vaping or (of course) Smoking being safe. Of note, there is a statement that pharmaceutical nicotine is safe and a reference to the safety of e-cigs.
The reference, (Konstantinos E. Farsalinoscorresponding and Riccardo Polosa (2014) and as far as I can tell) does not give a specific number for relative safety.
( http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4110871/)
So it would seem the “95% safer” claim lacks credibility in scientific terms though may have benefits in political terms, however there is plenty of room for wiggle and argument.
Not quite sure what you mean by the first line? I do not believe any CDC estimates about ST.
It is misleading to refer to what Nutt et al. did as a model. That would imply that it was capable of telling us something about reality that goes meaningfully beyond simply spitting the inputs back out as outputs. Their process did no such thing. Your description of what it is — a political process to make a non-scientific argument about policy priorities (based on **enormous** unstated normative assumptions) — seems much better than thinking of it as scientific or a model. As I note, even if they were providing something other than made-up numbers, their amalgamation would not be a measure of anything meaningful, yet that is what the PHE authors used.
Line 1 was a reference to “Calculating the comparative mortality risk from smokeless tobacco versus smoking”. Attributions for tobacco related disease. (Table 1). How robust are the CDC numbers?
Use of model. Fair point. Maybe I should have used ‘Model’. I have done this sort of thing when doing Quality Facilitation, within manufacturing, and would have referred to it as a tool or, as you say, a process that may help with decision making.
Pingback: More on the ecig advocates’ descent toward junk science | Anti-THR Lies and related topics
Pingback: Remember, what the US government wants to do to ecigs, they have already done to smokeless tobacco | Anti-THR Lies and related topics