by Carl V Phillips
Fairly often (e.g., in the previous post) I make reference to the concept that extraordinary claims require extraordinary evidence. That is, if something seems extremely unlikely based on a great deal of accumulated knowledge or an understanding about how the world works, and you wish to claim it is true, you really need to have done some tight work. It is a good principle in science. Research does not produce scientific knowledge without adherence to principles like this (note that there are no “rules” in science, so we have to make do with evolved principles).
Today I am thinking of that in terms of a new study that was reported in this BBC story, “E-cigarettes ‘help more smokers quit'” (quotes from there).
No I am not talking about this:
Public Health England says e-cigarettes are 95% less harmful to your health than normal cigarettes, and when supported by a smoking cessation service, help most smokers to quit tobacco altogether.
That was a non-extraordinary claim that was apparently based on nothing. I only quote this as part of my campaign to prove to Clive Bates et al. that when fake numbers like this (which in this case grossly overestimates the reasonable assessment of the risk from vaping) are out there, they get reported as if they were valid point estimates.
Nor was I talking about this:
But health professionals say the most effective way to quit smoking remains through prescription medication and professional support from free local NHS stop-smoking services.
The actual claim that is being made — that at least two people who might be called health professionals say this — is obviously true. The fact that the BBC wrote a patently false statement in a way that was designed to suggest it was accepted fact is also not all that interesting or surprising.
No, I was talking about the crux claim:
The authors [Robert West and his peeps] say vaping may have helped about 18,000 extra people in England successfully give up smoking in 2015.
The team, from University College London and Cancer Research UK, say theirs is an observational study, and therefore cannot prove direct cause and effect.
I include the second part of that as pure rhetoric, to point out that these authors apparently think that there is some type of real-world research that can prove cause and effect. That is not an extraordinary claim; it is simply wrong. Same with the suggestion that “direct” has any meaning in this context, though that word may have been the reporter’s fault.
The extraordinary claim is that the vast majority of smokers who switched to vaping would have quit anyway in 2015 if e-cigarettes had not been an option. No, they do not directly say that, but it is right there: Several multiples of 18,000 smokers switched to vaping that year or used e-cigarettes as a transition to abstinence. If vaping only “helped” 18,000 of them, then the authors are saying that most who switched to vaping as an ongoing practice (let alone tried it as a transition measure) were not caused to quit by vaping. Put another way, most switchers would have quit smoking in 2015 anyway and so their use of e-cigarettes was just a waste of their time and money.
Given the portion of switchers who report that switching was the only thing that ever worked for them, and given the many previous occasions on which the average smoker considers that he tried to quit but failed, this is quite extraordinary. This does not mean the claim flatly contradicts what we know (unlike a couple of the above points); it is possible. But it seems exceedingly unlikely.
What is the evidence? Well I have not fully gotten my head around it yet (but have been tasked with doing so, so expect me to circle back on this), but basically it is a complicated and assumption-laden model. Such models do not tend to create extraordinary evidence, especially when they are presented as just-so stories rather than being thoroughly tested for weakness with every doubt the researchers could throw at them.