posted by Carl V Phillips
In the previous posts we pointed out how Christina Gratziou’s anti-THR politics-disguised-as-science could be recognized as lies (even by someone who did not understand the science) and explained why her concluding about real disease outcomes from a short-term biomarker study were inherently flawed. This did not even address the what she reported about the research, which I will do today.
Here is the content (all of it):
The study included 8 people who had never smoked and 24 smokers, 11 with normal lung function and 13 people with either chronic obstructive pulmonary disease (COPD) or asthma.
Each person used an electronic cigarette for 10 minutes. The researchers then measured their airway resistance using a number of tests, including a spirometry test.
The results showed that for all people included in the study, the e-cigarette caused an immediate increase in airway resistance, lasting for 10 minutes. In healthy subjects (never smokers) there was a statistically significant increase in airway resistance from a mean average of 182% to 206%.
In smokers with normal spirometry there was a statistically significant increase from a mean average of 176% to 220%. In COPD and asthma patients the use of one e-cigarette seemed to have no immediate effect to airway resistance.
So, what does this mean (other than the obvious conclusion from the first paragraph: that this was a ridiculously small study to be drawing any conclusions from)? One good answer is “nothing”. That is, this was sent out to newspaper reporters who passed it on to the general public, and what it meant to them was absolutely nothing. So those readers skipped past the science and read only the lying assertions that were tacked onto it, assuming that those complicated bits supported the assertions; the lab results were simply there to camouflage the fact that the press release was little more than a political statement.
How can there be any honest motive for sending this information to the general public? I am in well into the 99th percentile of the population in terms of ability to understand this science — easily more like 99.99th — and I could not make much sense of it. Fortunately, I know people in the 99.9999th, so I could ask a few question.
Also, before I started writing about this I asked a few questions of Gratziou, via the press contact listed on her press release. The contact assured me that she would get the questions to Gratziou and I would hear back. I did not hear back. Real scientists generally like to explain their work to other scientists; liars, on the other hand, do not like follow-up questions. (In case it is not obvious, my questions were simple and technical (and thus in no way impolite), and only about the scientific bits.)
What did the researchers do and find? That is not clear even to the expert readers. “Spirometry” is not an adequate description of the measurement method. It is also not clear even to experts what the units of measure mean. Absolute measures would have been more useful, especially lacking any definition for the “100%” that the percentages are based on. It could be that the first of each pair of percentages was the baseline and the second was the result after the exposure, or it could be that the baseline was called 100% and these are the ranges of the final results. The former is the natural way to read the sentences, but would mean that the smokers had clearer airways to start with than the nonsmokers, which seems unlikely.
So even for experts, the information about results presented to the world was meaningless. That does not, however, mean there is not useful information about what was wrong with what they did.
As noted in the previous post, the biggest limitation (even worse than the size of the study) is that it was based on just the immediate reaction to a first exposure. Moreover, the subjects were exclusively people who were not used to vaping, and were given the exposure under contrived and medicalized (and thus disconcerting) circumstances. Would an experienced vaper get the same effect? There is reason to believe not, since vapers quite often report that they had some airway irritation (which would cause the observed effect) when they first started vaping, but it seemed to disappear. This would have been useful to report, though it is not clear whether Gratziou failed to report it because she was hiding it or because her knowledge is so limited that she does not even know it. It is similarly not clear whether these lab-technicians-posing-as-scientists intentionally chose only subjects that would produce uninformative results, or did so out of ignorance.
That “lab technicians” remark is not a slight against techs — we would not get anywhere in lab and clinical sciences without good techs. But scientists need to think beyond, “I have a machine, I have a person, let’s see what happens.” In particular, in a case like this, they need some calibration (aka “controls”). A scientist knows (contrary to popular belief) that it is not always necessary to have a null exposure control group or trial (“placebo control”), but sometimes, like in this case, the results will be impossible to make sense of without it.
Airway resistance measurement is finicky, sensitive to exactly what is being done and subjects’ behavior, as well as anxiety and other factors. We really do not know what the methodology was, in terms of exactly what tools were used and how they were administered, or how they dealt with the challenge of finicky results. (This has led some optimistic commentators to say that we need to wait for the paper to come out; I will wager that we will not know even then — any takers?) But we do know that a lot of things could have produced the results, and there is no comparison group or trial to help deal with that.
A comparison trial where the subjects just sucked slightly flavored air through a tube — ideally without knowing they were merely doing that, but at the very least even without such blinding — and then were measured would have been a useful comparison. Also useful would have been a comparison where the subjects inhaled just steam (water vapor), which creates short-term airway irritation for many people. This would have provided a meaningful comparison for the quantitative results. Instead of reporting utterly meaningless numbers, they could have reported, for example, that the effects were basically the same as someone vaping pure water.
A more subtle point is hidden in the phrase “a number of tests”. Dishonest researchers who are trying to get a particular result — and we have established that this describes the present case — frequently take multiple measures and report only the results they like. In toxicology, this consists of doing lots of different trials with minor variations of the exposure and reporting the one that finally produces the “right” results. In epidemiology or econometrics this consists of running multiple statistical models on the data and only reporting one of them. In this case, it sounds like multiple lab tests were run, but only one of them was reported. The mathematical analysis of the results of such cheating is complicated and usually oversimplified, but with only intuitive math it should be easy to see that any method that creates a scattering of results and reports only the “best” one will bias the results in the direction of the dishonest author’s political preferences.
So, to summarize, we do not merely have a case where the authors made dramatic claims that were not supported by the reported results. Rather, we basically have no results, since they are meaningless as presented and highly suspect also. The only thing that we know for sure is that the methodology was never designed to produce useful information, or at least not useful at more than the pilot “let’s see if we can even do these lab measurements before designing some useful research that uses them” level. This means that the most fundamental lie in all of this is not the bald political assertions about there being “harm” or that you might as well smoke which were not supported by the science; rather, it is the implicit claim that any science at all was being reported.