by Carl V Phillips
As I alluded to in the previous post, I am working on a project to review the quality of peer review of papers in the THR space. The first step in that is to write a review of the original submission to the journal. It will then be compared to what the journal’s reviewers actually wrote. It just so happens that a paper that came out today and that is currently dominating the chattering on the topic — “Associations between e-cigarette access and smoking and drinking behaviours in teenagers”, by Karen Hughes, Mark A Bellis, Katherine A Hardcastle, Philip McHale, Andrew Bennett, Robin Ireland, and Kate Pike — happens to fit the criteria for inclusion in our study. So I went ahead and wrote my review of it so I could share it before everyone moves on to a new shiny object du jour.
(Note to my coauthors on that project: BC, IB, and CB, you must stop reading now(!) until you have completed your review of the submitted version. Seriously. And do not follow any of the links either.)
This is a different approach to criticizing the paper from a typical blog post that just argues about the conclusions and press coverage. Those exist already too; you can find what Clive Bates wrote and linked out to here. But that approach does not plumb the depth of what is wrong with the paper itself, and thus the reasons it should never have been published as is, even apart from the political spinning of the results.
Also, our exercise is to review the initial submission, not the final version. However, in this case, basically none of the problems I noted had been corrected, not the most blatant errors (incorrect reporting of statistics) nor the most obvious uncontroversial omissions (the failure to report the year the data was collected in the abstract). Thus, basically the same scathing review applies to the final version of the paper as the submitted version.
The review itself is 11 pages of text — longer than the original paper itself. This is typical for good reviews of papers with serious problems. It takes a lot longer to explain a problem than it does to make the error. Also note that in anticipation of post it here, I explained a few points in more detail than I might have — not that this would not be of value in a real review too, but I probably would not normally bother to do it. In consideration of that length, I am not pasting the text, but rather it is here: Phillips review of Hughes et al (pdf). I suspect that people who believe in the journal peer review process think that most reviews somehow resemble what I wrote. But read on.
For those who do not want to read the whole thing (though I recommend it — I think it offers some insight that you would never have if you just read the fly-over criticisms of study conclusions) here are some summary observations. The indented material is copied directly from the review.
The most notable feature of both the abstract and the discussion, as well as the introduction, are the conclusions that in no way follow from the reported research. These should all be removed. The research shows some completely unsurprising of e-cigarette use, basically about what anyone would have guessed without seeing the data. The authors then claim that this somehow supports (vague, unspecified) interventions to prevent access to e-cigarettes. But even apart from the fact that the authors never analyze the implications of their policy proposals (this paper contains no policy analysis, and thus cannot be the basis for making any policy pronouncements) there is not even a link between their study and the conclusions. They do not in any way show that the results they observed represent a harm that needs to be solved, even apart from the fact that the results confirm what we would have predicted even without them. They do not so much as assert there is a harm to connect the dots between the statistics and the policy demands (not that mere assertion would be sufficient – but they do not even do that!). The two are wholly unrelated. Moreover, as discussed below, to the extent that these particular results address the policies they propose, they actually make quite a good case that the policies are likely to be harmful on net and/or useless.
…and later…
“In adults, e-cigarettes are typically used by smokers to help them reduce or quit tobacco use and uptake levels among non-smokers are thought to be very low (typically <1%).[21-24] Here, however, almost one in twenty (4.9%) teenagers who had never smoked conventional cigarettes reported having accessed e-cigarettes.” Here begins the political spinning of the results to portray them as showing basically the diametric opposite of what they really show. Those two sentences imply that the latter suggests e-cigarettes are not primarily being used for harm reduction by minors also. But to the extent that the poor data can address that question, it strongly suggests the exact opposite. Five percent of the non-smoking subjects had tried an e-cigarette compared to 70% of smokers. The former number is low compared to the portion trying cigarettes, alcohol, cannabis, and quite possibly (in this demographic) hard drugs and unprotected sex too. As has been extensively noted elsewhere, and as is common knowledge, teenagers experiment with things, whereas adults are relatively set in their ways. Since it is a safe bet that most of that 5% were not regular users, there is nothing to see here.
What we do see from the data, however, and that the authors pointedly fail to note in their discussion, is that (a) e-cigarette use was overwhelming concentrated among those who could be using it for harm reduction and (b) for every measure of risky or rule-breaking behavior they had, there was an association with increased e-cigarette use. Point (a) supports the claim that e-cigarettes play the same role for teenagers that they do for adults. Even if they are not intentionally being used for harm reduction and in attempts to quit (though they might be – the survey ostentatiously fails to ask the useful questions on that point), they are serving that purpose via partial substitution and familiarize the subjects with a low-risk alternative that they might more easily switch to if they later concluded smoking was unwise. Point (b) suggests that e-cigarette trying (and thus presumably use) is part of a constellation of behaviors, most of which are far riskier and more harmful than e-cigarette use. To the extent that these behaviors are substitutes (and there is certainly no reason to believe they are complements, though they might have no interdependence) that is also harm reducing.
Moreover, those points run flatly contrary to the authors’ conclusions about implementing restrictions. If e-cigarettes’ role in the lives of teenagers is primarily for tobacco harm reduction, and incidentally reducing harm in other cases, as this data tends to support, then it might be harmful to restrict access. Since e-cigarette use is concentrated among teenagers who are already regularly illicitly smoking and drinking, as the data clearly shows, then it is not clear why they expect imposing the same status restrictions on e-cigarettes would matter. (Indeed, they acknowledge this in their discussion even as they ignore its implication: “teenagers who access e-cigarettes are already familiar with strategies to bypass age legislation on restricted products.”) This is not to say that restrictions are not a good idea; that is a much larger question that requires analyzing points far beyond the present study. The point is that to the extent the present study provides input for the debate, it tends to argue that restrictions are possibly harmful and largely pointless.
Since the authors’ obvious political preferences suggest that they will refuse to draw these obvious conclusions from their results, they should instead just eliminate the political discussion. It is inappropriate as it appears anyway, since it is full of unstated and unanalyzed premises (e.g., that there is some viable restriction that will actually reduce ease of access). The results of the study should just be reported as what they are (after the extensive corrections noted above), without the authors misinforming the reader about how to interpret them.
This is the crux of why this paper is harmful to the world, of course. But it is not the only problem. It would be possible for someone to do a reasonably decent study, report the results reasonably well, and still paste on some political conclusions that are in no way supported. Indeed, that probably describes about half of what is published in journals with “public health” in the title (with most of the rest not qualifying because of those “reasonably” clauses; they still have unsupported policy conclusions). Still this one is an outlier even among the “public health” literature.
I identify numerous issues with the survey itself and out-and-out errors in how the results were presented. These are errors that should have been corrected regardless of whether the authors wanted to write a thinly veiled political advocacy piece and regardless of whether the journal decided to let them do it.
Some of the blatant errors would have been trivial to correct, like the authors’ reference to confounding, which does not exist when you are not making causal claims, or the omission of some of their statistics from the tables. Others would have been impossible to correct, like the terrible survey questions this was based on and the fact that the sampling method made it basically a convenience sample, but should have been acknowledged and dealt with. Some were substantive bright-line errors, like performing inappropriate statistical tests and reporting non-results as if they meant something.
Apparently none of these were fixed, even the simple stupid errors whose correction would not have even interfered with the authors’ desire to write an activist broadside.
Can you guess why?
It is because those worshipped and fetishized reviews that the journal collected did not address any of the problems. You can find them here and here. The authors’ response to those reviewers that I reference is here.
The first reviewer, Ziyad Ben Taleb, a PhD candidate at Florida International University, said
The authors of this study have done a great job in conducting this research.
which is a rather odd assessment of a study that was based on a badly-asked question from a convenience sample, and that made numerous errors in reporting the research. Indeed, in his single page of comments he notes,
You have used the Tobacco Survey that covers schools in North West England. Is this representative for other teens in the rest of the country? How would it compare to South of England. I think the authors should discuss this issue.
He is apparently oblivious to the obvious fact that the survey is not even representative for northwest England, or even for the particular cities where it took place. This comment did result in a trivial improvement in the paper, with the added buried recognition, “Thus findings should not be considered representative of all 14-17 year olds in England or the North West region.” But if they know it is not representative of anyone, why exactly did the authors continue to make a big deal about 0.5% differences in some of their results?
He continues,
The authors stated that student for whom no marker of deprivation was available were excluded from analyses (n=1,253 cases). How would this will affect the internal validity of the study? Dose those 1,253 participant may have different characteristics than those selected? (The issue of selection bias). I believe this should be discussed as well. [all the grammar errors were in his original]
This is a point I also noted (in a somewhat different way). But then I went on to say that it really does not matter. The authors responded to the reviewer that they were adding, “Students that could not be assigned to a deprivation quintile were excluded from analysis and therefore represent additional potential bias in the final sample.” This did not actually respond to his question about what effects this might have had (which can be calculated), but the editors let them get away with this throw-away line that will be ignored by everyone rather than actually trying to address the problem.
As I noted in my review, because this is a convenience sample, there is no point in even talking about selection bias — not because there is none, but because it is so overwhelming that there is no point in even talking about it. You simply have to say that the results are representative of no one other than the respondents (as they did per the above) and then not try to extract inappropriate results given that (which they did not stop doing). But given that the authors, reviewers, and editors apparently did not understand that, the response should have been to try to quantify the bias, not say “oh, yeah, that is another problem” and then ignore it for purposes of their analysis.
The reviewer then catches one mis-description of a statistical method in the tables, which was corrected. I overlooked this because I was focused on more fundamental flaws (like the failure to even report what their statistical model was). I am honestly not sure whether or not I would have noticed that if this had been a good paper. This further points out the inadequacy of journal review: The chances of two readers of a paper catching even all the simple errors like that are small.
The reviewer then suggests making the paper even worse:
One of the interesting finding the this study find is that almost quarter of teenagers that had accessed e-cigarettes had also tried smoking conventional cigarettes but not liked them. The authors provided the explanation that ‘’it is likely that flavorings make e-cigarettes an attractive option to teenagers who would otherwise be put off conventional cigarettes by their taste’’. I believe that there is a room for alternative explanation, since we can’t establish the temporal relation, it’s also possible that teenagers have tried conventional cigarettes first and probably didn’t like the taste, then tried e-cigarettes, but perhaps, once they get hooked on nicotine they will initiate smoking again. Therefore, the reader should know that e-cigarettes would not be considered as a path away from cigarettes.
To their credit, the authors refused to take this suggestion and said that they were intentionally avoiding making gateway claims. (But, funny thing — when they put out their press release (below), they did make the gateway claim!) However, I pointed out about this passage that it is pure speculation that their study had no way to address, and so should have not been included at all.
His last comment (and I am reporting the entirety of his review) was:
What are the possibilities of information bias? Is it possible that the rates of access to e-cigarettes are even higher than what was reported, in case there was a social desirability bias in completing the survey?
The authors responded by adding, “Equally, as with all surveys of self-reported social behaviours, students’ may have under or over reported e-cigarette access, smoking and drinking behaviours due to factors including social desirability, poor recall or lack of knowledge.” But so what? Again, acknowledging an obvious potential problem does nothing to deal with it. The authors could have tried to quantify what is known about this, but the comment as written and the response to it were utterly vacuous.
Ben Taleb declares that he has no competing interests. However, BMC rules for declaring competing interest require, to the publisher’s credit, not just the silly reporting of funding sources for the study, but disclosure of any relevant personal or political biases. Ben Taleb is the author of this rather silly anti-THR diatribe (“Switching cigarettes with [sic] snus is like changing a flat tire for [sic] another flat tire.”), indicating a clear political bias against THR.
And yet that was the better of the two reviews. The other, by Grace Kong, an Associate Research Scientist in Psychiatry at Yale, is even more paltry. She too praised the paper despite its glaring problems in her half-page review.
She noted that the authors failed to explain in the methods how they determined ex-smoking status, and the authors added a sentence providing the question actually asked. Bizarrely, neither the reviewer nor the authors figured out that this problem generalized to a need to correct general failure to report what questions the survey actually asked (which I noted).
She called for sample sizes to be added to the tables, which was worth doing, but was really a non-issue because mostly they were already there (despite Kong suggesting otherwise) and there was enough information to know the missing ones well enough. This was clearly a case of a reviewer going down a checklist for simple things you can say about a paper to give the illusion you reviewed it. At least it was better, from the perspective of believing that she actually even read the paper, than the next comment:
The analyses section is missing the description of the results presented in tables 2- 4
To which the authors quite reasonably replied, “We think this may be a mistake – the findings in tables 2 to 4 are already described in the results section, lines 194 to 222.” Indeed, my comment was that these descriptions were largely a waste of text, since they mostly just repeated what was more easily read from the tables, and the parts that just repeated the tables should be removed.
And finally — and, yes, what I reported was literally her entire review — she said,
The authors discuss that the rate of light smokers having tried an e-cigarette is 67%, which is a rate higher than those observed in France (33%) and South Korea (37%). I would like to point out that a survey study conducted in Fall 2013 in the USA also found that close to 60% of those who had tried e-cigarettes have also tried cigarettes (Krishnan-Sarin et al., 2014, NTR). Krishan-Sarin et al.’s study along with the current study’s findings, conducted around similar times confirms that the co-use of rates may be rising.
I pointed out there were problems with this passage in my review also. Except I noted the real problem, that the authors absurdly claimed that the comparison to other studies represented a time trend when they were studying entirely different populations, and moreover (based on nothing at all) that the trend was caused by marketing. (I also pointed out the authors’ error in claiming that their question was broader than others. You will have to read my whole review to make sense of that.) That does not seem to have bothered Kong at all, even though she apparently read this part of the paper. The authors responded by adding, “However, findings are consistent with other recent studies reporting high levels of e-cigarette access among tobacco smokers (e.g. USA[20])…” And then they still finish the sentence by still making the same absurd claim about it being a time trend caused by marketing! I know that I sometimes write parody, but this is all true.
Kong also misrepresents that she has no conflict of interest, even though her job description seems to be all about tobacco control, including anti-ecig work, and she wrote a paper whose conclusion (which — surprise! — did not actually follow from the results) was “E-cigarette prevention efforts toward youth should include limiting e-cigarette flavors, communicating messages emphasizing the health risks of use, and changing social norms surrounding the use of e-cigarettes.” So obviously she had substantial conflict of interest she should have reported.
So, welcome to the real world of public health journal peer review: Start with a manuscript that is dominated by political conclusions that do not follow from the results, and is also chock full of clear scientific errors. Send it to two reviewers who share the authors’ disdain for THR (though they lie about that) and who clearly lack the scientific skills to spot even the glaring errors that could have been addressed without even changing the political rhetoric. Make a few utterly inconsequential changes. And publish it.
Then all that remains is for the journal to issue a press release about this travesty, touting political conclusions that go even further than those in the paper. The “public health” approach to science at work.
Tobacco harm reduction, anti-THR lies, and related topics 
This is appalling. Unfortunately, this sort of thing will continue.
The “public health” approach to science at work.”
I barely finished High School. Science was my least favorite subject and yet at this time in my golden years I am amazed that I can comprehend and grasp the total corruption dominating in this area of science, “tobacco control” and politics as it pertains to public health!
When will the logical truth be accepted by all sides❓
I met Robin Ireland yesterday in a radio studio, IMO a very blinkered man. (over all a pleasant chap though) He just can’t seem to see through his own BS.
Pingback: Vaping Research | warrior3995
Pingback: Peer review – are they really even trying anymore? | Anti-THR Lies and related topics
Pingback: New Phillips-Burstyn-Carter working paper on the failure of peer review in public health | Anti-THR Lies and related topics
Pingback: What is peer review really? (part 1) | Anti-THR Lies and related topics