Peer review in “public health” — Tobacco Control journal own-goal edition

by Carl V Phillips

Clive Bates prods me to write something about this editorial in the journal/political magazine/comic book, Tobacco Control, by Editor-in-Chief Ruth Malone, honoring their “top reviewers”. (Oh, wait, it is a British publishing house, so that should be: “honouring their toup reviewers”.) You can view it yourself, because it is open access, unlike their regular articles which they hide behind a paywall to inhibit real peer review (very few libraries subscribe to Tobacco Control, to their great credit). They really should have hidden this one from scrutiny too.

For those not familiar, UCSF denizen Malone is not so much the aggressive, calculating liar that some of her colleagues are. The word that comes to mind for her is “hapless”. She is is notorious on Twitter for endorsing Islamic State’s beheading-based anti-tobacco policies and gloating over the deaths of famous smokers, apparently without really understanding that was what she did. She is what you would get if one of the well-meaning and enthusiastic, but rather clueless, aging-hippie grandmothers found at anti-nuke rallies got hit by lightning and woke up to find herself the editor of a scientific journal (or, rather, a political magazine that apes a scientific journal).

Thus, I am sure she does not realize just how embarrassing this was.

The bit of the honuouring that really annoyed Clive was:

James Pankow: Jim’s toxicology expertise and his ability to help translate key ideas for our broad readership have proven so valuable for the journal. He rarely turns down a review, and re-reviews revised versions cheerfully. His wit is a plus!

Clive has been waging a campaign to get a different magazine (the NEJM) to retract a paper by Pankow that inaccurately claimed that e-cigarettes produce a dangerous level of formaldehyde, and that this makes vaping more hazardous than smoking (see: these posts). The paper has been criticized mainly on the grounds that Pankow et al. apparently used a set-up that subjected the e-cigarette liquid to an unrealistically high temperature, one that a properly-functioning high-quality e-cigarette would never reach and that produces an aerosol that is so acrid that no one would actually vape it. Maybe this is an example of that “wit” Malone gushes about.

But Pankow’s more fundamental failing in that fiasco was not his engineering, but his toxicology, the supposed expertise that Malone credits him with. Even if, hypothetically, his group had really discovered that this exposure happens under realistic conditions, everything that followed was toxicologically illiterate. The authors communicated the message — repeated in the pop media headlines — that because this one chemical was present (again, hypothetically) at a higher level than in cigarettes, vaping is thus more harmful than smoking. They failed to make clear that the risk from this hypothetical exposure would still be small (it is not what makes cigarettes harmful, obviously, so the fact it exceeds the levels produced by cigarettes is not informative or interesting). Instead they communicated that it will give vapers cancer. Moreover, they based the comparison on an assumption they buried, that someone is vaping copious quantities of overheated liquid, without establishing if this ever really happens.

So apparently Tobacco Control‘s best toxicology reviewer does not understand the importance of dose, the fact that exposure and disease are not the same thing, the fact that comparing on isolated chemical across exposures is not useful, nor that if you cook something too hot, it gets yucky. Sounds about right.

However, even though this was Clive’s main criticism of Malone’s list, I would say that it ranks no higher than third on the list of why this list of hunours was an “own goal” (Clive’s words). At least Pankow and company did research that legitimately showed something — that the chemical profile of vapor changes in a particular unfortunate way if you get it way too hot, so try to avoid making that mistake — even though they apparently did not understand their own results. I am not sure anything Kelvin Choi has ever done was even that useful.

Kelvin Choi: E-cigarette research manuscripts have become more frequent submissions as these products emerge as a public health challenge. Kelvin’s reviews of such manuscripts were described by a senior editor as always being timely, objective and constructive—no small challenge in a topic area that is fraught with controversy.

So someone who is an aggressive crusader against e-cigarettes is “objective and constructive”, huh? Well, I suppose that in spite of his obvious political views, which permeate his “research”, it is possible that when writing an anonymous journal review that will never be subject to any critical assessment he has the integrity to… hahaha — sorry, I just can’t finish that sentence.

And, of course, this topic area is “fraught with controversy” only in the same sense that the history of life on Earth is fraught with controversy. For the latter, you have all the real scientists in general agreement and trying to work out the details, with the “controversy” coming from a disturbingly large number of adherents to some fictional stories, who believe — regardless of the science — that everything was concocted sometime in recent memory. (Hmm, I might be mashing up the modern cultural impacts of the Hebrew creation myth with the creation of Scientology. Anyway, you get the idea.) In the present case, the “controversy” has a similar origin: a bunch of narrow-minded “believers” (like Choi and presumably most of the other honuuries) ignore or lie about the science when it does not fit their mythology.

What I would say was the most telling and embarrassing of “hounurs”, however, is not about the individual, but the role:

Pascal Diethelm: Pascal is another workhorse reviewer whose keen eye for detail, savvy grasp of the political and policy implications of studies, and thoughtful, considerate comments have helped many an author avoid pitfalls and publish successfully. He actually checks the reference list!

Now it does not surprise me that Tobacco Control only has one reviewer who checks the reference list. They apparently do not have any who check to see if the conclusions follow from the data. But I am not entirely sure why you would want to publicize that. (As I said: hapless.)

But the really telling bit here is about political and policy implications. Basically every “policy implications” statement in a Tobacco Control research report could be prefaced with, “Even though we have not conducted any policy analysis and have no idea what actually works in the real world, and even though our conclusions do not really follow from our results and we certainly have no idea if they are supported by other evidence, we are going to use our little research project as an excuse to write unsupported declarations about our personal inexpert policy preferences, which are….” It would be bad enough if the journal just admitted that they let these through without review. But in this paragraph, Malone is suggesting that they actively engage with the nonsense before publishing it.

And that was not the only example of that particular admission:

Cynthia Callard: Cynthia has a great eye for making sure researchers consider how their work may apply (or not) to tobacco control efforts in low-income countries, and for helping authors consider the larger social and policy contexts within which their work matters. Her reviews are incisive and offer creative ways to address manuscript weaknesses.

That sounds a lot more like the job of a press office than of a scientific reviewer. Which, again, sounds about right for Tobacco Control.

One reasonably respectable researcher made the list. I expect he wishes he didn’t:

Frank Chaloupka: Frank is a true workhorse. In addition to providing initial screening on economics-related manuscripts, he takes on a huge regular reviewing load and returns prompt, critical, objective and useful reviews on economic topics that can be hard for the journal’s multidisciplinary readership to interpret.

I really would not want “personally responsible for the quality of the economic analysis published in Tobacco Control” to be part of my bio.

Reading these and the rest of the huounors, you would notice that not a single one of them credits the reviewers with bringing research methodology or analytic skills beyond what they learned in their intro classes in school. There are several mentions of copyediting skills and correcting authors’ weak English, but no mention of correcting their statistical methods or logic. Of course, given the quality of methods and logic published in Tobacco Control, this would be another case of damning someone by crediting them with it.

Malone goes on to list three characteristics that she thinks define a good review. Two are, “The comments are phrased in ways that are constructive,” and “The review is submitted on time.” Meh. The third is, “Comments consider methods, structure of the paper and its importance to the field.” I am not sure which is sadder: the suggestion that actually analyzing the content is no more important than tone or meeting arbitrary deadlines, or that analyzing the methods and structure of the paper is considered a characteristic of a particularly good review for Tobacco Control rather than just being a minimum standard for something to be counted as a review at all. She could have written that a good review should not just be picture of a cat — it is not like that would be any less “goes without saying”.

On the other hand, in fairness to Malone, it is probably a rare reviewer for Tobacco Control that considers the methods of the research, and even rarer that someone finds the glaring flaws in the analysis (which she does not mention). Recall our analysis of the reviews provided to a generally more honest journal; for most of the journal reviews, it would indeed have been a big improvement if they had followed this advice.

Malone refers to journal reviewers as “generally invisible” and “unheralded”, which is true. But given the choice between that and being “credited” with the quality of the content in Tobacco Control (or, indeed, most any “public health” journal), invisibility would be the wise choice.