Peer review in “public health” — Tobacco Control journal own-goal edition

by Carl V Phillips

Clive Bates prods me to write something about this editorial in the journal/political magazine/comic book, Tobacco Control, by Editor-in-Chief Ruth Malone, honoring their “top reviewers”. (Oh, wait, it is a British publishing house, so that should be: “honouring their toup reviewers”.) You can view it yourself, because it is open access, unlike their regular articles which they hide behind a paywall to inhibit real peer review (very few libraries subscribe to Tobacco Control, to their great credit). They really should have hidden this one from scrutiny too.

For those not familiar, UCSF denizen Malone is not so much the aggressive, calculating liar that some of her colleagues are. The word that comes to mind for her is “hapless”. She is is notorious on Twitter for endorsing Islamic State’s beheading-based anti-tobacco policies and gloating over the deaths of famous smokers, apparently without really understanding that was what she did. She is what you would get if one of the well-meaning and enthusiastic, but rather clueless, aging-hippie grandmothers found at anti-nuke rallies got hit by lightning and woke up to find herself the editor of a scientific journal (or, rather, a political magazine that apes a scientific journal).

Thus, I am sure she does not realize just how embarrassing this was.

The bit of the honuouring that really annoyed Clive was:

James Pankow: Jim’s toxicology expertise and his ability to help translate key ideas for our broad readership have proven so valuable for the journal. He rarely turns down a review, and re-reviews revised versions cheerfully. His wit is a plus!

Clive has been waging a campaign to get a different magazine (the NEJM) to retract a paper by Pankow that inaccurately claimed that e-cigarettes produce a dangerous level of formaldehyde, and that this makes vaping more hazardous than smoking (see: these posts). The paper has been criticized mainly on the grounds that Pankow et al. apparently used a set-up that subjected the e-cigarette liquid to an unrealistically high temperature, one that a properly-functioning high-quality e-cigarette would never reach and that produces an aerosol that is so acrid that no one would actually vape it. Maybe this is an example of that “wit” Malone gushes about.

But Pankow’s more fundamental failing in that fiasco was not his engineering, but his toxicology, the supposed expertise that Malone credits him with. Even if, hypothetically, his group had really discovered that this exposure happens under realistic conditions, everything that followed was toxicologically illiterate. The authors communicated the message — repeated in the pop media headlines — that because this one chemical was present (again, hypothetically) at a higher level than in cigarettes, vaping is thus more harmful than smoking. They failed to make clear that the risk from this hypothetical exposure would still be small (it is not what makes cigarettes harmful, obviously, so the fact it exceeds the levels produced by cigarettes is not informative or interesting). Instead they communicated that it will give vapers cancer. Moreover, they based the comparison on an assumption they buried, that someone is vaping copious quantities of overheated liquid, without establishing if this ever really happens.

So apparently Tobacco Control‘s best toxicology reviewer does not understand the importance of dose, the fact that exposure and disease are not the same thing, the fact that comparing on isolated chemical across exposures is not useful, nor that if you cook something too hot, it gets yucky. Sounds about right.

However, even though this was Clive’s main criticism of Malone’s list, I would say that it ranks no higher than third on the list of why this list of hunours was an “own goal” (Clive’s words). At least Pankow and company did research that legitimately showed something — that the chemical profile of vapor changes in a particular unfortunate way if you get it way too hot, so try to avoid making that mistake — even though they apparently did not understand their own results. I am not sure anything Kelvin Choi has ever done was even that useful.

Kelvin Choi: E-cigarette research manuscripts have become more frequent submissions as these products emerge as a public health challenge. Kelvin’s reviews of such manuscripts were described by a senior editor as always being timely, objective and constructive—no small challenge in a topic area that is fraught with controversy.

So someone who is an aggressive crusader against e-cigarettes is “objective and constructive”, huh? Well, I suppose that in spite of his obvious political views, which permeate his “research”, it is possible that when writing an anonymous journal review that will never be subject to any critical assessment he has the integrity to… hahaha — sorry, I just can’t finish that sentence.

And, of course, this topic area is “fraught with controversy” only in the same sense that the history of life on Earth is fraught with controversy. For the latter, you have all the real scientists in general agreement and trying to work out the details, with the “controversy” coming from a disturbingly large number of adherents to some fictional stories, who believe — regardless of the science — that everything was concocted sometime in recent memory. (Hmm, I might be mashing up the modern cultural impacts of the Hebrew creation myth with the creation of Scientology. Anyway, you get the idea.) In the present case, the “controversy” has a similar origin: a bunch of narrow-minded “believers” (like Choi and presumably most of the other honuuries) ignore or lie about the science when it does not fit their mythology.

What I would say was the most telling and embarrassing of “hounurs”, however, is not about the individual, but the role:

Pascal Diethelm: Pascal is another workhorse reviewer whose keen eye for detail, savvy grasp of the political and policy implications of studies, and thoughtful, considerate comments have helped many an author avoid pitfalls and publish successfully. He actually checks the reference list!

Now it does not surprise me that Tobacco Control only has one reviewer who checks the reference list. They apparently do not have any who check to see if the conclusions follow from the data. But I am not entirely sure why you would want to publicize that. (As I said: hapless.)

But the really telling bit here is about political and policy implications. Basically every “policy implications” statement in a Tobacco Control research report could be prefaced with, “Even though we have not conducted any policy analysis and have no idea what actually works in the real world, and even though our conclusions do not really follow from our results and we certainly have no idea if they are supported by other evidence, we are going to use our little research project as an excuse to write unsupported declarations about our personal inexpert policy preferences, which are….” It would be bad enough if the journal just admitted that they let these through without review. But in this paragraph, Malone is suggesting that they actively engage with the nonsense before publishing it.

And that was not the only example of that particular admission:

Cynthia Callard: Cynthia has a great eye for making sure researchers consider how their work may apply (or not) to tobacco control efforts in low-income countries, and for helping authors consider the larger social and policy contexts within which their work matters. Her reviews are incisive and offer creative ways to address manuscript weaknesses.

That sounds a lot more like the job of a press office than of a scientific reviewer. Which, again, sounds about right for Tobacco Control.

One reasonably respectable researcher made the list. I expect he wishes he didn’t:

Frank Chaloupka: Frank is a true workhorse. In addition to providing initial screening on economics-related manuscripts, he takes on a huge regular reviewing load and returns prompt, critical, objective and useful reviews on economic topics that can be hard for the journal’s multidisciplinary readership to interpret.

I really would not want “personally responsible for the quality of the economic analysis published in Tobacco Control” to be part of my bio.

Reading these and the rest of the huounors, you would notice that not a single one of them credits the reviewers with bringing research methodology or analytic skills beyond what they learned in their intro classes in school. There are several mentions of copyediting skills and correcting authors’ weak English, but no mention of correcting their statistical methods or logic. Of course, given the quality of methods and logic published in Tobacco Control, this would be another case of damning someone by crediting them with it.

Malone goes on to list three characteristics that she thinks define a good review. Two are, “The comments are phrased in ways that are constructive,” and “The review is submitted on time.” Meh. The third is, “Comments consider methods, structure of the paper and its importance to the field.” I am not sure which is sadder: the suggestion that actually analyzing the content is no more important than tone or meeting arbitrary deadlines, or that analyzing the methods and structure of the paper is considered a characteristic of a particularly good review for Tobacco Control rather than just being a minimum standard for something to be counted as a review at all. She could have written that a good review should not just be picture of a cat — it is not like that would be any less “goes without saying”.

On the other hand, in fairness to Malone, it is probably a rare reviewer for Tobacco Control that considers the methods of the research, and even rarer that someone finds the glaring flaws in the analysis (which she does not mention). Recall our analysis of the reviews provided to a generally more honest journal; for most of the journal reviews, it would indeed have been a big improvement if they had followed this advice.

Malone refers to journal reviewers as “generally invisible” and “unheralded”, which is true. But given the choice between that and being “credited” with the quality of the content in Tobacco Control (or, indeed, most any “public health” journal), invisibility would be the wise choice.


6 responses to “Peer review in “public health” — Tobacco Control journal own-goal edition

  1. I would like to say that Tobacco Control is uniquely awful and that standards are maintained by less partisan publications but in my experience this is no longer the case. I was recently horrified by a very poor example of tobacco control junk science produced by activists posing as scientists that appeared in a Nature Publishing Group journal .

    I could understand how it happened, in that the authors had failed to mention something that makes their findings very probably unsafe, but having brought the matter to their attention, I expected the NPG editorial team to take my criticism seriously.

    The evidence I supplied warranted at least a detailed investigation of the paper and a request to the authors for proper sensitivity analysis, but I was brushed off with a feeble appeal to authority and an assertion that the findings were OK because the authors said so. Failure to publish any results to back up paragraphs of waffle, or, in fact, any data at all, is apparently no problem at NPG.

    I now have no faith whatsoever in NPG’s peer review or editorial processes. I note that the journal in question invites authors to suggest peers to review their papers. Pal review anyone?

    If we cannot trust the likes of NPG then perhaps we should accept that rationality is dead and look forward to a post enlightenment era in which authors can write pretty much what they like provided that it is “on message” Perhaps Tobacco Control is ahead of its time and science, honesty and honour are superfluous to requirements in modern public health publishing.

    • Carl V Phillips

      I have written extensively about peer review, of course. You can find a lot of it here or linked via the “peer review” tag. In particular, I have noted that it is clear that the journal gatekeeping model of peer review, which was briefly just fine in the 20th century, is now outmoded and cannot work for a variety of reasons. Only some form of open peer review — crowdsourcing and all that — has any hope of success in all but the narrowest fields.

      One implications of that is anyone who says “it is in a peer-reviewed journal, and therefore…” is utterly clueless. And if they are someone familiar with the process rather than a naive outside observer, they are also a liar. (Unless they finish the sentence “…it was written in the format that fits into a journal.”)

    • Carl V Phillips

      Oh, and you should publish — one way or another — your critique of that paper. That is the only way that crowdsourced review can be forced on the “public health” people, who are not interested in real reviews and thus will not be changing their approach voluntarily.

  2. Deep in the trenches are you Carl Phillips and Chris Oakly and you have earned the respect of those of us that value truth, honor and the discipline of applied science!

  3. The style of those blurbs is reminiscent of a half-hearted letter of “recommendation” written about an employee who was neither well-liked nor particularly good at their job.

    “Bob was a somewhat important member of our team here at Jimmy Joe’s Tractor Supply. He showed great intrepidity in arriving for work on time most days, his dedication to giving customers the correct amount of change was perfectly adequate, and he performed all his other duties with a single-minded commitment to sloth and mediocrity. I have no doubt he will make a valuable addition to your organization.”

  4. Pingback: Utter innumeracy: six impossible claims about tobacco most “public health” people believe before breakfast | Anti-THR Lies and related topics

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s