Public health publishing is fundamentally unserious: evidence from a single measure of area

by Carl V Phillips

Sometimes an error matters because of its effects. Sometime it matters because what it says about its causes.

I was late to this nice piece by Roberto Sussman (a guest post at Brad Rodu’s blog) that takes down a recent silly paper out of University of California about environmental deposition on surfaces resulting from vaping exhalate. They do not actually call it “third-hand vapor”, though they all but do so, explicitly likening it to the myths (which they endorse, of course) about “third-hand smoke”. For the analysis of the science, please read Roberto’s piece, because here I am just focusing on a single gaffe and its implications.

As background, note that this that this came from the supposedly respectable tobacco controllers at UC, including Benowitz and Talbot, not the utter loons in Glantz’s shop. It was published not in some random online journal, but in the supposedly respectable flagship journal of the tobacco control movement, BMJ’s Tobacco Control.

Reading Sussman’s piece, I came across this, which he quoted from the original paper:

“After 35 days in the field site, a cotton towel collected 4.571 micrograms of nicotine. If a toddler mouthed on 0.3 m²[squared meters] or about 1 squared feet of cotton fabric from suite #1, they [sic] would be exposed to 81.26 mg [micrograms] of nicotine”. 

Sussman’s post is analytic, but it was written as an essay and so I was reading it fairly casually. That is, I was not trying to actively check each bit of the math as I read it, as I would when reading a research report. But even a quick glance across that passage was enough for me to trip up and notice the error. A square meter is about ten square feet, and thus 0.3 m^2  is about 3 square feet. Sussman, who was reading the original paper carefully for purposes of criticizing it, of course also caught this error and noted it in his next paragraph.

In theory this affects the thesis of the paper, which is based on the premise of a toddler sucking out all the nicotine that has accumulated in a towel that has sat untouched in a vape shop for a month. (Yes, believe it or not, that is really the premise of the analysis.) So the error means that the magical vacuuming toddler is given credit for extracting 3 ft^2 worth of accumulation by sucking the heck out of a mere 1 ft^2 of fabric.

However, this is not one of those convenient errors that creates artifactual results that matter First, every bit of this scenario is obvious nonsense, as Sussman explains, and every step grossly exaggerates the real-world exposures. And, second, even with all that, the tripled quantity is still trivial. So it is not like was the common type of “error” from tobacco control research, one done intentionally to get the result the authors want. It merely changes the result from “a silly premise that despite its huge overstatement still only yields a trivial exposure” to “a silly premise that yields an exposure that is three times as high, but is still trivial.” It is obvious that the conclusions of the paper (“environmental hazard” — i.e., landlords should be pressured to not host vape shops) were in no way influenced by the results.

In addition, it is a pretty stupid intentional “error” to make. It is a bright-line error, which appears right there in the text, as if someone had written 2+2=5. The typical tobacco controller “errors” consist of such tricks as conveniently not mentioning that a crucial variable makes the entire result go away (which only very careful readers catch), or fishing for a model that produces the most politically favorable result and pretending it was the only version of the model ever run (which is easy to detect, but impossible to prove).

No, it is clear that this was a mere goof. Someone who is not so good with numbers was thinking “a meter is about three feet, so it must be that a m^2 is about three ft^2”. Oops.

But here’s the thing: Whoever was doing the calculations for the paper made that goof, but more significantly did not catch it on further passes through the material. In other words, no one ever thought carefully about the calculations. Then someone transferred the calculation notes into text of the paper without noticing the error at that point. The other authors of the paper (there were four total) reviewed the calculation and the paper without ever engaging their brains enough to notice the error, and let it go out the door. Or perhaps they never even reviewed the calculations they were signing-off on, and perhaps not even the paper.

Keep in mind that perhaps you, dear reader, might not notice this error on a quick read. Perhaps you did not even know that a m^2 is about 10 ft^2. But anyone who does science, and is burdened with the hassle of dealing with stupid non-SI American units of measure, knows stuff like this intuitively. As I said, I noticed it without even thinking about it, just like you would notice a misspelling even though you are not actively looking for mispelings as you read. Sussman noticed it, and he is a scientist who probably never sees mention of non-SI units in his work, and who lives in a normal country that uses SI units (i.e., “the metric system”) in everyday communication. It is apparent that none of the authors of the paper ever read it as carefully as he did.

The American authors, who need to be literate in translating from American units to scientific units, should have noticed it. It is a safe bet that if prompted, “there is an error in that sentence,” they would figure it out in a few seconds. So the point here is not that they do not know the units or how to do arithmetic, but that they did not pay enough attention to their own calculations to notice the simple error. They never really cared about the calculations, as evidenced by the conclusions that are not actually supported by the results.

They were not the only ones. The reviewers and editor(s) at BMJ Tobacco Control also did not read the paper carefully enough to catch the error. As I have noted at length on this page, journal peer-review in public health is approximately useless. A generalist copy editor would probably have caught it, but presumably BMJ TC does not employ one despite being hugely profitable.

This also means that no one other than the aforementioned seven or eight individuals read the paper carefully. Indeed, it is quite possible that no one else read the paper at all before it appeared in the journal. From the perspective of serious science, is actually the biggest problem in public health research evident here: not circulating a paper for comments before etching it in stone, but rather creating a “peer-reviewed journal article” out of what is effectively a superficially polished first-draft of a scientific analysis. Anyone who actually wants to get something right makes sure a lot of people read it critically before they commit to it.

Many errors in public health articles are a bit complicated, and pretty clearly happen because the authors and reviewers do not know enough science to know they were errors. Many others are pretty clearly intentional on the part of the authors, and signed off on by reviewers because they are incompetent, inattentive, and/or complicit in wanting to disseminate the disinformation. But a stupid error like this illustrates something different: Public health authors and journals are simply not even trying to do legitimate analysis.