SRNT believes research should be replicated (when they don’t like the results)

by Carl V Phillips

My attention was called to this gem of an editorial, “Conflicts of Interest and Solicited Replication Attempts” by the Nicotine and Tobacco Research (NTR) Editor-in-Chief, Marcus Munafò. NTR is the journal of the Society for Research on Nicotine and Tobacco (SRNT) and the slightly more honest and scientifically sound of the anti-tobacco journals. This editorial offers a new and different reflection on just how out of touch with real science tobacco controllers are.

The crux is the following, which appears halfway through the text:

We are piloting a new scheme to solicit independent, direct replications of studies published in Nicotine & Tobacco Research which we feel are sufficiently important to warrant this, and where independent replication will be particularly valuable. Our hope is that the replication team can work closely, and in good faith with the original discovery team to ensure that materials and methods are as close to the original as possible.

If you are not laughing it is because you have been poisoned by seeing too much of what passes for science in public health. Replication of research is, of course, one of the most fundamental tools of scientific inquiry. The fact that a journal would feel the need to mention that replication might be welcome speaks volumes about the relationship between that field of inquiry and real science. In fairness to the author, those volumes are entirely accurate: Replication of research is basically never done in public health, not even in the apolitical subfields, much less the political “public health” subfields like tobacco control. Whatever someone concludes from one study is just (highly erroneously) treated as fact. Anyone who actually spends the time doing a replication will find that journals are not interested in publishing it because “it has already been done.”

To give the author more credit than his actual words deserve, it is true that the pseudo-replication that occurs in the field could be done in ways that better genuinely replicate previous research. What gets incorrectly called replication of results consists of fundamentally different studies of different populations, with the data analyzed differently. In epidemiology and any social science, the problem of different populations is sometimes intractable. Even if you use the exact same sampling method again, the calendar time has changed, which often matters, and this further means you need to study people who are a different age or from a different cohort than those originally studied. The data analysis part of the problem can be fixed, however. A long time ago, I suggested that health researchers studying a phenomenon that someone else has already studied should present the results of the same statistical analysis that previous authors used — as best as can be determined from the lousy methods reporting in the field. (They should feel free to also present the results of a different model they think is better.) This was in the context of not allowing every researcher to cherrypick their modeling choices to twist their data into a preferred result, as is the standard practice in the field. But it also serves to better replicate (or fail to replicate) previous results. No one does this in public health.

In further fairness to the author, he is not merely suggesting that replication studies are welcome, but soliciting them. Oh, wait, he is not actually suggesting that replication studies are welcome. He is only soliciting them. He is saying that particular replication studies are welcome, the ones that the journal solicits. So what are they soliciting?

We have identified a study that we believe meets these criteria of timeliness, potential importance, and the potential to be influenced by conflicts of interest, both financial and non-financial. This is the report by Shiffman and colleagues that e-cigarette flavors appeal more to adult smokers than to non-smoking teens, and that interest in flavors is low for both groups. The question of the role of flavors in e-cigarettes in promoting these products to young people, and in particular nonsmokers, is of central importance to the ongoing debate regarding what role, if any, e-cigarettes can play in reducing harms associated with tobacco use. This is a topic that elicits strong feelings (and therefore unconscious biases) on both sides of the debate. A strong, well-powered, independent, direct replication study, conducted in good faith, will hopefully bring clarity to this important question.

I am sure you are shocked to learn that the single study for which they are soliciting a replication (let’s be serious: soliciting a repudiation) is the one that threatens a sacred cow of tobacco control, that interesting flavors act as a Pied Piper for children. Should anyone try to replicate any of the studies out there that produce absurd conclusions about behavior or about the miraculous effects of some minor intervention? Nah. Should anyone replicate a study with enormous implications? Nah. Stick with this one.

You may recall that I pointed out that the real results of this study are rather less important than is widely implied. It showed only that the mere mention of flavor descriptors in a survey setting does not cause someone who is not inclined to use a product (in particular, teenagers who are not particularly interested in trying e-cigarettes) to declare that they are more interested in trying it. It does not address (it could not) the questions of whether actually being offered or trying a particular flavor could entice someone to vape, or whether the existence of particular flavors keeps someone vaping who would have otherwise stopped. It is a pretty trivial result, both in terms of its real implications and its content (no one in their right mind would have failed to predict the general results of such a survey). This study is really not a good candidate for carbon-copy replication; a more robust study that got closer to what is genuinely of interest and genuinely in question would be far better. But it is inconvenient for the mythology, so it is the one study in the annals of NTR that needs to be replicated. Go figure.

The kicker is the rest of that paragraph:

Those interested in conducting such a replication attempt should contact the Editor-in-Chief directly. We also welcome suggestions for other studies suitable for this mechanism.

Yes, that’s right. We can’t just have anyone go off and do a replication of a study. No, no one wants that. It has to be the right researchers. Also, please do not go replicating other studies without suggesting it first and getting approval.

So who are the “right” researchers. We can find the answer in the continuation of the first quoted paragraph:

This kind of adversarial collaboration, where groups with differing hypotheses or conflicts of interest work together to construct and implement a study that satisfies both groups, has been suggested by Kahnemann and Klein as a means by which contentious issues in science can be resolved.

Yes, only ideological adversaries — those who wish to show that flavor descriptors are Pied Pipers — need apply. Tobacco controllers assume that no one is actually interested in figuring out the truth (a mirror-image fallacy) so in their minds the only way to play it is like a courtroom trial: Assume that everyone is doing “science” because they are paid to argue a particular position, and hope that the truth is more convincing. Needless to say, courts of law are not exactly the best ways to resolve scientific questions.

It is worth noting that the cited Kahnemann and Klein paper — and indeed those authors’ larger contributions to questions of truth-seeking — is not about the trivial act of replicating a simple study. Their work is about seeking to resolve thoughtful disagreements, among people who understand all the available evidence, that result from competing heuristics and biases that even serious thinkers bring to the table. It is not about simplistic narrow study results, let alone allegations of intentional bias. I suspect that the concept of real thoughtful disagreements among genuine truth-seekers is such a foreign concept for tobacco control that this was missed completely.

Oh, and when I say “need apply”, I was not kidding. The paragraph continues:

We will review initial expressions of interest, and select the strongest to work with the original study team. The final protocol will then be peer reviewed before data collection begins. If this is approved, acceptance in principle will be offered, and we will publish the resulting full paper. This is similar to the Registered Reports format offered by some journals.

Yes, that is right, you can only replicate a study with expressed prior permission from the journal editor. If you just go off and do it on your own — because, you know, you just want to do science — you can assume it will be rejected by the journal. This is just like registering planned research — well except for the wee difference that any scientist who wants can register their research and analysis plans without seeking permission from the grandees.

Of course the obvious reason for this science theater is that some people (perhaps at the FDA?) got annoyed that researchers who are not in thrall to the tobacco control monopoly of funding (Schiffman et al. were commissioned by NJOY to do that study) did research and that NTR published it. This is evident in the first half of the editorial, which is all about conflict of interest (COI). Or rather, it is about tobacco control’s usual misunderstanding of COI, which they think means “anyone who has an interest that conflicts with our special interest political goals.”

There is little point in saying much about most of that content because it is the same old stuff. Blah…blah…”tobacco industry has a shameful history” (never mind that there is no such actor as “the tobacco industry” and every actual actor involved with that is long since retired or dead)…blah…blah. It is quite interesting to note this bit, however:

The tobacco research community occupies an interesting position; financial vested interests include those of both the tobacco industry (1) and the pharmaceutical industry (2). Unlike fields that study spontaneously occurring diseases, we have to pay attention to two major sources of financial interests.

Yes, let’s see… one… two… can anyone think of any others? Hmm…. Oh, wait, I have one: How about the enormous gravy train of anti-tobacco funding that pays for about 99% of the published research on the topic? Could it be that grants and salaries that are specifically earmarked for vilifying tobacco products and tobacco users — like, say, a hundred million dollars in FDA funding devoted to denigrating e-cigarettes and thus justifying FDA policies — might be a COI? Particularly given that anyone who dares say something contrary to the party line gets their funding cut off? Or could there be a problem that people getting such money need to justify their income by generating “evidence” that there is a huge intractable problem that is not getting better, and simultaneously generating “evidence” that what they are doing is generating amazing benefits?

Oh, wait, I forgot my previous observation. We are not talking about a conflicts with actual honest science. When tobacco controllers refer to “conflict of interest” the word “conflict” means “conflicts with my personal political interests.” Indeed, trying to do actual honest science is a COI in their view.

Also amusing is:

However, there is also a third source of conflicts of interest, which receives much less attention – the beliefs, preconceptions and pet theories of individual scientists (3).

I left in the reference number just to emphasize that the author thinks this is such a foreign and radical concept that he has to cite it to someone. Yeeeah. I mean who would believe that someone might skew his research just because he has a seething hate-filled desire to see tobacco products stricken from the face of the earth, and that anything that furthers that aim is justified regardless of the financial, social welfare, or ethical harms it does?

Whenever tobacco controllers ape the language of science, science becomes another one of their victims. It happens when they talk about conflict of interest, completely denigrating what the concept really means. The same thing happens to epidemiology, sociology, economics, and the other sciences they pretend to employ. With this editorial, they manage to add the scientific concept of replicating research to their list of victims.

Public health people, you can look forward to the creation of Centers for Inconvenient Study Replication center grants from NIH and FDA. Start writing your applications now.

[Update: After some conversations that followed from this post, I realized that I had understated just how silly it is to replicate this particular study. It is not just a witch hunt (though it is that too), but a stupid waste of resources. This is definitely a case where “more research is needed” means “different research on this point needs to be done — something other than this weak proxy for the real questions of interest — before we can say much.” There is no way a repeat of this study can tell us what we (or, rather, some people) really want to know about the effects of actual flavors (not mere words) in the real world (not mere cheap talk). In a politicized fight against science deniers, it is sometimes worth doing a simple study that produces an incredibly obvious and predictable result, simply because some people are absurdly claiming that the opposite is true. But why would you possibly want to do it twice? It is quite amusing that the people who want to repeat it, and get the same inevitable result again, are the very science deniers who don’t like the result.]