There is never no evidence (part 2)

posted by Carl V Phillips

This post continues the general discussion of how claims that there is “no evidence” to evaluate a particular claim are always nonsense from a scientific perspective.  It picks up directly from Part 1, so if you have not read that, you might want to.

Of the three scenarios presented in Part 1, it should be immediately evident that they constitute a randomized clinical trial or RCT (the second), a typical observational epidemiologic study (the third), and a series of what might be called “adverse event reports”, “case studies”, or “anecdotes” depending on how someone was trying to spin the result (the first).  Many people who speak the language of science, but do not actually understand science, would insist that the RCT necessarily provides the best evidence; I will come back to that.  They would also insist that the “anecdotes” do not provide any useful information because “anecdotes”…, well…, um, because they just don’t!  If forced to try to explain why, they would not be able to, because they do not really understand what they are talking about.

If they did, they would realize that another descriptor for the series of adverse event reports is “a collection of very compelling experiments”.  Recall the (real life) example from Part 1:

Thousands of people living near electric generating wind turbines have reported experiencing a particular pattern of serious health problems that began when the turbines started operating.  A large portion of those reports describe how the problems abate when someone spends days away from home and reverse when they move away.  The conclusion is that the wind turbines are causing the health problems.

Before dissecting that, consider how this is very similar to a familiar observation in the context of THR with the (also true) variation on the theme:

Thousands of former smokers have reported that they quit by switching to e-cigarettes or smokeless tobacco, and have published their stories.  A large portion of those reports describe how the former smokers tried many of the officially sanctioned (sometimes called “proven”) methods for quitting but always returned to smoking and were generally quite unhappy until they did, and how they are very happy using the alternative.  The conclusion is that switching to low-risk alternatives is an effective way for many people to quit smoking.

A huge portion of all scientific knowledge comes not from formalized studies and statistical analysis, but from simple experiments that address exactly what we want to know.  People who naively think that RCTs are always the best source of information have sort of learned this lesson, but in a perverted form that actually reduces their understanding of science.  They think a complicated stylized experiment that addresses some variation of what we really want to know is always best, whereas a more useful, less formalized experiment tells us nothing.

In both of the above scenarios — wind turbines and THR — people have performed experiments on themselves, experiments that address exactly the question that is being answered:  Is spending days and nights close to the turbines causing these diseases?  Can I successfully quit smoking if I use a low-risk alternative instead?

This study method is no different from the most basic study method familiar to all of us in our everyday lives.  For example, if you think that a particular food gives you stomach pain, you note whether that pain begins after eating the food; you then avoid eating the food for a while and see if that pain is absent, and then try it again to see if the pain returns.  It is obvious how to translate the resulting observations into a causal conclusion about the food and the pain.  For an even simpler illustration, our usual study design to figure out if a particular switch turns on a particular light is to flip the switch – nothing more complicated is needed.  If there was any doubt about the result, a few more flips of the switch provide overwhelming evidence.

Careful readers might have noticed that the wind turbine experiments provide a bit more data in one way, with the same people switching back and forth.  This contrasts with a smoker who successfully switches, and is obviously not likely to start smoking again to see if the e-cigarettes will help her quit again.  But the smokers have more data in the form of trying to quit using other methods, which allows the additional conclusion “…effective even for people for whom trying to quit other ways does not work…”.  Science does not have easy rules of thumb — you make use of what you have.

That is the main lesson here.

Anyone who believes that we do not have evidence of these products helping smokers quit is a complete idiot.  Anyone who asserts that as a scientific conclusion is a liar.  There is just no way to sugar coat that.  They can legitimately argue that the individual reports cannot tell us what portion of smokers will find switching to be an effective and welfare-enhancing way to quit.  They can even argue that, based on these reports alone, we cannot be sure that there is not some other tool for quitting that is better by some measure.  (We can make a good case that this is not true based on other information — it is pretty clear that just deciding to be abstinent by just stopping cold turkey or adopting THR are by far the most effective and most pleasant options for more people than other methods — but that does require other evidence.)

But it is totally obvious that a single one of these reports represents evidence that for someone, THR is a preferred option, and (depending on the details of the report) works when everything else seems to fail.  Multiple that by thousands of reports and we have a lot of evidence that this is true for many people.

Oh, and when I say “evidence”, that means scientific evidence.  Any evidence that lets us evaluate worldly phenomena is scientific evidence.  Scientific evidence is anything that lets us answer scientific questions.  It does not matter what form it takes if it addresses the questions, and it certainly does not matter where it is published.

…in the continuation, I will contrast this definitive evidence from simple individual reports with the near-uselessness of some complicated studies…