There is never no evidence (part 2)

posted by Carl V Phillips

This post continues the general discussion of how claims that there is “no evidence” to evaluate a particular claim are always nonsense from a scientific perspective.  It picks up directly from Part 1, so if you have not read that, you might want to.

Of the three scenarios presented in Part 1, it should be immediately evident that they constitute a randomized clinical trial or RCT (the second), a typical observational epidemiologic study (the third), and a series of what might be called “adverse event reports”, “case studies”, or “anecdotes” depending on how someone was trying to spin the result (the first).  Many people who speak the language of science, but do not actually understand science, would insist that the RCT necessarily provides the best evidence; I will come back to that.  They would also insist that the “anecdotes” do not provide any useful information because “anecdotes”…, well…, um, because they just don’t!  If forced to try to explain why, they would not be able to, because they do not really understand what they are talking about.

If they did, they would realize that another descriptor for the series of adverse event reports is “a collection of very compelling experiments”.  Recall the (real life) example from Part 1:

Thousands of people living near electric generating wind turbines have reported experiencing a particular pattern of serious health problems that began when the turbines started operating.  A large portion of those reports describe how the problems abate when someone spends days away from home and reverse when they move away.  The conclusion is that the wind turbines are causing the health problems.

Before dissecting that, consider how this is very similar to a familiar observation in the context of THR with the (also true) variation on the theme:

Thousands of former smokers have reported that they quit by switching to e-cigarettes or smokeless tobacco, and have published their stories.  A large portion of those reports describe how the former smokers tried many of the officially sanctioned (sometimes called “proven”) methods for quitting but always returned to smoking and were generally quite unhappy until they did, and how they are very happy using the alternative.  The conclusion is that switching to low-risk alternatives is an effective way for many people to quit smoking.

A huge portion of all scientific knowledge comes not from formalized studies and statistical analysis, but from simple experiments that address exactly what we want to know.  People who naively think that RCTs are always the best source of information have sort of learned this lesson, but in a perverted form that actually reduces their understanding of science.  They think a complicated stylized experiment that addresses some variation of what we really want to know is always best, whereas a more useful, less formalized experiment tells us nothing.

In both of the above scenarios — wind turbines and THR — people have performed experiments on themselves, experiments that address exactly the question that is being answered:  Is spending days and nights close to the turbines causing these diseases?  Can I successfully quit smoking if I use a low-risk alternative instead?

This study method is no different from the most basic study method familiar to all of us in our everyday lives.  For example, if you think that a particular food gives you stomach pain, you note whether that pain begins after eating the food; you then avoid eating the food for a while and see if that pain is absent, and then try it again to see if the pain returns.  It is obvious how to translate the resulting observations into a causal conclusion about the food and the pain.  For an even simpler illustration, our usual study design to figure out if a particular switch turns on a particular light is to flip the switch – nothing more complicated is needed.  If there was any doubt about the result, a few more flips of the switch provide overwhelming evidence.

Careful readers might have noticed that the wind turbine experiments provide a bit more data in one way, with the same people switching back and forth.  This contrasts with a smoker who successfully switches, and is obviously not likely to start smoking again to see if the e-cigarettes will help her quit again.  But the smokers have more data in the form of trying to quit using other methods, which allows the additional conclusion “…effective even for people for whom trying to quit other ways does not work…”.  Science does not have easy rules of thumb — you make use of what you have.

That is the main lesson here.

Anyone who believes that we do not have evidence of these products helping smokers quit is a complete idiot.  Anyone who asserts that as a scientific conclusion is a liar.  There is just no way to sugar coat that.  They can legitimately argue that the individual reports cannot tell us what portion of smokers will find switching to be an effective and welfare-enhancing way to quit.  They can even argue that, based on these reports alone, we cannot be sure that there is not some other tool for quitting that is better by some measure.  (We can make a good case that this is not true based on other information — it is pretty clear that just deciding to be abstinent by just stopping cold turkey or adopting THR are by far the most effective and most pleasant options for more people than other methods — but that does require other evidence.)

But it is totally obvious that a single one of these reports represents evidence that for someone, THR is a preferred option, and (depending on the details of the report) works when everything else seems to fail.  Multiple that by thousands of reports and we have a lot of evidence that this is true for many people.

Oh, and when I say “evidence”, that means scientific evidence.  Any evidence that lets us evaluate worldly phenomena is scientific evidence.  Scientific evidence is anything that lets us answer scientific questions.  It does not matter what form it takes if it addresses the questions, and it certainly does not matter where it is published.

…in the continuation, I will contrast this definitive evidence from simple individual reports with the near-uselessness of some complicated studies…

Advertisements

7 responses to “There is never no evidence (part 2)

  1. Pingback: There is never no evidence (part 1) | Anti-THR Lie of the Day

  2. Carl, while I agree with the general sense of what you’re saying, I think it’s also important to note some of the reasons *why* so-called “anecdotal evidence” can be quite questionable.

    1) Often when people use the term or when advocates use anecdotal evidence to make a case they are using just a FEW instances and simply assuming that either (a) They are enough to prove the point. Or (b) that there are many more out there that are unknown. Usually neither assumption is warranted by anything more than the faith of the advocate making the case.

    2) There are some very serious confounders built into anecdotal evidence. Two would be: (a) people who experience something odd are more likely to make the experience known than the far more many people who experience nothing odd and are quiet. And (b) People who’ve been psychologically conditioned toward expecting an experience due to something are more likely to actually feel that they’ve HAD that experience and ascribe it to that “cause.”

    In talking about the economic effects of smoking bans, several of the SmokersClub.com folks realized a number of years ago that while we didn’t have the money, skills, and data to produce formal economic studies that would have a good shot at being published that we DID have a WEALTH of “anecdotal data.” We spent months gathering it together from news articles and emails and eventually came up with a list of roughly 1,000 establishments hurt or closed by smoking bans. As I like to explain when presenting that list, “Anecdotal evidence is when Aunt Polly says that Billy told her that the guy down the street’s cousin felt a certain way after doing a certain thing. But if I have a THOUSAND people, ALL of who have said they felt that certain unusual way after doing that certain unusual thing, then that’s no longer just anecdotal evidence!” (even though, in a sense, it is.) See our listing at:

    http://www.smokersclub.com/banloss3.htm

    and bear in mind that that listing was basically compiled over just several months by about four of us.

    Regarding such things as e-cigs helping people give up regular cigarettes for the long term and Fire Safe Cigarettes causing people all sorts of problems, we’ve moved beyond the Aunt Polly anecdotal stage into the “real evidence that there’s SOMETHING to this!” stage.

    – MJM

    • Carl V Phillips

      I will explain a bit about when anecdotes fail to provide useful information as I continue the series. The problem is selection bias rather than confounding. Selection bias occurs when the story was told *because* of the co-occurrence of the events that were said to be linked, and that co-occurrence might not be very common. Confounding is when there really is a correlation (across the entire population, not just those reported), but it is not because E is causing D.

      The additional point, about there not really being an association, but people thinking there is, is a different matter. For many outcomes, like quitting smoking, it does not really matter — if someone thinks that something is helping him quit smoking, and he quits smoking as a result, then it really did help him quit smoking.

  3. Pingback: There is never no evidence (part 2) | Tobacco Harm Reduction | Scoop.it

  4. Thanks! Yes! You’ve explained it well! I kinda understand the guts, just get lost in the terminology a bit sometimes!

    :>
    MJM

  5. With you Carl on most of this message and glad you took the time to go through it in an analytical way. Though when you say, “Anyone who believes that we do not have evidence of these products helping smokers quit is a complete idiot. Anyone who asserts that as a scientific conclusion is a liar. There is just no way to sugar coat that.” – I find it better, even if it amounts to ‘sugar coating’ to speak to ideas rather than judging the character (or firmly attaching labels). So example of this would be to instead say, “Anyone who believes that we do not have evidence of these products helping smokers quit is engaging in idiocy. Anyone who asserts that as a scientific conclusion is lying. There is just no way to sugar coat that.” Perhaps a nitpicky point on my part, but one I think matters for sake of discussion.

    • The calling out of individuals, as actors, in this blog is quite intentional. I guess it is kind of like prosecuting shoplifters rather than just stopping them. It is costly and ugly to worry about the actor rather than just the act, but if we do not, then the behavior is effectively treated as acceptable if you can get away with it. I will write more about this in a post soon.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s