Tag Archives: methodology

by Carl V Phillips

Why am I writing a post under a heading that is so incredibly obvious?  Because in the world of the tobacco control industry, even incredibly obvious truths are often ignored.

Survey research that asks people what they have done or experienced is often the only source of scientific data that addresses those questions.  Also, when we are interested in people’s personal preferences or guesses about something, some sort of survey is often the only way to find out.  The problem comes in when someone who does not understand science — or whose job description includes pretending to pretend to not understand — says, “hey, we use survey data as scientific fact when studying behavior and exposures, and opinion polls look similar to behavior and exposure surveys, so it must be that the results of opinion polls can be used as scientific data.”  Um, yeah.

Surveys about opinions are, of course, evidence of what people think, which is interesting for answering some questions.  But those are questions about belief/knowledge/understanding/confusion, not about the physical world.  Some of you might recall that the whole “third hand smoke” scam traces back to a survey where random people with no expertise were actively tricked into saying they believe that it is a hazard.

There are methods of aggregating the opinions of people with some expertise to crowdsource a legitimate prediction about some event.  It only works with predictions, though, because it requires placing a bet on the outcome which are paid out when the outcome is determined.  This is how we determine the probability of a sports team winning a game and also has been used in some clever tools for predicting elections.  Those who respond to these surveys (other than with small self-entertainment bets that are not going to be big enough to affect what the crowd predicts) are self-selected people who think that they know enough to come out ahead on their bets, not just the average person on the street.  And, importantly, there is a punishment (losing the money you bet) for expressing an opinion that is uneducated, or that you know to be wrong — this is not cheap talk.

Contrast this with a recent “study” that used an opinion poll to “predict” the effects of plain packaging of cigarettes.  The “researchers” asked a handful of people in the tobacco control industry, presumably many of whom are directly invested in the plain packaging boondoggle, what will happen and reported the result as if it was a useful prediction.  Needless to say, the prediction of the effect of taking away brand logos by the people who have run out of useful things to suggest, was an absurdly large impact.

I would write more, but there is no way to usefully add to what Snowdon (who reported this story) already wrote about this, so give his very funny post a quick read.

The limitation of this survey is just the obvious point that the respondents are not only ridiculously biased, but they have absolutely no incentive to give an accurate prediction or to refrain from predicting if they lack confidence in their opinion.  The “researchers”, had they been interested in actually learning something, could have asked the respondents to place a bet, but did not.  Without a bet, there is no incentive to tell the truth because there are absolutely no penalties in the tobacco control industry for making incorrect predictions or scientific declarations that are clearly shown to be false.  It would be delightful to see ANTZ “researchers” and “experts” being held to account for lying and forced to do a Lance Armstrong, begging for forgiveness, promising (without credibility) to never do it again, and begging to not have to give up the hundreds of millions of dollars they swindled by lying.

Another recent example of dumb polling is more troublesome in its implications.  A media blast by the Schroeder Institute for Tobacco Research and Policy Studies (Steve Schroeder must just be so proud of the “research” that is coming out attached to his name) claimed that it is a good idea to mandate a lowering of the nicotine content of cigarettes because a majority of random Americans were tricked into saying they thought it was a good idea by a survey.  (For more details, see this article, which unlike the usual churnalism includes good analysis by Michael Siegel and others.)  It turns out that this “majority” consists of forty-something percent, but we do not expect basic numeracy from tobacco control, so I will just move on.

What does this survey really tell us?  It tells us that the tobacco control industry’s efforts to confuse people about the source of harms and benefits from smoking have been rather successful, though surprisingly not quite as successful as one might have guessed.  Obviously it tells us nothing about whether such a policy would actually be a good idea by any measure.  It does not even tell us whether people really have this belief in any meaningful sense, or if they merely decided to agree with the statement while blasting through a survey.  Polling people about something they have never given any serious thought is unlikely to provide useful information, even if they have no incentive to be dishonest and even if they might know something about it. Even a poll of people asking them how far they could drive given the gas currently in their car is well within their expertise to answer, but their answers would not be very accurate.   They would just give a snap answer without bothering to go check how much gas they have, let alone calculate their mileage.

It is obviously worse when the question is well beyond people’s expertise.  We know, after all, that a majority (a term I am using like the Schroeder Institute people do, to mean “at least a substantial minority”) also believe, without any scientifically defensible basis whatsoever, that: we should not worry about the future because the gods are going to end life on Earth within our lifetimes; that it is healthier to eat “organic” foods; that screening mammography provides a major health benefit; that Iraq posed a threat to the US in 2003; that current-tech wind turbines are an environmentally friendly way to generate electricity; and that cutting government spending in an economic depression characterized by zero-lower bound interest rates.

The problem is that for all but the first two of those, policies have conformed to the opinion of that majority (or “majority”) that is unmoored by the facts and the science.  What distinguishes the first two from the others?  For all of the others, the rich and powerful people profited by keeping people ignorant and getting them to believe, and thus support, something that is false.  While the tobacco control industry is not nearly as rich and powerful as those who have profited from the mammography, wind turbine, and Iraq War boondoggles, they differ only in degree, not in their willingness to foster ignorance to support their cause.  So there is little doubt that they will use the cultivated ignorance to further their agenda.

Reducing nicotine content reduces the benefit of smoking a cigarette while not reducing the harm.  The same would be true for adding some harmless but foul-tasting chemical to the cigarettes.  The main difference is that the latter option would almost certainly cause people to smoke less, while everything we know suggests that lowering nicotine content will cause people to smoke more.  If people had been polled about a proposal to add the foul chemical to cigarettes, it is likely that a smaller “majority”, maybe 25%, would support that too — because they basically favor prohibition, and that would be roughly equivalent.  But no one would mistake that for scientific evidence that the policy would have a positive impact on the world.  Whatever someone might think of the ethics of intentionally lowering the quality of cigarettes to discourage smoking, it seems that the dumbest possible way to do it is in a way that makes people want to smoke more.  Make no mistake, reducing nicotine is benefit reduction, not harm reduction.

[Aside: It is worth noting that while all the evidence suggests that substantially lowering the nicotine content would increase harm, this does not mean that substantially raising the nicotine content would reduce harm.  So much of smoking behavior is habitual (and many people seem to smoke enough that at the margin their nicotine receptors should already be saturated) that it is not entirely clear that higher nicotine would reduce total smoking, unless accompanied by some other change like making the cigarettes shorter (“same great nicotine with only half the stick”).  But at that point, why not just advocate replacing some of someones daily cigarettes with smoke-free nicotine?  That would have the advantage of encouraging a complete switch, as well as avoiding several obvious downsides.]

To sum up, there is a lot to be learned about perception, propaganda, and ethics from looking at results of surveys like this.  But there is nothing useful to be learned about science or science-related policy.  That finally circles my thinking back to a survey from 2003 that is a large part of why I was inclined to address these points.  It is still cited as if it actually represents an estimate of the risk of smoke-free alternatives, a practice I have repeatedly criticized but apparently never posted about.  But this is already enough for the day, so I will try to come back to that.