“Reason(s) you vape” questions on surveys are generally stupid

by Carl V Phillips

Another single-thought impulse post. I just saw a flurry of tweets about the evidence for the importance of flavors, based on survey responses. These surveys ask vapers to rank or score their reasons for vaping or what they like about vaping. I was reminded, once again, of just how bad survey research skills are in public health.

A survey can legitimately ask a question like “does having characteristic X make you more favorably disposed to do/like/vote Y?” It can even — more tenuously — ask how much so. What it cannot do is figure out what Y’s most important characteristics are. It cannot even rank them.

Why not?

Consider a version of it for a political candidate. It is reasonable to ask questions like “Does Senator Warren’s success with the Consumer Financial Protection Bureau make you more likely to vote for her for president? Answers: Yes, No, Her what now?” or “As a Warren supporter, how large a role did her CFPB work play in your support? Answers: None, Very Little,…, Lots.” If you have several questions like the latter, you can even rank the scores. But the question “which of these are the most important reasons you support her….”, followed by interpreting the answers as defining all that is important, is nonsense.

For me, the answer to that includes “she is not Thanos in drag” and “there is no indication that she would launch an invasion of Belgium.” You can probably think of some others to add there. Apply this to a consumer product: “What are your most important reason for choosing a Subaru?” Among my favorites are “they do not burst into flame when you start the engine” and “there is no record of one ever acting like Stephen King’s Christine.”

Bear with me — I am not just being silly. It is basic economics that a product or other good has a variety of characteristics, and you value it based on all of those. Some of those characteristics are the absence of fatal flaws, like I just quipped. This observation alone is sufficient to show the absurdity of questions about “the most important characteristic of e-cigarettes to you.” Not having any of a long list of potential fatal flaws would top the list, but does not even make the list.

But that is just the start of the problem. Without a range of quantification (which, notice, was unnecessary in the Warren-CFPB question), even questions about “how important is this to you” are utter nonsense:

“Is the gas mileage that Subarus get important to you?”

“Nah, not much.”

“So you would still buy one if it got 12 miles per gallon?”

“What? No! I was assuming you were just talking about it being a bit better or worse than other vehicles in its class.”

Similarly, a vaper who says “flavors are not very important to me” is not saying he would still vape if it tasted like chewing up an aspirin. If he says price is not important, he is not saying he would still vape at ten times the cost, or even double.

A survey is on solid ground with questions like “Do you prefer X types of flavors?” or even “Would you stop vaping if they only flavors available were…?” (However, the answers to the latter would need to be recognized as cheap talk, wherein most subjects probably have not thought through the answer carefully enough to really predict their behavior.) But the questions that get asked are complete fails, and the interpretations of the answers are worse.

They fail most completely in the rankings we see of what are the most important characteristics. These particularly include the subset of those that say things like “only 10% of adult vapers say the flavors they like are the most important characteristic” so banning flavor categories will not have much effect on them. First, there is the “chewing an aspirin” or gas mileage problem: The subjects are assuming only some minor possible variation in the status quo when they answer. Each subject is making a different assumption. This alone renders the responses largely meaningless.

Second, and closely related, just because something is low ranked does not mean a big enough departure from the status quo would not change your mind or behavior. (“How important to you is the candidate’s stand on abortion?” “Not very.” “So your opinion is not affected by her position is that it should be mandatory?” “Wait, what?”) Every characteristic of vapes that is asked about — flavor, price, health risks, convenience, similarity to smoking, etc. — could be changed enough to cause any vaper to quit.

Third, and most fundamentally, every ranking from one of these failed surveys is based on subjects’ unknown and inconsistent assumptions about what departures from the status quo are being considered. Someone who indicates she is relatively unconcerned with a car’s “reliability” is obviously not including the “burst into flames” scenario in what it means to be less reliable. But is she saying “I don’t much care if it has to go to the repair shop once a year” or “…once every two years” or “…twice a year”? We don’t know. Worse, it would be obvious nonsense to say “she cares more about style than reliability, but oh by the way, she would never buy a car that had to go in for repairs twice a year.” Yet something like that is exactly what is always being claimed by those writing or citing these awful surveys of vapers.

In short, every interpretation of these surveys I have ever seen is utter nonsense, and whoever wrote these survey questions should get out of the research business.

(I cannot decide if I am being a bit hyperbolic with that, or if I stand by it literally.)

4 responses to ““Reason(s) you vape” questions on surveys are generally stupid

  1. Good post.
    Couldn’t the questions in any given survey be manipulated to get the outcome that the author/s wanted in the first place? Of course the answers would be open to interpretation by the author/s.

    • Carl V Phillips

      Yes, certainly. That is always the case, which is why you should never trust a report from a survey in which they do not report exactly what they asked (and what they asked right before it — lead-in questions are a good way to manipulate).

      But in this case we have people who think they are actually learning something from the questions when really they are not. The questions (and interpretations of the answers) fail not because someone is gaming the results (though that may be happening also). They fail because the answers to the questions are completely uninterpretable (and thus no one who knew what they are doing would design such questions).

  2. Thank you Carl, once again illuminating the difference between a loose associative reaction vs what measurable outcomes are. In this ‘field’ it seems the two become almost wholly interchangeable which has bothered a lot of us.

    As you point out, preference is a highly variable factor. What I remember of a very rudimentary discussion on the subject is that there have to be some end-points to even begin to frame the result, as preference is subjective. Even setting end-points on the scale color the result. Prevalence on the other hand, can be a more objective measure.

    A problem I’ve seen is the substitution of one for the other, vis: If they take away fruit/candy flavors (the most prevalent) then a large percentage of vapers will switch away from vaping or will go to the black market. There is no good evidence for that conclusion – it merely attempts to subtract the prevalence from the whole in the assumption that the costs are cetaris paribus. We know from other work (your’s included) that the speculative result is that they won’t switch away, but rather change to a black market source. Even then, we can’t be certain this will actually occur as the economic cost of obtaining product through illicit means hasn’t been quantified. [How willing are you to switch to the black market if it’s a misdemeanor fine of $100 vs if it’s a felony with 10yr prison sentence?]

    Perhaps a Sunday Science Lesson on the basics of how this all works, when done whell, and what is being passed off as “science” is in order?

    • Carl V Phillips

      I have never really pulled together all my thoughts on the junk surveys and even junkier interpretation. Maybe that would be good.

      Your main thought is one that I have written about before, that I summarize as “favorite does not equal necessary condition.” It is obviously true, as you note. There is *some* version of every product that a give individual likes best. That obviously does not mean in a world where that version did not exist, they would not consume the product at all. Consider the counterfactual: There is some other version of the product that *could* exist, but does not, that some people would like better than their current favorite. Yet they are still consuming. Yet so much gets interpreted that way (kind of an opposite-ish version of the error I was focused on).

      As I noted in the post, I agree with your point about cheap talk, and have made sure to point it out when I wrote up my work. Not only does someone not know the parameters of the hypothetical alternative choice — as with “what will the black market really be like?” — but they do not really know their preferences all that well before trying out the options they have never tried.

Leave a Reply to rbnye Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s