“Reason(s) you vape” questions on surveys are generally stupid

by Carl V Phillips

Another single-thought impulse post. I just saw a flurry of tweets about the evidence for the importance of flavors, based on survey responses. These surveys ask vapers to rank or score their reasons for vaping or what they like about vaping. I was reminded, once again, of just how bad survey research skills are in public health.

A survey can legitimately ask a question like “does having characteristic X make you more favorably disposed to do/like/vote Y?” It can even — more tenuously — ask how much so. What it cannot do is figure out what Y’s most important characteristics are. It cannot even rank them.

Why not?

Consider a version of it for a political candidate. It is reasonable to ask questions like “Does Senator Warren’s success with the Consumer Financial Protection Bureau make you more likely to vote for her for president? Answers: Yes, No, Her what now?” or “As a Warren supporter, how large a role did her CFPB work play in your support? Answers: None, Very Little,…, Lots.” If you have several questions like the latter, you can even rank the scores. But the question “which of these are the most important reasons you support her….”, followed by interpreting the answers as defining all that is important, is nonsense.

For me, the answer to that includes “she is not Thanos in drag” and “there is no indication that she would launch an invasion of Belgium.” You can probably think of some others to add there. Apply this to a consumer product: “What are your most important reason for choosing a Subaru?” Among my favorites are “they do not burst into flame when you start the engine” and “there is no record of one ever acting like Stephen King’s Christine.”

Bear with me — I am not just being silly. It is basic economics that a product or other good has a variety of characteristics, and you value it based on all of those. Some of those characteristics are the absence of fatal flaws, like I just quipped. This observation alone is sufficient to show the absurdity of questions about “the most important characteristic of e-cigarettes to you.” Not having any of a long list of potential fatal flaws would top the list, but does not even make the list.

But that is just the start of the problem. Without a range of quantification (which, notice, was unnecessary in the Warren-CFPB question), even questions about “how important is this to you” are utter nonsense:

“Is the gas mileage that Subarus get important to you?”

“Nah, not much.”

“So you would still buy one if it got 12 miles per gallon?”

“What? No! I was assuming you were just talking about it being a bit better or worse than other vehicles in its class.”

Similarly, a vaper who says “flavors are not very important to me” is not saying he would still vape if it tasted like chewing up an aspirin. If he says price is not important, he is not saying he would still vape at ten times the cost, or even double.

A survey is on solid ground with questions like “Do you prefer X types of flavors?” or even “Would you stop vaping if they only flavors available were…?” (However, the answers to the latter would need to be recognized as cheap talk, wherein most subjects probably have not thought through the answer carefully enough to really predict their behavior.) But the questions that get asked are complete fails, and the interpretations of the answers are worse.

They fail most completely in the rankings we see of what are the most important characteristics. These particularly include the subset of those that say things like “only 10% of adult vapers say the flavors they like are the most important characteristic” so banning flavor categories will not have much effect on them. First, there is the “chewing an aspirin” or gas mileage problem: The subjects are assuming only some minor possible variation in the status quo when they answer. Each subject is making a different assumption. This alone renders the responses largely meaningless.

Second, and closely related, just because something is low ranked does not mean a big enough departure from the status quo would not change your mind or behavior. (“How important to you is the candidate’s stand on abortion?” “Not very.” “So your opinion is not affected by her position is that it should be mandatory?” “Wait, what?”) Every characteristic of vapes that is asked about — flavor, price, health risks, convenience, similarity to smoking, etc. — could be changed enough to cause any vaper to quit.

Third, and most fundamentally, every ranking from one of these failed surveys is based on subjects’ unknown and inconsistent assumptions about what departures from the status quo are being considered. Someone who indicates she is relatively unconcerned with a car’s “reliability” is obviously not including the “burst into flames” scenario in what it means to be less reliable. But is she saying “I don’t much care if it has to go to the repair shop once a year” or “…once every two years” or “…twice a year”? We don’t know. Worse, it would be obvious nonsense to say “she cares more about style than reliability, but oh by the way, she would never buy a car that had to go in for repairs twice a year.” Yet something like that is exactly what is always being claimed by those writing or citing these awful surveys of vapers.

In short, every interpretation of these surveys I have ever seen is utter nonsense, and whoever wrote these survey questions should get out of the research business.

(I cannot decide if I am being a bit hyperbolic with that, or if I stand by it literally.)