Peer review of: Dunbar et al. (Rand Corp), Disentangling Within- and Between-Person Effects of Shared Risk Factors on E-cigarette and Cigarette Use Trajectories From Late Adolescence to Young Adulthood, Nicotine & Tobacco Research, 2018

by Carl V Phillips

For an explanation of what this post is, please see this brief footnote post.

The paper reviewed here is available at Sci-Hub. The paywalled link is here.

—–

The typical “gateway” paper consists of observing the exposure of whether subjects (typically teenagers) have, at baseline, engaged in a particular behavior (vaping, in this case), and then observing the association with an outcome behavior (in this case, smoking). There is also an even worse collection of papers that do not even assess the order of events and simply look at whether prevalent ever-exposed status is associated with prevalent smoking. All of these suffer from the obvious fatal problem that a positive association is inevitable because inclination to ever vape is associated with inclination to ever smoke. In a counterfactual world in which vapor products did not exist, someone who vaped in the real world would be more likely to smoke than average, and this would obviously not be caused by (nonexistent) vaping. In short, since a positive association is inevitable, regardless of whether the hypothesis “vaping causes smoking” is true, observing a positive association obviously tells us nothing about about the hypothesis.

The present paper attempts to improve upon the standard worthless analysis. This is a commendable goal, and there is information value in what was done (unlike most gateway papers). However, the contributed information is very modest and does not actually support the authors’ conclusions. In particular, they claim that their results support the gateway hypothesis, and that they do so in ways that the usual longitudinal studies do not. This is simply false.

Though a few dishonest actors in this space pretend otherwise, everyone knows that different people have different propensities to use a tobacco product, and the degree of propensity is going to be highly correlated across products, creating an obvious confounding problem. There are often covariates available that are associated with the latent propensity variable, and the more honest (but still fatally flawed) attempts at assessing gateway effects attend to them. However, the authors usually make the mistake common throughout health research: “We threw all the variables we happened to have into a single model, and therefore there must be no remaining confounding.”

The present authors go in a different direction and focus on how well the covariates they happen to have (mental health scores, other substance use, and the usual simple demographic information) predict the exposure. Their basic results show the usual expected patterns: Inclination to use one drug or product is associated with greater inclination to use another, as are what society judges to be inferior states of mental health.

The bells and whistles added by these authors consist of separate within- and between-person analyses. Between- vs within-subject analysis can do various things. But in this case there is really only a single somewhat interesting question they tease out, whether the variables (this particular set of variables that they happen to have!) predict whether someone starts smoking once you already know she started vaping. The answer is no, based on the data they happened to have (three waves from 2015 to 2017, recruited students attending school near Los Angeles, a total of 2039, mostly age 19 at the final wave) and the model they used.

This is interesting as a normal incremental bit of science. However, it suffers from the limits faced by all epidemiology. There are no constants in epidemiology, and this will probably be different for a different population (place and time) and also, critically in this case, for a different set of collected covariates. It is notable that the authors indicate no awareness of the fact that their result may not generalize, even though this is far more true for the topic at hand than on average: Tobacco use behavior varies hugely across cultures, and they studied it in a single population; a behavior that was first exploding during the study period and will, of course, not remain novel in the future).

But mildly interesting does not imply revelatory (despite reports that the academic tobacco controllers who fancy themselves real scientists are going gaga about it). It most certainly does not support the gateway hypothesis as the authors claim. The authors seem to think that their methods do something to overcome the problem of not being able to observe the latent “inclined to use (or increase use of over time) a tobacco product” variable. They do not. There is simply no reason whatsoever (and the authors present no argument that there is such a reason) to believe that these methods do not suffer from the exact same problem as the various other analysis that lack the particular bells and whistles.

The authors also seem to be oblivious to the difference between predictors and causes. In particular, regarding the observation that the covariates lose their predictive value about smoking once it is known someone vapes, they suggest that if this were not the case then trying to reduce cannabis use could reduce smoking, but since it is the case then discouraging vaping will reduce smoking. This is fatally flawed reasoning.

To fully understand the flaws in their reasoning, consider an analogy (not realistic, obviously, but illustrative): Is eating at a fast-food salad place a gateway to eating at the burger-and-fries place in the same neighborhood? As covariates we have measures of the transport routes someone uses (say, public transit routes or roads they commonly drive) and their job description. The covariates are pretty good predictors of between-person variation: People who travel routes that access that neighborhood are far more likely to go to the burger place than those who do not; people with jobs with wages associated with eating out at the fast food level are also more likely, especially if the jobs probably include a decent lunch break and might be located in that neighborhood. Now you consider people who have eaten at the salad place (the main exposure of interest) and observe how much more likely than average they are to later go to the burger place. And, lo, the covariates no longer help you predict that. Of course they do not. They have been screened (in the predictive logic sense of the word) by the salad data: You now know for sure that they are sometimes in the neighborhood and that they eat fast food. The covariates that roughly predict each of these are now uninformative, just as “has vaped” renders rough predictions about whether someone would ever use a tobacco product uninformative.

Does that somehow suggest that the salad place [analogously: vaping] is a gateway? Of course not. If it did not exist, they would still be in the neighborhood and in the market for fast food [still be inclined to use tobacco products]. Indeed, the salad place [vaping] is probably protective against eating at the burger place [smoking] (if it did not exist they would have gone somewhere else [consumed a different product] from the start). These are exactly the same problems with the standard gateway longitudinal analyses. Despite their rhetoric to the contrary, the authors have done nothing to improve on that.

In fact, what they have demonstrated is that (for this particular dataset), the latent variable “propensity to use any tobacco product” — the confounder that renders all gateway papers to date uninformative — is such a strong predictor that if you have a very strong proxy for it (having vaped) then all the covariates that supposedly control for confounding are uninformative. Contra to how the authors interpreted it, this observation tends to support the conclusion that those other variables have limited value as deconfounders and thus the main criticism of all the gateway claim literature is actually a bit stronger than it was before. That is, we now can be more confident in the existing belief (among experts) that the deconfounder variables are not actually very good at controlling for the confounder of interest.

This is a somewhat interesting result, and it would move the science forward incrementally if the authors had actually reported it accurately and authors in this space cared about improving their work.

Another mildly interesting result (again, with very limited implications due to the use of one set of measurements for one population) is that poorer mental health status is associated with smoking uptake but not vaping. The authors fail to note this among their conclusions, but this is plausibly indicative of a recognition that smoking is a self-destructive act, while vaping is not.

Many of the results are presented using test statistics and uninterpretable measures of association. This is not appropriate for a paper of this kind. It is not difficult to translate those statistics into a relative risk (or, better, absolute risk) measure that the reader can make sense of without doing their own calculations. This suggests the authors are not genuinely trying to communicate their results, but only their (erroneous) conclusions. Or perhaps they are just indulging in using interesting methods to crank out a lot of results, seemingly unaware that since these results do not extrapolate across populations all that well, overly precise details like this are pointless.

The methods are fairly well described, with references to question banks or explicit statements of what questions were asked. There appears to be no serious problems with misrepresenting important variables or simply not having good measures, as is common in other papers (e.g., only measuring “ever tried a single puff” and characterizing this as being a vaper).

The model fishing that is probably present in the paper is that typically found in economics papers, rather than epidemiology papers. That is, instead of playing with which variables and functional forms to use (which looks pretty clean) they used unnecessarily complicated models to examine a fairly simple relationship. (Any additional precision provided by the complicated models is just window-dressing, since the precision of estimates is not really a relevant issue here.) They undoubtedly tried various versions of these models, without acknowledging this fact, and reported only one. It seems likely these authors picked the model with the statistically “best” result, rather than the politically “best” result as is common in this space.

The Introduction suffers from the problem that conclusions from previous studies in the space are presented uncritically, even when they have obvious fatal flaws or are total junk. Indeed, elsewhere in this paper, the authors demonstrate sophistication that is not typical in public health research, and thus presumably know many of the prior papers are obvious junk. Yet the literature review still seems like it was written by an undergraduate, as is typical. This is a major problem, but otherwise the Introduction is far better than what is typical in the space (an amateurish politicized essay about the broad subject matter that usually just demonstrates the authors’ incompetence in the field). It actually introduces the reader to the relevant background and concepts for understanding the research, and includes almost nothing more.

The Discussion is a proper analysis of the results, not a tangential political rant. However, the content of it is fatally flawed for the reasons noted above.

The authors falsely declare they have no conflicts of interest. They work for a company that depends on U.S. government health agency grants, in particular for this and similar work, and those agencies have clearly indicated that they endorse gateway claims, and they make a practice of funding those who endorse them. This is a clear financial conflict of interest, even if the authors have no personal political preferences that also favor the conclusions they reached.