Tag Archives: epidemiology methods

Let’s try to get our criticisms right, shall we? (More on the recent “vaping causes heart attack” study)

by Carl V Phillips

Sigh. We are supposed to be the honest and scientific ones in the tobacco wars. But we won’t be if we are not, well, scientific. Case in point are the criticisms of the recent paper with Glantz’s name on it that has been erroneously said to suggest that vaping doubles the risk of heart attack.

Incidentally, the meaningless statistic in the paper is a RR of 1.8, which is not double. Also, when the paper was originally written as a student class project (not by science students, mind you, but by medical students), that statistic was 1.4. That was when Glantz heard about it, managed to get the kids to put his name on the paper, and taught them how to better cook their numbers. That “contribution” has him being called the lead author.

The paper is junk science. So are most of the criticisms of it. If only someone with expertise in these methods had written a critique of it that people could look to. Oh, wait, here’s one in The Daily Vaper from February. That was based on a poster version of the paper, but as I noted in the article, “It has not yet appeared in a peer-reviewed journal, but it will, and the peer-review process will do nothing to correct the errors noted here.” I wish I could claim this was an impressive prediction, but it is about the same as predicting in February that the sun will rise in August.

You can go read that if you just want a quick criticism of the paper, and also look at the criticism on this page of some hilarious innumeracy Glantz piled on top of it. In the present post I am mostly criticizing the bad criticisms, though at the end I go into more depth about the flaws in the paper.

About half the critiques I have seen say something along the lines of “it was a cross-sectional study, and therefore it is impossible to know whether the heart attacks occurred before or after someone started vaping.” No. No no no no no. This is ludicrous.

Yes, the data was from a cross-sectional survey (the 2014 and 2016 waves of NHIS, mysteriously skipping 2015). And, yes, we do not know the relative timing (as discussed below). But “therefore it is impossible to know” (or other words along those lines)? Come on. A cross-sectional survey is perfectly capable of measuring the order of past events. Almost every single cross-sectional survey gives us a pretty good measure of, for example, whether someone’s political views were formed before or after the end of the Cold War. Wait! what kind of wizardry is this? How can such a thing be known if we do not have a cohort to follow? Oh, yeah, we ask them their age or what year they were born. Easy peasy.

Almost every statistic you see about average age of first doing something — a measure of the order in which events occurred (e.g., that currently more Americans become smokers after turning 18 than before, but most extant smokers started before they were 18) — is based on cross-sectional surveys that ask retrospective questions. It is perfectly easy to do a survey that asks heart attack victims the order in which events occurred. Indeed, any competent survey designed to investigate the relationship in question would ask current age, age of smoking initiation and quitting, age of vaping initiation and quitting, and age at the time of heart attack(s), ideally drilling down to whether smoking cessation was just before or just after the heart attack if they occurred the same year. We would then know a lot more than the mere order. But NHIS does not do that because, as I noted in the DV article, it is a mile wide and an inch deep. It is good for a lot of things, but useless for investigating this question. It can be used, as it was here, for a cute classroom exercise to show you learned how to run (not understand, but run) the statistical software from class. But only an idiot would think this paltry data was useful for estimating the effect.

(A variation on these “therefore it is impossible” claims is the assertion that because it is a cross-sectional study, it can only show correlation and not causation. I am so sick of debunking that particular bit of epistemic nonsense that I am not even going to bother with it here.)

So, we do not know the order of events. We can be confident that almost all the smokers or former smokers who had heart attacks smoked before that event. We do not know whether subjects quit smoking and/or started vaping before their heart attacks. Given that vaping was a relatively new thing at the time of the surveys, whereas heart attacks were not, it seems likely that most of the heart attacks among vapers occurred before they started vaping. This creates a lot of noise in the data.

A second, and seemingly more common, erroneous criticism of the analysis is that this noise has a predictable direction: “Smokers had heart attacks and then, desperate to quit smoking following that event, switched to vaping, thereby creating the association.” Again, no no no. Heart attacks do cause some smokers to become former smokers, but there is little reason to believe they are much more likely than other former smokers to have switched to vaping. Some people will have heart attacks and quit smoking unaided or using some other method. Indeed, I am pretty sure (not going to look it up, though because it is not crucial) that most living Americans who have ever had a heart attack experienced that event before vaping became a thing. So if they quit smoking as a result of the event, they did not switch to vaping. Also it seems plausible that the focusing event of a heart attack makes unaided quitting more likely than average, as well as making “getting completely clean” more appealing.

Of course, an analysis of whether behavior X causes event Y should not be based on data that includes many Y that occurred before X started. That much is obviously true. NHIS data is not even a little bit useful here, which is the major problem. There is so much noise from the heart attacks that happened before vaping this that the association in the data is utterly meaningless for assessing causation.

But there is no good reason to assume that this noise biases the result in a particular direction. If asked to guess the direction of the bias it creates, a priori, I probably would go in the other direction (less vaping among those who had heart attacks compared to other former smokers). The main reason we have to believe that the overall bias went in a particular direction is that the result shows an association that is not plausibly causal. We know the direction of the net bias. But this is not the same as saying we had an a priori reason to believe this particular bit of noise would create bias in a particular direction. When we see a tracking poll with results that are substantially out of line with previous results, it is reasonable to guess that random sampling error pushed the result in a particular direction. But we only conclude that based on the result; there was not an a priori reason to predict random sampling error would go in a particular direction.

Moreover, we do not have any reason to believe that the net bias was caused by this particular error, because it has a rather more obvious source (see below).

Sometimes we do have an a priori reason to predict the direction of bias caused by similar flaws in the data, as with the previous Glantz paper with an immortal person-time error (explained here, with a link back to my critique of the paper). If the medical students had engaged in a similar abuse of NHIS data to compare the risks of heart attack for current versus former smoking, then the direction of bias would be obvious: Heart attacks cause people to become former smokers, which would make former smoking look worse than it is compared to current smoking. I suspect that people who are making the error of assuming the direction of bias from the “Y before X” noise are invoking some vague intuition of this observation. They then mistranslate it into thinking that former smokers who had a heart attack are more likely to be vapers than other former smokers.

This brings up a serious flaw in the analysis that I did not have space to go into in my DV article: The analysis is not just of former smokers who vape, but includes people who both smoke and vape, as well as the small (though surprisingly large) number of never-smokers who vape. If vaping does cause heart attacks, it would almost certainly do so to a different degree in each of these three groups. For reasons I explored in the previous post, different combinations of behaviors have different effects on the risk of an outcome. Vaping probably is protective against heart attack in current smokers because they smoke less than they would on average. If a smoker vapes in addition to how much she would have smoked anyway, the increased risk from adding vaping to the smoking is almost certainly less than the (hypothesized) increased risk from vaping alone. Whatever it is about vaping that increases the risk (again, hypothetically), the smoking is already doing that. Thus any effect from adding vaping to smoking would be small compared to the effect from vaping compared to not using either product. Most likely the effect on current smokers would be nonexistent or even protective.

Indeed, this is so predictable that if you did a proper study of this topic (using data about heart attacks among vapers, rather than vaping among people who sometime in the past had a heart attack; also with a decent measure of smoking intensity — see below), and your results showed a substantial risk increase from vaping among current smokers, it would be a reason to dismiss whatever result appeared for former smokers. This is especially true if the estimated effect was substantial in comparison to the estimate for former- or never-smokers. If you stopped to think, you would realize that your instrument produced an implausible result, and thus it would be fairly stupid to believe it got everything else right. This is a key part of scientific hypothesis testing. Of course, such real science is not part of the public health research methodology. Nor is stopping to think.

It is a safe bet that the students who did this analysis understand none of that, having never studied how to do science and lacking subject-matter expertise. Glantz and the reviewers and editors of American Journal of Preventive Medicine neither understand nor care about using fatally flawed methods. So the analysis just “controls for” current and former smoking status as a covariate rather than separating out the different smoking groups as it clearly should. This embeds the unstated — and obviously false — assumption that the effect of vaping is the same for current, former, and never smokers. Indeed, because “the same” in this case means the same multiplicative effect, it actually assumes that the effect for current smokers is higher than that for former smokers (because their baseline risk is higher and this larger risk is being multiplied by the same factor).

Though they did not stratify the analysis properly, it is fairly apparent their results fail the hypothesis test. The estimate is driven by the majority of vapers in the sample who are current smokers, so they must have had a substantially greater history of heart attacks.

There is a good a priori reason to expect this upward bias, as I noted in the DV article, but it is not the reason voiced in most of the critiques. It is because historically vapers had smoked longer and more than the average ever-smoker. This is changing as vaping becomes a typical method for quitting smoking, or a normal way to cut down to having just a couple of real cigarettes per day as a treat, rather than a weird desperate attempt to quit smoking after every other method has failed. Eventually the former-smoking vaper population might look just like the average former-smoker population, with lots of people who smoked lightly for a few years and quit at age 25, and so on. But in the data that was used, the vapers undoubtedly smoked more than average and so were more likely to have a heart attack (before or after they started vaping).

Controlling for smoking using only “current, former, never” is never adequate if the exposure of interest is associated with smoking history and smoking causes the outcome, both of which are obviously true here. If there are no such associations then there is no reason to control for smoking, of course. Thus basically any time you see those variables in a model, you can be pretty sure there is some uncontrolled confounding due to unmeasured smoking intensity. In this case, you can be pretty sure that its effect is large and it biases the association upward.

In short, the results are clearly invalid. There are slam-dunk criticisms that make this clear. So let’s try to stick to those rather than offering criticisms that are as bad as the analysis itself. Ok?

Advertisements

Dual use and the arithmetic of combining relative risks

by Carl V Phillips

It was called to my attention that UCSF anti-scientist, Stanton Glantz, recently misinterpreted the implications of one of his junk science conclusions. Just running with the result from the original junk science (which I already debunked) for purposes of this post, Glantz make the amusing claim that because vaping increases heart attack risk by a RR=2 and smoking by a RR=3 (set aside that both these numbers are bullshit) then dual use must have a RR=5. WTAF?

First off, there is no apparent way to get to 5 except by pulling it out of the air. It is apparent that Glantz thinks he was adding the risks: 2+3=5. Except you cannot add risks that way. Every first-semester student knows the formula for adding risks, which is based on the excess risk. Personally I have always thought that having students memorize that as a formula, rather than making sure they inuit it, is a major pedagogic failure. But that aside, they do memorize the formula, which subtracts out the baseline portion of the RR then adds it back, as should be obvious: (RR1 – 1) + (RR2 – 1) + 1. So, the additive RR = 2-1 + 3-1 + 1 = 4. Think about it: If you “added” Glantz’s way then two risks that had RR=1.01 (a 1% increase in risk) would add to 2.02 (more than double). Or two exposures that reduced the risk by 10% (RR=0.9) would add to an increased risk, RR=1.8. Not exactly difficult to understand why this is wrong.

Additivity of risks is a reasonable assumption if the risk pathways from the exposures are very independent. The excess risk of death caused by both doing BASE jumping and smoking is basically just the excess risk of each added together. (A bit less because if one kills you, you are then not at risk of being killed by the other.) If the risks from the two exposures travel down the same causal pathways (or interact in various other ways), however, adding is clearly wrong. If vaping causes a risk (for heart attack in this example, though that does not matter), then smoking almost certainly causes the same risk via the same pathway. There is basically no aspect of the vaping exposure that is not also present with smoking (usually more so, of course). When this is the case, there are various possible interaction effects. One thing that is clear, however, is that simply adding the risks as if they did not interact is wrong.

The typical assumption built into epidemiology statistical models is that the risk multiply. This is not based on evidence this is true, but merely on the fact that it makes the math easier. The default models that most researchers tell their software to run, having little or no idea what is actually happening in the black box, build in this assumption. It is kind of roughly reasonable for some exposures, based on what we know. In the Glantz case, this would result in a claim of RR = 2 x 3 = 6, which is also not the same as 5.

So, for example, if a certain level of smoking causes lung cancer risk with RR=20, and a certain level of radon exposure causes RR=1.5, then if someone has them both, it is not unreasonable to guess that the combined effect causes RR=30. The impact on the body in terms of triggering a cancer and then preventing its growth from being stopped seems like it would work about like that. On the other hand, there are far more examples where the multiplicative assumption is obviously ridiculous. If BASE jumping once a week creates a weekly RR for death of 20, and rock climbing once a week has RR=2, doing each once a week obviously adds, as above, for RR=21, rather than multiplying to 40. (Aside: most causes of heart attack are probably subadditive, less than even this adding of the excess risks, as evidenced by dose-response curves that flatten out, as with smoking.)

But importantly, notice the “each once a week” caveat. That addresses the key error with the stupid “dual use” myths by specifying that the quantity of each activity was unaffected by doing the other. If, on the other hand, someone is an avid BASE jumper, doing it whenever he can get away, and he takes up rock climbing, the net effect is to reduce his risk. The less hazardous activity crowds out some of the more hazardous activity. This, of course, is what dual use of cigarettes and vapor products (or any other low-risk tobacco product) does. This is not complicated. Every commentator who responds to these dual use tropes — and I am not talking epidemiology methodologists, but every last average vaper with any numeracy whatsoever — points this out. Vaping also does not add to the risk of smoking because it almost always replaces some smoking rather than supplementing it. In this case, using Glantz’s fictitious numbers, it would mean the RR from dual use would fall somewhere between 2 and 3. Not added. Not multiplied. Not whatever the hell bungled arithmetic that Glantz did. Between.

As I said, everyone with a clue basically gets this, though it is worth going through the arithmetic to clarify that intuition. It is not clear whether Glantz really does not understand or is pretending he does not — as with Trump, either one is plausible for most of of his lies. Undoubtedly many of his minions and useful idiots actually believe it is right. The “dual use” trope gets traction from the fact that interaction effects from some drug combinations are worse than the risk of either drug alone. Many “overdose” deaths are not actually overdoses (the term that should be used for all drug deaths is “poisonings” to avoid that usually incorrect assumption), but rather accidental mixing of drugs that have synergistic depressant effects, often because a street drug was secretly adulterated with the other drug.

But as already noted, that is obviously not the case with different tobacco products, whose risks (if any) are via the same pathways. Even if total volume of consumption was unaffected by doing the other (as with “each once a week”) the risks would not multiply and would probably not even add. Since that is obviously not true — since in reality, consuming more of one tobacco product means consuming less of others — the suggestion is even more clearly wrong. In fact, using the term “dual use” to describe multiple tobacco products makes no more sense than saying that about someone who smokes sticks that came out of two different packs of Marlboros on the same day.

In the context of tobacco products, the phrase “dual use” is inherently a lie. It intentionally invokes the specter of different drugs (or other exposure combinations) that have synergistic negative effects. That is not remotely plausible in this case. It also intentionally implies additivity of the quantity of exposure (“doing all this, and adding in this other”) when it is actually almost all substitution, as with which pack you pull your cigarette from. To the extent that it increases total consumption of all products, this is a minor effect (a smoker who vapes not only as a partial substitute, but also occasionally when he would not have smoked even if he did not vape). This only matters to someone who does not care about risk, let alone people, and only cares about counting puffs.

There is a long list of words and phrases that when used by “public health” people should make you assume they whatever they are saying is a lie: “tobacco” (when used as if it were a meaningful exposure category), “addictive” (meaningless for drugs with little or know functionality impacts), “chemical” (a meaningful word, but invariably used because it sounds scary), and “carcinogen” (when used as a dichotomous characterization, without reference to the relevant dosage and risk). “Dual use” should be added to this list, in the same general space as “chemical”, another word that is inherently just a simple boring technical descriptor, but that is almost exclusively used to falsely imply negative effects.