Tag Archives: methodology

My recent contribution to Clive’s weekly reading list

by Carl V Phillips

As some of you know, Clive Bates puts out a weekly somewhat-annotated list of PubMed-indexed articles that are related to low-risk tobacco products and/or tobacco harm reduction (the search string for that appears at the end of what follow). It is a great resource; if you do not receive it, I am sure he would be glad to add you to the distribution list. As part of a planned projected that I have alluded to before, I am working on how to reinterpret this as an annotated weekly suggested reading (or knowing-about) list. To that end, this week I was a “guest editor” for Clive’s distribution list, and I thought I should share what I wrote here to broaden the audience. Yes, it is a little weird to publish a one-off “weekly reading” that is mostly based on an existing format that you might not be familiar with. But you should be able to get the idea. Hopefully I will be producing one every week before too long.

In the meantime, here is what I wrote that went out via Clive’s distribution lists. Sorry for the weird formatting — it is an artifact of the way the original PubMed search was formatted. Yes, I could have fixed the for aesthetics to re-optimize for this blog’s formatting, but since they do not hinder comprehension, I am not going to bother — sorry.

Greetings everyone. Carl V Phillips here, doing Clive’s list this week. I am trying out a new format for it, as follows: (1) They are not listed in the order that popped from the PubMed search string, but rather is in order of how worth reading they are. Obviously this is my own rough blend of various considerations, including importance of what is being addressed, value of what was produced, how potentially influential it is, and how much reader effort it takes to get value from it (note that I put relatively little weight on the latter). I have left the serial numbers from the search on the entries in case anyone wants to recreate the usual ordering. I add a full-text link if I think there is anyone other than specialists in the particular area would want to look at the full text. (2) I am not limiting this to PubMed-indexed papers. I am including popular press and policy statements (and would have included blogs but there were not any apparent candidates this week).

Continue reading

People who report health risks as percentage changes are (often) liars

by Carl V Phillips

I have been having an ongoing conversation with Kristin Noll-Marsh about how statistics like relative risks can be communicated in a way that allows most people to really understand their meaning.  There is more there than I can cover in a dozen posts, but I thought I would at least start it.  I have created the tag “methodology” for these background discussions about how to properly analyze and report statistics (“methodology” is epidemiologist-speak for “how to analyze and report data”).

Most statistics about health risks are reported in the research literature as ratio measures.  That is, they are reported in terms of changes from the baseline, as in a risk ratio of 1.5, which means take the baseline level (the level if the exposures that are being discussed are absent) and multiply by 1.5 to get the new level.  This is the same as saying a 50% increase in risk.  It turns out that these ratios are convenient for researchers to work with, but are inherently a terrible way to report information to the public or decision makers.  There is really no way for the average person to make sense of them.  What does “increased risk, with an odds ratio of 1.8” mean to most people?  It means “increased risk”, full stop.

Every health reporter who puts risk ratios in the newspaper with no further context should be fired (some of you will recall my Unhealthful News series at EP-ology).  But the average person should not feel bad because it is likely that the health reporter — and most supposed experts in health — cannot make any more sense of it either.

The biggest problem is that a ratio measure obviously depends on the size of the baseline.  When the baseline is highly stable and relatively well understood, then the ratio measure makes sense.  This is especially true when that deviation from the baseline is actually better understood than actual quantities.  So, for example, we might learn that GDP increased by 2% during a year.  Few people have any intuition for how big the GDP even is, so if that were reported as “increased by $X billion” rather than the ratio change, it would be useless.  Of course, that 2% is not terribly informative without context, but the context is one that many people basically know or that can easily be communicated (“2% is low by historical standards, but better than the recent depression years”).

By contrast, to stay on the financial page, you might hear that a company’s profits increased by 10,000% last year.  Wow!  Except that might mean that they profited $1 the year before and got up to $100 last year.  Or it might be $1 billion and $100 billion.  The problem is that the baseline is extremely unstable and not very meaningful.  This contrasts yet again with a report of revenue (total sales) increasing by 50%, which is much more useful information because a company’s sales, as opposed to profits, are relatively stable and when they change a lot (compared to baseline), that really means something concrete.

So returning to health risk, for a few statistics we might want to report, the baseline is a stable anchor point, but not for most reported statistics.  It is meaningful to report that overall heart attack rates are falling by about 5% per year.  The baseline is stable and meaningful in itself (the average across the whole population), and so the percentage change is useful information in itself.  This is even more true because we are talking about a trend so that any little anomalies get averaged out.  By contrast, telling you that some exposure increases your own risk of heart attack by about 5% per year is close to utterly uninformative, and indeed probably qualifies as disinformative.

As I mentioned, the ratio measure (in forms like 1.2 or 3.5) are convenient for researchers to use.  You probably also noticed me playing with percentage reporting, using numbers you seldom see like 10,000%.  This brings us to the reporting of risk ratios in the form of percentages as a method of lying — or if it is not lying (an attempt to intentionally try to make people believe something one knows is not true), it is a grossly negligent disregard for accurate communication.

Reporting a risk ratio of 1.7 for some disease may not mean much to most people, but at least that means it is not misleading them.  There is a good way to explain it in simple terms, something like, “there is an increase in risk, though less than double”.  If the baseline is low (if the outcome is relatively uncommon) then most people will recognize this to be a bad thing, but not too terribly bad.  So the liars will not report it that way, but rather report it as “a 70% increase”.  This is technically accurate, but we know that it is very likely to confuse most people, and thus qualifies as lying with the literal truth.  Most people see the “70%” and think (consciously or subconsciously), “I know that 70% is most of 100%, and 100% is a sure thing, so this is a very big risk.”

(As a slightly more complicated observation:  When these liars want to scare people about a risk, they prefer that a risk ratio come in at 1.7 rather than a much larger 2.4.  This is because “70% increase” triggers this misperception, but”140% increase”, while still sounding big and scary, sends a clear reminder that the “almost a sure thing” misinterpretation cannot be correct.)

The problem here is that people — even fairly numerate people when working outside areas they think about a lot — tend to confuse a percent change and a percentage point change.  When the units being talked about are percentages (which is to say, probabilities, as opposed to the quantities of money like the above examples) that are changing by some percentage of that original percentage, this is an easy source of confusion that liars can take advantage of.  An increase in probability by 70 percentage points (e.g., from a 2% chance to a 72% chance) is huge.  An increase of 70 percent (e.g., from 2% to 3.4%) is not, so long as the baseline probability is low, which it is for almost all diseases for almost everyone.

There seems to be more research on this regarding breast cancer than other topics (breast cancer is characterized by an even larger industry than anti-tobacco that depends on misleading people about the risks, and there is also more interest in the statistics among the public).  It is pretty clear that when you tell someone an exposure increases her risk of breast cancer by 30%, she is quite likely to freak out about it, believing that this means there will be a 1-in-3 chance she will get the disease as a result of the exposure.

Reporting the risk ratio of 1.3 will at least avoid this problem.  But there are easy ways to make the statistic meaningful to someone — assuming someone genuinely wants to communicate honest information and not to lie with statistics to further a political goal or self-enrichment.  The most obvious is to report the relative risk based on the absolute risk (the actual risk probability, without reference to a baseline), or similarly report the risk difference (the change in the absolute risk), rather than ratio/percentage.  This is something that anyone with a bit of expertise on a topic can do (though it is a bit tricky — it is not quite as simple as a non-expert might think).

Reporting absolute changes is what I did when I reported with the example of 2% changing to 3.4% (or, for the case of 1.3, that would be changing to 2.6%).  The risk difference when going from 2.0% to 3.4% would be 1.4 percentage points, or put another way, you would have a 1.4% chance of getting the outcome as a result of the exposure. Most people are still not great at intuiting what probabilities mean, but they are not terrible.  At least they have a fighting chance.  (Their chances are much better when the probabilities are in the 1% range or higher, rather than the 0.1% range — once we get below about 1% intuition starts to fail badly.)

To finish with an on-topic example of the risk difference, what does it mean to say that smoke-free alternatives cause 1% of the risk of serious cardiovascular even (e.g., heart attack, stroke) of smoking?  [Note: that this comparison is yet another meaning of “percent” than those talked about above — even more room for confusion!  Also, this is in the plausible range of estimates, but I am not claiming it is necessarily the best estimate.]  It means that if we consider a man of late middle age whose nicotine-free baseline risk is 5% over the next decade, then his risk as a smoker is 10%.  Meanwhile, his risk as a THR product user would be 5.05%.  Moreover, this should still be reported as simply 5% (no measurable change) since the uncertainty around the original 5% is far greater than that 0.05% difference.

Opinion surveys provide information about personal beliefs and behavior – only!

by Carl V Phillips

Why am I writing a post under a heading that is so incredibly obvious?  Because in the world of the tobacco control industry, even incredibly obvious truths are often ignored.

Survey research that asks people what they have done or experienced is often the only source of scientific data that addresses those questions.  Also, when we are interested in people’s personal preferences or guesses about something, some sort of survey is often the only way to find out.  The problem comes in when someone who does not understand science — or whose job description includes pretending to pretend to not understand — says, “hey, we use survey data as scientific fact when studying behavior and exposures, and opinion polls look similar to behavior and exposure surveys, so it must be that the results of opinion polls can be used as scientific data.”  Um, yeah.

Surveys about opinions are, of course, evidence of what people think, which is interesting for answering some questions.  But those are questions about belief/knowledge/understanding/confusion, not about the physical world.  Some of you might recall that the whole “third hand smoke” scam traces back to a survey where random people with no expertise were actively tricked into saying they believe that it is a hazard.

There are methods of aggregating the opinions of people with some expertise to crowdsource a legitimate prediction about some event.  It only works with predictions, though, because it requires placing a bet on the outcome which are paid out when the outcome is determined.  This is how we determine the probability of a sports team winning a game and also has been used in some clever tools for predicting elections.  Those who respond to these surveys (other than with small self-entertainment bets that are not going to be big enough to affect what the crowd predicts) are self-selected people who think that they know enough to come out ahead on their bets, not just the average person on the street.  And, importantly, there is a punishment (losing the money you bet) for expressing an opinion that is uneducated, or that you know to be wrong — this is not cheap talk.

Contrast this with a recent “study” that used an opinion poll to “predict” the effects of plain packaging of cigarettes.  The “researchers” asked a handful of people in the tobacco control industry, presumably many of whom are directly invested in the plain packaging boondoggle, what will happen and reported the result as if it was a useful prediction.  Needless to say, the prediction of the effect of taking away brand logos by the people who have run out of useful things to suggest, was an absurdly large impact.

I would write more, but there is no way to usefully add to what Snowdon (who reported this story) already wrote about this, so give his very funny post a quick read.

The limitation of this survey is just the obvious point that the respondents are not only ridiculously biased, but they have absolutely no incentive to give an accurate prediction or to refrain from predicting if they lack confidence in their opinion.  The “researchers”, had they been interested in actually learning something, could have asked the respondents to place a bet, but did not.  Without a bet, there is no incentive to tell the truth because there are absolutely no penalties in the tobacco control industry for making incorrect predictions or scientific declarations that are clearly shown to be false.  It would be delightful to see ANTZ “researchers” and “experts” being held to account for lying and forced to do a Lance Armstrong, begging for forgiveness, promising (without credibility) to never do it again, and begging to not have to give up the hundreds of millions of dollars they swindled by lying.

Another recent example of dumb polling is more troublesome in its implications.  A media blast by the Schroeder Institute for Tobacco Research and Policy Studies (Steve Schroeder must just be so proud of the “research” that is coming out attached to his name) claimed that it is a good idea to mandate a lowering of the nicotine content of cigarettes because a majority of random Americans were tricked into saying they thought it was a good idea by a survey.  (For more details, see this article, which unlike the usual churnalism includes good analysis by Michael Siegel and others.)  It turns out that this “majority” consists of forty-something percent, but we do not expect basic numeracy from tobacco control, so I will just move on.

What does this survey really tell us?  It tells us that the tobacco control industry’s efforts to confuse people about the source of harms and benefits from smoking have been rather successful, though surprisingly not quite as successful as one might have guessed.  Obviously it tells us nothing about whether such a policy would actually be a good idea by any measure.  It does not even tell us whether people really have this belief in any meaningful sense, or if they merely decided to agree with the statement while blasting through a survey.  Polling people about something they have never given any serious thought is unlikely to provide useful information, even if they have no incentive to be dishonest and even if they might know something about it. Even a poll of people asking them how far they could drive given the gas currently in their car is well within their expertise to answer, but their answers would not be very accurate.   They would just give a snap answer without bothering to go check how much gas they have, let alone calculate their mileage.

It is obviously worse when the question is well beyond people’s expertise.  We know, after all, that a majority (a term I am using like the Schroeder Institute people do, to mean “at least a substantial minority”) also believe, without any scientifically defensible basis whatsoever, that: we should not worry about the future because the gods are going to end life on Earth within our lifetimes; that it is healthier to eat “organic” foods; that screening mammography provides a major health benefit; that Iraq posed a threat to the US in 2003; that current-tech wind turbines are an environmentally friendly way to generate electricity; and that cutting government spending in an economic depression characterized by zero-lower bound interest rates.

The problem is that for all but the first two of those, policies have conformed to the opinion of that majority (or “majority”) that is unmoored by the facts and the science.  What distinguishes the first two from the others?  For all of the others, the rich and powerful people profited by keeping people ignorant and getting them to believe, and thus support, something that is false.  While the tobacco control industry is not nearly as rich and powerful as those who have profited from the mammography, wind turbine, and Iraq War boondoggles, they differ only in degree, not in their willingness to foster ignorance to support their cause.  So there is little doubt that they will use the cultivated ignorance to further their agenda.

Reducing nicotine content reduces the benefit of smoking a cigarette while not reducing the harm.  The same would be true for adding some harmless but foul-tasting chemical to the cigarettes.  The main difference is that the latter option would almost certainly cause people to smoke less, while everything we know suggests that lowering nicotine content will cause people to smoke more.  If people had been polled about a proposal to add the foul chemical to cigarettes, it is likely that a smaller “majority”, maybe 25%, would support that too — because they basically favor prohibition, and that would be roughly equivalent.  But no one would mistake that for scientific evidence that the policy would have a positive impact on the world.  Whatever someone might think of the ethics of intentionally lowering the quality of cigarettes to discourage smoking, it seems that the dumbest possible way to do it is in a way that makes people want to smoke more.  Make no mistake, reducing nicotine is benefit reduction, not harm reduction.

[Aside: It is worth noting that while all the evidence suggests that substantially lowering the nicotine content would increase harm, this does not mean that substantially raising the nicotine content would reduce harm.  So much of smoking behavior is habitual (and many people seem to smoke enough that at the margin their nicotine receptors should already be saturated) that it is not entirely clear that higher nicotine would reduce total smoking, unless accompanied by some other change like making the cigarettes shorter (“same great nicotine with only half the stick”).  But at that point, why not just advocate replacing some of someones daily cigarettes with smoke-free nicotine?  That would have the advantage of encouraging a complete switch, as well as avoiding several obvious downsides.]

To sum up, there is a lot to be learned about perception, propaganda, and ethics from looking at results of surveys like this.  But there is nothing useful to be learned about science or science-related policy.  That finally circles my thinking back to a survey from 2003 that is a large part of why I was inclined to address these points.  It is still cited as if it actually represents an estimate of the risk of smoke-free alternatives, a practice I have repeatedly criticized but apparently never posted about.  But this is already enough for the day, so I will try to come back to that.