by Carl V Phillips
I have composed this at the request of Gregory Conley, who recently had the nightmarish experience of trying to explain science to a bunch of health reporters. It is just a summary, as streamlined as I am capable of, of material that I have previously explained in detail. To better understand the points, see this post in particular, as well as anything at this tag. For a bit more still, search “RCT” (the search window is the right or at the top, depending on how you are viewing this).
- RCTs, like every other study design, have advantages and disadvantages.
Background: The one fundamental advantage is that they eliminate systematic confounding. A second feature that is sometimes and advantage, but sometimes a disadvantage, is that they study a clearly-defined specific intervention, rather than the collection of similar but not identical exposures that occur in the real world. Clear disadvantages include volunteer subjects being different from the average person and subjects knowing they are being studied which affects behavior. The fatal disadvantage in many cases is that it is physically impossible to conduct an RCT that address the real questions of interest, and so the RCT addresses a question that is merely sort of similar to them.
- The reason there is a myth that RCTs are always better is that for actual medical treatments for diseases, the advantage is crucial and the disadvantages are not great.
Background: There is a huge problem of confounding-by-indication for medical treatments (roughly: people who seem most likely to benefit from a particular treatment are more likely to receive that particular treatment), so eliminating confounding is particularly important for answering the question, “what would happen if we gave this treatment to everyone?” The requirement of using one tightly-defined intervention is largely an advantage. Most patients will volunteer for studies, so the subject selection is not all that odd, and behavior has limited effect on physical treatment outcomes. What can be studied is pretty close to exactly the question of interest; medical treatment is always an intervention, so an experiment about the effects of an intervention is on-target.
- But just the opposite is true for studies of behavior, where the advantages are of minimal importance, or actually become disadvantages, and the disadvantages constitute fatal flaws.
- The questions we are most interested in for THR are things like, “does the availability of e-cigarettes in the real world — along with real-world knowledge and other realities — cause people who would otherwise have continued to smoke to quit?” An RCT can only answer a fundamentally different question, along the lines of, “what happens when you give the odd smokers who volunteered greater access to and knowledge about e-cigarettes than they had before?” We are interested in the real-world questions, not the effects of concocted interventions that will only occur in the experiments.
Additionally: Those who try e-cigarettes are self-selected — they are those who seem most likely to benefit from them — but this is part of the reality we want to study. That is, eliminating such confounding-by-indication is useful for answering the question “what would happen if you imposed this intervention on every smoker” but is totally wrong for understanding what happens in the real world, where the self-selection is an important part of the real experience of interest. Thus, the main advantage of RCTs actually becomes mostly a disadvantage when studying real-world phenomena rather than controlled medical treatments.
- The most frequently recited concerns about THR RCTs — that they are too expensive and that they do not offer subjects the “right” choices of products — are red herrings. These are not the fundamental problems.
Background: RCTs are certainly more expensive than many other study designs, but that cost is not one of their fatal flaws. The obsession with making sure the intervention includes high-quality products and proper instruction misses the point: The intervention can be made more or less effective, but it is still an intervention that is not at all similar to the real-world experience we are interested in understanding.