Sunday Science Lesson: mistaking necessity for virtue in study design

by Carl V Phillips

Yes, I have written versions of this before, but I never tire of the topic, mostly because of how much damage the errors do to science and health policy.  I get reminded of it every time I travel through a European or European-influenced airport.

Most scientific knowledge (which is just a fancy way of saying “knowledge” — I am just coopting the phrase from those who try to imply that the adjective is meaningful) comes from easy observations — e.g., “there are a lot more women than men in this population” requires only looking around.  Sometimes a bit of knowledge of interest gets a bit more complicated and we need to actively use measurement instruments — e.g., “this is heavy” is easy, but “this has a mass of 44.21 kg” requires careful methods and a good scale.  Finally, something that we want to know might be completely beyond our ability to assess without complicated methods — e.g., “does a lifetime of exposure to E double the risk of disease D” requires a complicated statistical analysis of thousands of people.  The point here is that just because those methods are necessary for the latter does not mean they are necessary — or even useful! — for easier observations.

To elaborate on the concept, I will start with my favorite analogy to it:  Airports/stations need to communicate to thousands of people when their plane/train/bus leaves and where to board it, and until we all have reliable connectivity in our pockets (good realtime phone apps personalize our information and can make this all moot), this will continue to be provided using overhead displays.  These were originally written and updated by hand, and then replaced by some amazing and clever mechanical devices, and are now video monitors.  But fundamentally nothing has changed, and that is the problem, because airports are not train stations, or more particularly, flights are not train trips.

Consider what you naturally know and can easily remember when you arrive at an airport or train station.  Most obvious to you, you know your identity, which is sufficient to find your vessel (using the phone app, though it has always been sufficient to visit the check-in desk), but we still need displays which are quick to access, instantly updated, and always available.  To make the displays usable you need to know something other than your identity.  You surely know where you are going and approximately when you are leaving, and this is all you need to identify your flight.  There are seldom multiple departures from one airport to a particular other airport at close to the same time (particularly since you also easily remember which airline you are flying).  This does not work so well for trains, however, because it can be that almost half the trains leaving a station make a particular stop because every train going a particular direction passes that station and stops.

This also means there is a difference in what can be communicated via the monitors, because planes land in just one place, whereas the same train stops several or dozens of places and not all can be listed.  Thus, train stations are forced to have their passengers to drill down further and make an effort to remember something that is not quite so intuitive: the exact minute of the scheduled departure time, which is how they identify the vessels (usually along with one target destination, either the end of the line or the most major station on the way, which is intuitive to remember).

You probably see where I am going with this:  American airports and those following their style display departing flights based where they are going, alphabetically by city.  This a great system since everyone knows where they are going and is so skilled at searching by alphabetical order that they can quickly glance to the right range of the list to find the city name.  European-style airports have been designed by people who seem to think they are train stations, and list flights by the minute of departure.  This is a bad system because it requires passengers to make the extra effort to remember or check the exact minute of departure, and to step through a list of ordered numbers with varying gaps, which is much harder than alphabetical order because you cannot use instant intuition like “I am going to Philadelphia, so I will start direct my glance to 3/4 of the way through the list”.

Like a complicated cohort study or clinical trial, the train-style listing is a cost necessity under particular conditions.  But such necessity is not a virtue of the method.  “It is needed at train stations, so it is the best we can do there” clearly does not imply “it is always best.”  Similarly, “we cannot figure out whether this exposure causes a .001 chance of that disease without a huge systematic study and a lot of statistic analysis” does not mean “we cannot figure out that e-cigarettes help people quit smoking without such a study”.  Even more absurd is the “reasoning” that leads to: “we cannot figure out which medical treatment works better without a clinical trial” and therefore “we cannot figure out if people like e-cigarettes without a clinical trial”.

Needless to say, the latter statement in each sentence is obviously false, and the proposed equivalences are moronic.  Just because the extra complication and effort is needed to ask a hard quantification does not mean that it is needed for an obvious qualitative conclusion.  Anyone who actually understands science at the grade-school level realizes that different research is needed to answer different questions.  It makes a bit more sense to use a clinical trial to try to understand adoption of THR than it does to use a particle accelerator to do it, but not a lot more.

Yet, of course, it is just such innumeracy that appears in the public discourse.  Just as habit leads many people to ignore common sense and insist that train-style displays at airports make sense, “public health” indoctrination also eliminates the common-sense level science that is taught in grade school.  It is reassuring to note that the claims about a particular type of study always being best, or even merely always being needed, are not made by actual scientists.  They always come from political activists or medics, and occasionally from incompetent epidemiologists (not actually a redundant phrase — just close to it).

I think this analysis also extends into dealing with thought-free analogies in regulation, such as “we do X with cigarette regulation and therefore should do it with products that are different in almost every way other than being tobacco” or “we require X for medicines that serve only to eliminate a disease, and therefore should require it for products that people use for enjoyment.”  I will leave that extension as an exercise.