Sunday Science Lesson: toxicology and “the chains” in American football

by Carl V Phillips

Those of you who read my series on fatal flaws in FDA’s proposed rule about limiting the nitrosamine NNN in smokeless tobacco (and presumably anyone reading this quick little tangent read those important and carefully crafted posts) might have tripped up over an oddity from the third post in the series. I quoted this from FDA’s proposed rule about how their key number, used for estimating the risk of cancer caused by some quantity of NNN, was calculated:

As defined by the EPA guidelines, the cancer slope factor (CSF) is “an upper bound (approximating a 95percent [sic] confidence limit) on the increased cancer risk from a lifetime exposure to an agent.

I noted (you can read the original for more detail) this means that when FDA estimated the dose-response for NNN, they did not use the point estimate generated by the underlying study, but inflated it by an arbitrary fudge factor (which is not actually an upper bound, as claimed, but is still much higher than the point estimate). This is obviously an error. There are arguments that using such inflation factors when setting standards (e.g., how much of a potentially toxic substance a facility is allowed to emit) are appropriate, to err on the side of caution. But an inflation factor, creating a number higher than what the data suggests is the best estimate, obviously does not give us the best estimate for the actual dose-response. I also observed that the model used to translate the data from rodent megadose studies into an estimate for the effects of realistic human exposures was fraught with huge, undoubtedly incorrect, assumptions that made the final result nearly worthless, even apart from this.

So you might be asking why such lousy models and arbitrary fudge factor rules even exist. They are clearly grossly inappropriate for what FDA was doing — no ambiguity there. But presumably they serve some purpose, or they would not exist.

I found myself flashing back to when I was ten or twelve years old and a fan of American football. There is a process in that game that occasionally occurs, in which a very close judgment has to be made about whether the offensive team advanced the ball the required ten yards to get a “first down”. (That is all you need to know. Obviously most American readers will know more details. Also, I realize I do not even know whether what I describe is still done at professional levels, given that it could be replaced by imaging and computers, but is at least still presumably done in high school games.) At that point, two officials run in from the sidelines carrying “the chains”, a pair of posts connected by a ten-yard chain. One of them places one end at the starting point for the required ten yards of progress. The other then pulls the chain taut and observes whether the current placement of the ball is a little past his post or a little short of it.

You might wonder why. It is no easier to identify the exact starting point, and measure from it, versus just identifying the exact ending point needed for the first down. Since the play will usually have moved the ball sideways, it is not as if someone can just remember the exact blade of grass the ball was on at the start; it is necessary to eyeball the corresponding point on an imaginary line across the field. Also, the ball is not a single point. And the current placement of the ball after the last play was somewhat arbitrary too. So why not just eyeball the spot that is ten yards further (using as a guide, in either case, the markings of yards that are painted on the field, though not necessarily exactly on the line the ball is on)? Further contemplation reveals that the answer lies in game theory.

If one official has to eyeball a target point near where the ball is sitting on the ground, either just ahead of it or just behind it, he is full-on deciding whether to award the first down. That creates a huge amount of pressure and also creates an enormous potential for exercise of any bias that official is feeling for whatever reason. It could be nefarious bias. But it could be an innocent moral struggle such as, “I denied them the last close call that could have gone either way, so I owe them this one that could go either way.” Or it could be an attempt at beneficence in violation of the procedural rules like, “where the ball is sitting is short of my estimated spot, but my colleague who decided where to place the ball after the last play really should have put it further forward and I can fix his error.” But when the first official eyeballs his spot ten yards back, he cannot be sure whether an inch one way or another even matters and can just do it mechanically without all those inconvenient thoughts cropping up. Of course, the colleague could exercise nefarious bias when he chooses where to pick his spot; an inch forward or an inch back are both plausible estimates of the starting point. But the complicated mechanism reduces the temptation to exercise such bias somewhat, and strongly reduces effects of the “I owe them this one” or “I can fix it” factor.

Regulators setting an allowable level of potentially harmful effluent, contaminant, or ingredient also have to draw a line. The right place to draw the line is hugely uncertain, both in terms of what levels are actually harmful and the political decision about what level of harm should be allowed (this contrasts with the American football analogy). Getting it right is pretty much impossible. Still, issues like those facing the football referee can be avoided. If regulators are allowed to draw the line when looking at exactly where the ball is sitting, as it were, they are deciding such things as “this product is fine, but its leading competitor is banned,” or “the facilities operated by our boss’s biggest campaign donor all just squeak in under the line.” That would not be good.

So instead they create a rule that says “make an estimate based on this crazy dubious model and then inflate the result by this predefined arbitrary factor, and draw a line based on that.” This does not eliminate directional bias (intentionally trying to be more or less stringent) in defining the models or inflation factors, or in interpreting the underlying data. But it does help avoid someone saying “hey, if I just bump this limit down from 7.5 to 6.8, I can really stick it to that company that I have always hated.” Since the proper line is enormously uncertain, that would be easy to do.

For the same reason, it does not matter so much that many of the steps in the defined process are just silly. You can still get outcomes where experts largely agree that the standard spit out by the sketchy complicated (but well-defined) process is way too low or too high. But even then, at least it offers a starting point for debate that was not just someone capriciously making up a number. Most of the time, the genuine uncertainty is sufficient that the result of the process might actually be the optimal number.

Circling back to the FDA, it is worth noting that their proposed rule in no way resembles this clumsy, but arguably justifiable, process. They were not following a rule that spit out a quantitative standard that, while probably non-optimal, was at least non-arbitrary. No, they misused elements of this process to (inaccurately!) estimate the effects of their proposed standard. But their standard itself was still an arbitrary and capricious number that was pulled out of the air. This was done with the clear view of exactly which products would make the cut, which would have to be re-engineered, and which would be banned. This is exactly the bright-line decision about who wins and who loses that those football and normal regulatory rules are designed to prevent.

Well, I should say that FDA thought they had a clear view of exactly which products would be affected. As noted in the first post in my series, they actually made a factor-of-four arithmetic error that means far more products would be affected and far more banned than they intended. But the point is still that they were misusing the trappings of a process that is designed to avoid exactly such picking-and-choosing, while still trying to engage in arbitrary picking-and-choosing.