Since it’s been a while, I’ll reaffirm my pre-hiatus policy on linkposts:

I’d like to use your attention responsibly. To that end, I want to avoid spraying links to whatever’s recently hijacked my brain. When I share a link, I’ll do my best to make it a classic: a text I’ve had time to consider and appreciate, whose value has withstood the vagaries of the obsession du jour, my own forgetfulness and mental flux; something that changed my mind or (and) continues to inform my outlook; a joy to read and re-read. A piece of my personal Internet canon. Anyway, don’t get your hopes up.

My previous post, on fair tests with unfair outcomes, is a narrow piece in a broad dialogue about quantitative measures in prediction and decision-making. I pointed out one specific statistical fact that shows up in certain situations, and observed that it’s valuable to consider “statistical fairness” not just of a test but of the decisions and consequences that follow from it. Outcomes that are fair in the ways we care about don’t necessarily flow from tests that give unbiased predictions, especially when we care about error rates of decisions more than point estimates from predictions (i.e. more than what some people think of as the part of the whole business that needs to be fair or not).

One much more important piece of this dialogue is Tetlock’s Expert Political Judgment. I strongly recommend it, for its substantive contribution to what we can say on the subject, its practical grounding, and its thread of doing better through intellectual humility. It also serves as a great example of thorough and even-handed analysis, and the patient exhaustion of alternative explanations.

Another such piece is Laurence Tribe’s Trial By Mathematics. Tribe systematically considers mathematical methods in civil and criminal trials. His analysis applies much more broadly. Along the way he makes a number of subtle points about probability, Bayesianism, and utility, as well as about the tradeoffs between trial sensitivity and specificity. In his words:

A perhaps understandable pre-occupation with the novelties and factual nuances of the particular cases has marked the opinions in this field, to the virtual exclusion of any broader analysis of what mathematics can or cannot achieve at trial—and at what price. As the number and variety of cases continue to mount, the difficulty of dealing intelligently with them in the absence of any coherent theory is becoming increasingly apparent. Believing that a more general analysis than the cases provide is therefore called for, I begin by examining—and ultimately rejecting—the several arguments most commonly advanced against mathematical proof. I then undertake an assessment of what I regard as the real costs of such proof, and reach several tentative conclusions about the balance of costs and benefits.

He then stands back and asks about the broader effects of adopting a quantitative rule, not in terms of a utility calculation but as a matter of legitimacy and confidence in the justice system:

As much of the preceding analysis has indicated, rules of trial procedure in particular have importance largely as expressive entities and only in part as means of influencing independently significant conduct and outcomes. Some of those rules, to be sure, reflect only “an arid ritual of meaningless form,” but others express profoundly significant moral relationships and principles—principles too subtle to be translated into anything less complex than the intricate symbolism of the trial process. Far from being either barren or obsolete, much of what goes on in the trial of a lawsuit—particularly in a criminal case—is partly ceremonial or ritualistic in this deeply positive sense, and partly educational as well; procedure can serve a vital role as conventionalized communication among a trial’s participants, and as something like a reminder to the community of the principles it holds important. The presumption of innocence, the rights to counsel and confrontation, the privilege against selfincrimination, and a variety of other trial rights, matter not only as devices for achieving or avoiding certain kinds of trial outcomes, but also as affirmations of respect for the accused as a human being—affirmations that remind him and the public about the sort of society we want to become and, indeed, about the sort of society we are.

I’m always reluctant to excerpt things like this—particularly summaries of arguments like in the second quote—because it feels like I’m implying that the summary is the extent of what you’ll get out of it, and that it’s not worth reading in its entirety. Let there be no confusion: Trial By Mathematics belongs on any reading list about decision-making under uncertainty, and it’s only 65 pages.