Noise — Interactive Mindmaps

Key concepts: Introduction: Two Kinds of Error

1. Introduction: Two Kinds of Error

The Shooting Range Analogy: Distinguishing Bias from Noise

• Bias is systematic, predictable error (like a bent gunsight causing consistent misses).
• Noise is random, unwanted variability in judgments (like shots scattering widely).
• Some systems suffer from both bias and noise, compounding error.
• The analogy translates directly to human judgment: bias is directional inaccuracy, noise is inconsistency.

Measuring Noise Without Knowing the Truth

• Noise can be identified by examining variability alone, even when the correct answer is unknown.
• Disagreement among experts (e.g., doctors, judges) reveals noise regardless of who is right.
• This 'view from behind the target' makes noise a measurable problem in uncertain real-world decisions.

Evidence of Pervasive Noise in Critical Domains

• Medical diagnoses vary widely between doctors for the same patient.
• Judicial systems show extreme inconsistency (e.g., asylum grant rates from 5% to 88%).
• Professional forecasts, personnel decisions, and forensic science exhibit high variability.
• This noise creates unfairness ('refugee roulette') and economic waste.

Book Structure and Purpose

• Aims to elevate noise in public awareness as a distinct and damaging error.
• Will investigate noise in specific domains (criminal justice, insurance) via noise audits.
• Explores the psychology of why noise is overlooked and the advantages of algorithms.
• Provides practical noise-reduction techniques under the framework of 'decision hygiene'.
• Considers the appropriate level of noise, acknowledging it cannot always be eliminated entirely.

Key concepts: 1. Crime and Noisy Punishment

2. 1. Crime and Noisy Punishment

The Ideal vs. Reality of Justice

• Fundamental expectation: similar crimes under similar circumstances should receive similar sentences
• Historical reality: this ideal was routinely violated due to judicial discretion
• Systemic 'noise' refers to unwanted variation in professional judgments
• Ongoing tension between consistency and judicial discretion

The Problem of Unchecked Judicial Discretion

• Legal establishment celebrated discretion as humane and just
• Belief that rigid rules were dehumanizing and discretion allowed tailored punishments
• Resulted in sentences depending more on judge's views than case facts
• Created outrageous disparities: identical crimes received wildly different penalties

Evidence of Systemic Noise in Sentencing

• Judge Marvin Frankel ignited reform movement in 1970s, calling disparities 'arbitrary cruelties'
• Controlled studies showed 'astounding' and 'substantial disparity' among judges
• No consensus even on basic decisions like incarceration
• Real-world noise influenced by irrelevant factors: hunger, football games, birthdays, temperature

Sentencing Guidelines and Their Impact

• Sentencing Reform Act of 1984 created U.S. Sentencing Commission
• Mandatory guidelines used grid system based on offense severity and criminal history
• Explicit goal: reduce 'unfettered discretion' causing noise
• Studies confirmed guidelines significantly reduced interjudge disparity and racial biases

Criticism and Return of Discretion

• Critics argued guidelines were too rigid and mechanistic
• 2005 Supreme Court decision made guidelines advisory rather than mandatory
• Consequence: interjudge disparity in sentencing doubled
• Judges' personal characteristics (gender, politics) and racial disparities increased

Key Lessons and Ongoing Challenges

• Noise is pervasive in complex judgments across many fields
• Noise has serious consequences: unfairness, erosion of rule of law, high costs
• Noise can be measured and reduced through structured tools
• Balancing consistency with discretion remains a persistent and difficult conflict

Key concepts: 2. A Noisy System

3. 2. A Noisy System

Introduction to System Noise

• Concept introduced through a practical insurance company case study
• Noise represents unwanted variability in professional judgments within organizations
• Often invisible in daily operations but incurs massive costs and undermines fairness

The Hidden Cost of Inconsistency

• Insurance company's random case assignment created a 'lottery' for outcomes
• Executives predicted minimal variability (around 10%) which they considered tolerable
• Random assignment meant financial consequences for company and customers were subject to unwanted chance
• No one was aware of the full extent of the inconsistency problem

Conducting a Noise Audit

• Multiple professionals independently evaluated identical case descriptions
• Results revealed staggering inconsistency: 55% median difference for underwriters, 43% for adjusters
• Variability was roughly five times higher than executives had predicted
• Noise audit made previously hidden scatter of judgments visible
• Calculated annual cost likely in hundreds of millions of dollars from both overpricing and underpricing

Distinguishing Noise from Welcome Variability

• Personal taste diversity (film reviews, wine ratings) is expected and desirable
• Competitive settings (research teams, traders) require variation for innovation
• System noise occurs when single, randomly chosen individuals make binding organizational judgments
• In such systems, consistency is the goal and variability is unwanted
• Errors do not cancel out—both overestimation and underestimation are costly mistakes

Why Noise Remains Unseen

• Professionals and leaders operate under an 'illusion of agreement'
• Shared language and professional norms foster belief in similar worldviews
• Disagreements are dismissed as rare lapses or avoided through organizational habits
• Professionals gain confidence through repetition and fluency, not peer calibration
• Without noise audits, organizations address only egregious outliers while missing pervasive scatter

Key concepts: 3. Singular Decisions

4. 3. Singular Decisions

Defining Singular Decisions

• Characterized by uniqueness, made only once, and lacking prepackaged responses
• Examples include presidential crisis responses, military commands, and major personal choices
• Exist on a continuum with recurrent decisions but represent meaningfully different extremes
• Traditionally analyzed through causal, hindsight-driven narratives rather than statistical patterns

The Invisible Presence of Noise in Singular Decisions

• Noise cannot be directly observed or measured in singular decisions because the same problem is never repeated
• Noise is still present—a singular decision could have been different due to variability in human judgment
• Counterfactual thinking reveals noise by considering how different advisors, presentation of facts, or moods could alter outcomes
• The COVID-19 pandemic responses demonstrate observable noise; if only one country faced it, the noise would be invisible but real

Judgment as Imperfect Measurement

• Judgment is a form of measurement where the human mind is the instrument
• Like physical instruments, the mind produces errors consisting of both bias (systematic deviation) and noise (unwanted variability)
• A stopwatch exercise illustrates variability (noise) and systematic error (bias) in simple judgments
• When judgment aims at a true value, variability means error—different judgments cannot both be perfectly right

Key Implications and Takeaways

• Singular decisions are subject to the same psychological noise as recurrent decisions
• Noise in singular decisions is invisible but real, requiring counterfactual thinking to appreciate
• The best approach to a one-of-a-kind decision is to treat it as a recurrent decision that happens only once
• Strategies that reduce noise in recurrent decisions can also improve singular decisions
• Framing judgment as measurement clarifies noise as a fundamental component of error alongside bias