Noise

Introduction: Two Kinds of Error

Overview

Overview

The chapter opens with a vivid metaphor to distinguish between two fundamental types of error in human judgment: bias and noise. It argues that while bias—systematic deviation from accuracy—has dominated public discourse, noise—unwanted variability among judgments—remains a hidden scourge in organizations, leading to unfairness and economic waste. Through compelling examples and a clear framework, the chapter sets the stage for a deeper exploration of how noise permeates decisions and what can be done to reduce it.

The Shooting Range Analogy

Picture four teams at a shooting arcade, each with five members firing one shot. Team A’s shots cluster tightly around the bull’s-eye, showing precision. Team B’s shots are consistently off-target, demonstrating bias—a predictable error pattern, like a bent gunsight. Team C’s shots scatter widely with no clear bias, illustrating noise—random dispersion that makes predictions impossible. Team D suffers from both bias and noise. This analogy translates directly to human judgment: bias is systematic inaccuracy, while noise is the unsettling variability when different people assess the same situation.

Seeing Noise from Behind the Target

A crucial insight emerges: noise can be recognized and measured without knowing the true target. Imagine viewing only the backs of the shooting targets. You can’t tell if Team A or Team B is more accurate, but you instantly see the scatter in Teams C and D. Similarly, in human judgments, we can measure noise by examining variability alone, even when the correct answer is unknown. For instance, when doctors disagree on a diagnosis or executives vary in sales forecasts, their disagreement reveals noise, regardless of who is right.

Noise in the Wild: Startling Evidence

The chapter reveals scandalous levels of noise across critical domains. In medicine, different doctors diagnose the same patient with varying conditions. Child custody decisions depend heavily on which case manager is assigned. Professional forecasters disagree wildly on predictions, and asylum judges in the U.S. admit anywhere from 5% to 88% of applicants—a “refugee roulette.” Personnel decisions, bail rulings, forensic science, and patent grants all show high variability, undermining equity and efficiency. These examples underscore that noise is not a minor issue but a pervasive force with real-world consequences.

A Guide to the Journey Ahead

The book is structured in six parts to unpack noise and offer solutions. Part 1 investigates noise in criminal sentencing and insurance, introducing noise audits. Part 2 examines how to measure error in judgment. Part 3 explores predictive judgment and the noiseless advantage of algorithms. Part 4 delves into the psychology behind noise, explaining why people overlook it. Part 5 provides practical noise-reduction techniques, labeled decision hygiene. Part 6 considers the right level of noise, acknowledging that eliminating it entirely isn’t always feasible or desirable. Throughout, the narrative emphasizes that reducing noise can enhance fairness and save resources.

Key Takeaways

• Bias and noise are distinct errors: Bias is systematic deviation from truth, while noise is random scatter among judgments.
• Noise is measurable without knowing the truth: Variability alone exposes noise, making it assessable in uncertain situations.
• Noise has widespread, damaging effects: It causes unfairness in justice, inconsistency in business, and inefficiency in fields like medicine and forecasting.
• This book seeks to elevate noise in public awareness: By offering frameworks like decision hygiene, it aims to help organizations and individuals reduce noise for better outcomes.

Download Mindmap

If you like this summary, you probably also like these summaries...

The Psychology of Money

by Morgan Housel

Thinking Fast and Slow

by Daniel Kahneman

Same as Ever

by Morgan Housel

Noise

1. Crime and Noisy Punishment

Overview

This chapter begins by establishing a fundamental expectation of justice: similar crimes committed under similar circumstances should receive similar sentences. It then reveals the shocking reality that, for much of modern history, this ideal was routinely violated due to the vast discretion granted to individual judges. The chapter tracks the discovery of this systemic "noise"—unwanted variation in professional judgments—within the criminal justice system, the fierce advocacy to measure and reduce it through sentencing guidelines, and the complex, ongoing struggle between the desire for consistency and the value of judicial discretion.

The Problem of Unchecked Discretion

For decades, the legal establishment celebrated judicial discretion in sentencing as humane and just, believing it allowed judges to tailor punishments to the unique circumstances of each defendant and crime. This philosophy viewed rigid rules as dehumanizing. However, this widespread discretion created a system where the sentence a person received depended less on the facts of their case and more on the "views, predilections, and biases" of the particular judge assigned to it. This resulted in outrageous disparities where individuals convicted of virtually identical crimes could receive wildly different penalties, from probation to decades in prison.

Evidence of Systemic Noise

In the 1970s, Judge Marvin Frankel ignited a reform movement by publicly condemning these "arbitrary cruelties" as a failure of the rule of law. While his initial work was anecdotal, it prompted rigorous studies that quantified the problem. Researchers gave identical case files to dozens of judges and found "astounding" and "substantial disparity" in their sentencing decisions. In some experiments, there was no consensus even on whether incarceration was warranted. These controlled studies likely understated the real-world noise, as subsequent research showed sentences could be influenced by irrelevant factors like a judge's hunger, the outcome of a local football game, the defendant's birthday, or even the day's temperature.

The Push for Guidelines and Their Measurable Impact

Horrified by this evidence, Senator Edward Kennedy championed the Sentencing Reform Act of 1984, which created the U.S. Sentencing Commission. The Commission developed mandatory guidelines that used a grid system based on offense severity and criminal history to prescribe a narrow sentencing range. The goal was explicitly to reduce the "unfettered discretion" that caused noise. Multiple studies confirmed the guidelines were effective. They significantly reduced interjudge disparity, making sentence length less dependent on the "happenstance" of which judge was assigned. The guidelines curtailed not only noise but also certain biases, such as racial disparities in sentencing.

Criticism and the Return of Discretion

The guidelines faced intense criticism. Many judges argued they were too rigid and mechanistic, preventing them from delivering true justice tailored to a case's unique aspects. This debate culminated in a 2005 Supreme Court decision that made the guidelines advisory rather than mandatory. Most judges welcomed this change. However, data analysis in the advisory era revealed a clear consequence: interjudge disparity in sentencing doubled. Judges' personal characteristics—such as their gender, political affiliation, and individual propensity for harshness—began to influence outcomes more strongly, and racial disparities increased. The system had regressed toward the noisy state Judge Frankel had decried.

Key Takeaways

• Noise is pervasive in judgment: Complex, uncertain decisions—from sentencing to medicine to business—naturally produce disagreement, but the scale of this variability is often shockingly large and unjust.
• Noise has serious consequences: In criminal justice, system noise creates rampant unfairness, erodes the rule of law, and imposes high social and economic costs.
• Noise can be measured and reduced: The adoption of structured tools like sentencing guidelines demonstrated that systematic approaches can successfully decrease unwanted variability.
• Noise reduction is challenging: Efforts to impose consistency often face valid objections about the loss of flexibility, nuance, and human judgment. Balancing consistency with discretion remains a persistent and difficult conflict.

Download Mindmap

Noise

2. A Noisy System

Overview

This chapter introduces the concept of system noise through a revealing case study with an insurance company. It begins by describing how the authors' interest in noise was sparked not by dramatic events but by a practical consulting engagement. The core of the chapter explores a noise audit conducted at the company, which uncovered shocking levels of inconsistency in professional judgments—far beyond what executives anticipated. Through this example, the chapter illustrates how noise, often invisible in daily operations, can incur massive costs and undermine fairness in any organization where judgments are made.

The Hidden Cost of Inconsistency

The insurance company employed underwriters to set premiums and claims adjusters to estimate settlement costs. In both roles, cases were assigned essentially at random to available professionals, creating a "lottery" that determined the outcome. While some variability was expected, executives believed it would be minimal, guessing that differences between two experts on the same case would be around 10%. This level of noise was considered tolerable. However, the reality of this random assignment meant that the financial consequences for the company—and for customers—were subject to unwanted chance, with no one aware of the full extent of the problem.

Conducting a Noise Audit

To measure the actual variability, the company executed a noise audit. Detailed case descriptions were given to multiple underwriters and claims adjusters, who worked independently to provide estimates. The results were staggering. The median difference between two professionals was 55% for underwriters and 43% for adjusters—roughly five times higher than executives had predicted. For instance, if one underwriter quoted $9,500, another might quote $16,700 for the identical risk. Senior leaders quickly calculated that this noise likely cost the company hundreds of millions of dollars annually, due to both lost business from overpricing and losses from underpricing. The audit made visible a scatter of judgments that had previously been hidden, demonstrating that system noise was a major, expensive leak.

When Variability Is and Isn't Welcome

It's important to distinguish system noise from other forms of variability. In matters of personal taste—like film reviews or wine ratings—diversity of opinion is expected and even desirable. Similarly, in competitive settings such as multiple research teams racing for a vaccine or traders in a market, variation is essential for innovation and discovery. However, system noise occurs within an organization when a single, randomly chosen individual makes a binding judgment on its behalf. Here, consistency is the goal, and variability is unwanted. An example from an asset management firm showed that even among investors using the same data, estimates of a stock's fair value varied by 41%, highlighting that noise plagues many professional fields. Crucially, in such systems, errors do not cancel out; an overpriced policy and an underpriced one are both costly mistakes, just as inconsistent prison sentences represent dual failures of justice.

Why Noise Remains Unseen

A fascinating insight from the insurance study was that neither the professionals nor their leaders had previously recognized the enormous noise in their system. This blindness stems from an "illusion of agreement." In daily work, colleagues share a common language and professional norms, which fosters a belief that everyone sees the world similarly. Disagreements are often dismissed as rare lapses or avoided through organizational habits, like masking initial ratings in admissions processes to prevent conflict. Professionals grow confident in their own judgments through repetition and fluency, not through calibrating with peers. Without a mechanism like a noise audit, organizations can comfortably operate under the false assumption of consistency, only addressing egregious outliers in postmortems while missing the pervasive scatter in ordinary decisions.

Key Takeaways

• System noise—unwanted variability in professional judgments—is pervasive and costly in organizations, often remaining invisible without deliberate measurement.
• A noise audit, which involves multiple professionals evaluating the same case, can reveal shockingly high levels of inconsistency, far beyond what leaders typically anticipate.
• Noise is distinct from welcome diversity in tastes or competitive settings; it represents a problem when single, randomly assigned individuals make binding decisions.
• The illusion of agreement, reinforced by shared norms and conflict avoidance, often prevents organizations from recognizing their own noise until it is explicitly uncovered.
• Left unchecked, system noise leads to significant financial losses and unfair outcomes, as errors in individual judgments accumulate rather than cancel each other out.

Download Mindmap

If you like this summary, you probably also like these summaries...

The Psychology of Money

by Morgan Housel

Thinking Fast and Slow

by Daniel Kahneman

Same as Ever

by Morgan Housel

Noise

3. Singular Decisions

Overview

This chapter introduces a crucial distinction in judgment: the difference between recurrent decisions (made repeatedly by interchangeable professionals) and singular decisions (unique, one-of-a-kind choices). It tackles the challenging question of whether the concept of noise, so easily observed in the former, applies to the latter, ultimately arguing that it does, with significant implications for how we approach high-stakes, unique choices.

Defining Singular Decisions

Singular decisions are characterized by their uniqueness. They are made only once, lack a prepackaged response, and feature genuinely distinctive elements. Examples include a president's crisis response (like Barack Obama's handling of the 2014 Ebola outbreak), a military commander's fateful choice, or a personal decision to propose marriage or buy a specific house. While a continuum exists between singular and recurrent decisions, the extreme cases are meaningfully different. These decisions have traditionally been the domain of historians and management gurus, who analyze them through causal, hindsight-driven narratives, unlike the statistical, pattern-seeking approach used for recurrent decisions.

The Invisible Presence of Noise

The core challenge is that noise is defined as unwanted variability in judgments of the same problem. Since singular problems are never exactly repeated, we cannot directly observe or measure noise in them. However, this does not mean noise is absent. Just as a single shot from a noisy rifle team could have landed elsewhere due to an unsteady hand, a singular decision could have been different due to the inherent variability in human judgment.

The chapter advocates for counterfactual thinking to expose this noise. Consider the myriad factors that could have altered a singular outcome: if different experts with different backgrounds had advised the president, if the same facts were presented in a different order, or if key decision-makers were in a different mood. The wide variation in national responses to the COVID-19 pandemic provides observable evidence of noise; if only one country had been hit, the decision would have been just as noisy, but we would have had no variability to prove it.

Judgment as Imperfect Measurement

To understand why noise permeates all judgment, the chapter establishes a powerful analogy: judgment is a form of measurement where the instrument is the human mind. Whether a judge assigns a prison term, an underwriter sets a premium, or a doctor makes a diagnosis, they are assigning a value on a scale, with the goal of accuracy. Like any physical measuring instrument, the human mind is imperfect. Its errors consist of both bias (a consistent tendency to miss the true value in one direction) and noise (unwanted variability).

A simple stopwatch exercise illustrates this: attempting to tap out five consecutive ten-second intervals results in variable times (noise) and possibly an average that is systematically off (bias). Professional judgments are far more complex, but the same principles apply. When a judgment aims at a true value, variability means error—two different judgments cannot both be perfectly right.

Key Takeaways

• Singular decisions, though unique and non-repeatable, are still subject to the psychological noise that affects all human judgment.
• Noise in singular decisions is invisible but real. We must use counterfactual thinking to appreciate how different the same decision could have been under slightly different circumstances or with different people involved.
• This insight flips conventional wisdom. The best approach to a one-of-a-kind decision is not to treat it as entirely unique, but rather to view it as a recurrent decision that happens only once. Strategies that reduce noise in recurrent decisions should therefore improve singular ones.
• Judgment is measurement. Framing judgment as a mental act of measurement, with the mind as an imperfect instrument, clarifies that noise is a fundamental component of error, alongside bias.

Download Mindmap

If you like this summary, you probably also like these summaries...

The Psychology of Money

by Morgan Housel

Thinking Fast and Slow

by Daniel Kahneman

Same as Ever

by Morgan Housel