Dr. Wallace's Favorite Mental Models

Overfitting

Why a plan, model, or playbook that fits your past data perfectly can be worse than one that fits it only roughly — and how to tell which one you've built.

Data science & AI Decision making Pattern recognition ~9 min read

1Fast Overview

The idea in one line
A model, plan, or rule that matches a specific set of past cases so closely that it has absorbed their noise along with their signal — making it look impressively accurate on what already happened, and perform worse than a simpler version on what happens next.
Where it comes from
A core concept in statistics and machine learning, in use since at least the 1980s to describe models with more parameters or flexibility than the available data can justify. It is one half of the "bias-variance tradeoff" that every data scientist learns; the other half, underfitting, is the opposite failure.
The twist most people miss
A perfect fit to the past is often a warning sign, not an achievement. If a model, strategy, or rule explains 100% of what already happened, it has probably learned some of the randomness in that specific history — randomness that will not repeat in exactly the same way.
Why it matters at work
Anyone who builds a strategy, hiring process, template, or AI prompt from a small number of past cases — one breakout client, one winning quarter, one viral post — risks building something that only fits what already happened. Serious AI users should know this concept twice over: it governs how the models they rely on are trained and evaluated, and it names a trap in their own strategic thinking.

2Mental Model Compass

Field of originStatistics, machine learning & model fitting
Core mechanismA model or plan with too much flexibility for the data it is built on ends up fitting noise as if it were signal, trading robustness for a flawless fit to only the past
Best used forEvaluating AI model claims, hiring and forecasting processes, business strategy built on a few past wins, personal habits built on a handful of experiences
Watch out forUsing "overfit" as a reflexive insult against any decision based on experience, rather than actually checking for a signal-versus-noise problem

3Learn It: Step by Step

Step 1 — Two curves, one gap+
Train a flexible enough model on a set of cases and its accuracy on those exact cases will keep climbing, even to 100%. Its accuracy on new, unseen cases follows a different curve: it improves for a while, then peaks, then quietly gets worse even as training accuracy keeps rising. The widening gap between those two curves is overfitting happening in real time.
Step 2 — Signal and noise are mixed together+
Every real set of past cases contains two things tangled together: signal (the genuine, repeatable pattern) and noise (one-off randomness specific to those exact cases). A model that is complex enough can fit both perfectly. The trouble is that noise, by definition, does not repeat — so a model that has learned the noise has learned something that actively misleads it next time.
Step 3 — Why flexible models and narrow plans are the most exposed+
The more parameters a model has, or the more exceptions and special cases a plan accumulates, the easier it becomes to carve a shape that passes through every past data point exactly — including the noisy ones. A simpler model or a more general plan cannot bend that precisely, so it is forced to average over the noise instead of chasing it, which is usually a feature, not a limitation.
Step 4 — The same pattern outside a spreadsheet+
The identical mechanism shows up without any statistics software involved. A sales playbook built entirely around the exact messaging and timing that worked for one breakout client, a hiring rubric tuned to the precise quirks of one star employee, or a personal routine built around one unusually quiet month, are all overfitted: each one nails the case it was built from and struggles the moment anything is even slightly different.
Step 5 — The practical test+
The useful question isn't "does this explain everything that already happened?" It's "would this still work on a case I haven't seen yet, or does it only work because I built it to match the cases I already know?" If you cannot point to anything held back from the plan's construction to check it against, you cannot yet tell overfitting from a genuinely good fit.

4Examples Across Different Worlds

AI & machine learning
A model for predicting customer churn is tuned until it correctly classifies 99% of last year's customers. Deployed on this year's customers, accuracy drops sharply, because the model had learned quirks specific to last year's exact dataset rather than the durable signals behind churn.
Hiring
A company builds its interview scorecard entirely around the exact traits and answers of its single best-ever hire. The rubric screens out strong candidates who would succeed differently, because it is matching one person's profile rather than the underlying traits that predict success generally.
Business strategy
A retailer builds next year's entire marketing calendar around the exact promotions, timing, and messaging that drove one outstanding holiday season. When shopper behavior shifts even slightly, the tightly-tuned playbook underperforms a more general, flexible plan would have.
Personal productivity
Someone designs an elaborate time-blocking system around the precise, unusually quiet schedule of one slow month. The moment a normal month returns with its usual interruptions, the system breaks down completely instead of bending.
Investing
A trading strategy is adjusted and re-adjusted until it perfectly explains five years of historical price movements, including that period's specific quirks. It performs beautifully in backtesting and then loses money once it is run on live, future prices it was never tuned against.
Education & test prep
A student memorizes the specific quirks of one past exam — "the teacher always puts the hardest question third" — instead of learning the underlying material. A differently structured test from the same course catches them without any real preparation to fall back on.

5Worked Application

Situation: A growth team's one standout cold-email campaign last quarter converted far above average. Leadership wants the exact subject line, send time, and offer rolled out as the permanent company-wide playbook for every segment, immediately.

Applying the model: Before rolling it out unchanged, the team checks for overfitting by asking what in that campaign was signal (a genuinely transferable insight, such as leading with a concrete number instead of a vague benefit) versus noise (specific to that one list, that one week, that one competitor's outage that happened to boost response rates). They hold back two untouched segments as a genuine test rather than declaring victory on the original data alone. The general principle survives the test and gets adopted broadly; the exact wording and timing, which do not survive on the held-back segments, get treated as one example rather than the template.

ApproachWhat it optimizes forWhat happens on new data
Overfit-onlyCopy the exact winning email word-for-word to every segmentStrong on the original list, weak everywhere the noise doesn't repeat
Underfit-onlyGeneric, one-size messaging with no specifics at allMediocre everywhere; throws away the real signal along with the noise
Signal-checkedExtract the general principle, test it fresh before scaling itHolds up on new segments because it was never fit to their noise in the first place

This is a composite, illustrative scenario built to show the mechanism, not a specific published case study. The underlying signal-versus-noise distinction is a standard concept in statistics and machine learning; the exact split between signal and noise in any one real campaign requires looking at the actual data.

6Common Mistakes

Treating a perfect fit to the past as always good. A plan or model that explains 100% of what already happened is a flag worth checking, not automatically a trophy — perfect past fit is exactly what memorizing noise looks like.
Using "overfitting" as a vague insult. Calling any experience-based decision "overfit" without checking for an actual noise-versus-signal problem turns a precise diagnostic term into a lazy putdown.
Confusing it with "too much data." Overfitting is about model complexity relative to the data available, not data volume. More good data generally reduces overfitting; it does not cause it.
Ignoring that underfitting is just as real a failure. Overcorrecting toward "keep everything as simple as possible" throws away genuine signal too, and can perform just as poorly as the overfit version it was meant to fix.
Never checking against anything held back. Without a genuine holdout case, a fresh segment, or a new time period, there is no way to actually distinguish overfitting from a model or plan that is simply good.
Assuming the fix is always "add more rules." Patching every new failure with another special-case exception is itself an overfitting spiral; sometimes the fix is removing rules, not adding them.

7When Not to Use It

When the past genuinely is representative. In a stable, slow-changing environment, a close fit to history is not automatically overfitting — sometimes conditions really do stay that stable.
When there is no way to check against anything new. Without any holdout case or fresh data to test against, labeling something "overfit" is a hunch, not a diagnosis.
When the real problem is bad data, not model complexity. Garbage or biased input undermines any model, simple or complex alike — that is a data-quality problem, and simplifying the model will not fix it.

9Try It Yourself: Prompt Builder

Answer a few questions and build a ready-to-use prompt for checking whether a plan, model, or rule you're relying on is overfit to a narrow set of past cases.

Your generated prompt will appear here.
Copied!

10Save as Skill

Preview
Apply to My Problem

This is the portable skill version of this model — written so you can hand it to any AI assistant and get the same thinking applied to your own plans, models, and decisions.


      
      
      Copied!
    

    

      
      
      
      
Your generated prompt will appear here.

      
      Copied!