Ecological Validity Screen¶
Validity check — instantiates Heuristic Calibration and Confidence Judgment
Tests whether the environment a heuristic runs in is learnable enough — regular cues, prompt feedback — to justify any confidence at all before calibration even begins.
An Ecological Validity Screen asks a prior question that most calibration work skips: is this the kind of environment where a fast rule could ever be trustworthy? Before you measure how well a heuristic's confidence matches its hit rate, this screen checks whether the world it operates in even supports skilled intuition — whether the cues are regular and predictive, whether feedback is prompt and unambiguous, whether the situation is stationary rather than adversarial or drifting. Its defining stance is that confidence can be structurally unearned regardless of track record: in a low-validity environment, a rule that looks accurate is running on luck and selective memory, and no amount of confidence-tuning will fix it. The screen is a go/no-go gate on whether calibration is meaningful at all, not a measurement of how calibrated the heuristic is.
Example¶
A hiring manager is convinced she can read a candidate's fit in the first ten minutes of an unstructured interview, and she wants to make her "gut confidence" the deciding factor for borderline candidates. Before anyone tunes those confidence calls, an ecological validity screen interrogates the environment. Are the cues valid — does anything observable in ten minutes of chat actually predict on-the-job performance? Is feedback prompt and clean — does she ever learn, months later, how her confident hires and rejects actually panned out, in a form she can attribute back to the specific cues she used? Is the setting stationary, or does each role and team differ enough that last year's read doesn't transfer? The screen finds the environment is low-validity: cues are weak, feedback arrives late and tangled with a dozen other factors, and roles vary. The verdict is not "your confidence is miscalibrated by X points" but "this environment cannot support confident snap judgments at all — bound the use case to structured signals, or don't authorize confidence here."
How it works¶
The screen profiles the environment against the conditions under which fast judgment can become skill. It asks three things. Cue validity: do the signals the heuristic keys on bear a real, stable relationship to the outcome, or are they salient-but-spurious? Feedback quality: is the outcome observed, soon, and attributable to the judgment — or is it delayed, noisy, or never seen, so the judge never learns? Regime stability: is the environment stationary and non-adversarial, or does it shift, so that yesterday's regularities dissolve? Each answer places the setting somewhere on a spectrum from "kind" (regular, prompt-feedback, stationary) to "wicked" (irregular, feedback-starved, shifting).[n1] The screen then narrows or forbids the heuristic's use case accordingly — permitting confident calls only in the sub-conditions that pass, and registering the rest as boundaries where confidence must be withheld.
Tuning parameters¶
- Validity bar — how strong the cue-outcome relationship must be to license confidence. A high bar keeps you honest in wicked settings but may forbid a heuristic that is genuinely useful at the margin.
- Feedback-lag tolerance — how delayed or noisy feedback may be before the environment is judged unlearnable. Loose tolerance keeps more heuristics in play; tight tolerance protects against illusory expertise.
- Regime-stability window — how far back the environment must have held steady to count as stationary. A long window demands proven stability; a short one adapts faster but is fooled by lulls.
- Use-case narrowing granularity — whether the screen passes/fails the heuristic wholesale or carves out specific valid sub-conditions. Fine carving preserves useful pockets but complicates the boundary register.
When it helps, and when it misleads¶
Its strength is that it catches the most expensive calibration mistake before you make it: pouring measurement effort into tuning confidence for a heuristic whose environment can never support skilled judgment. It reframes "how confident should we be?" as "should confidence here exist at all?" — and in wicked environments the honest answer protects everyone downstream.
Its failure mode is misjudging the environment itself: declaring a setting wicked when a valid signal was merely hidden, and so forbidding a useful heuristic — or, worse, declaring it kind on the strength of a stable-looking recent stretch that is about to break. The classic misuse is running the screen once and treating the verdict as permanent, when validity erodes as adversaries adapt or conditions drift. The guarding discipline is to re-screen when the environment changes, to demand evidence of cue validity and feedback quality rather than asserting them, and to keep the screen's job narrow: it licenses or forbids confidence structurally; it does not measure the calibration of confidence that has already been licensed.
How it implements the components¶
heuristic_use_case_definition— the screen's output narrows or forbids the use case, pinning down exactly where the heuristic is (and is not) allowed to carry confidence.reference_environment_profile— cue validity, feedback quality, and regime stability are the environment profile the screen builds and scores.boundary_condition_register— sub-conditions that fail the screen are written into the register as boundaries where confidence must be withheld or lowered.
It does not anchor a specific case to a comparison population's base rate — that outside-view estimate is Reference Class Comparison; the two both profile environment, but the screen asks whether the environment is learnable at all while the comparison asks what the reference population's outcomes were. And it does not measure realized hit rates against stated confidence — that requires heuristic_track_record_evidence, which lives in Confidence Bucket Review.
Related¶
- Instantiates: Heuristic Calibration and Confidence Judgment — supplies the prior go/no-go on whether the environment can support confident judgment.
- Sibling mechanisms: Calibration Adjustment Rule · Challenge Case Set · Confidence Bucket Review · Expert Disagreement Calibration · Low-Confidence Escalation Trigger · Post-Outcome Recalibration Review · Prediction Journal · Reference Class Comparison · Reliability Diagram or Calibration Curve
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Ecological Validity Screen operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it tests whether the environment a heuristic runs in is learnable enough — regular cues, prompt feedback — to justify any confidence at all before calibration even begins.
Independent corroboration: The frozen evidence defines Ecological Validity Screen as 'Tests whether the environment a heuristic runs in is learnable enough — regular cues, prompt feedback — to justify any confidence at all before calibration even begins', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Psychology
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Judgment psychology cohered ecological validity of expertise around environments with regular cues and prompt, accurate feedback rather than experience alone.
Related originating lineages:
- Cognitive Science — Research on kind versus wicked learning environments and expert intuition supplied the learnability criteria.
Review resolution: Both current reviews place ecological_validity_screen primarily in psychology; the reconciled classification retains only lineages that materially shaped the mechanism and keeps breadth of origin separate from reach.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The kind vs. wicked learning environments distinction, developed by Robin Hogarth, separates settings with regular cues and prompt, accurate feedback (where intuition can become skill) from those with irregular patterns and missing or misleading feedback (where experience breeds confidence without accuracy). The screen is the operational test of which one you are in — closely related to Kahneman and Klein's conditions for trustworthy expert intuition. ↩