Diagnostic Plot Review¶
Visual diagnostic — instantiates Assumption-Light Inference
Reads fitted-data graphics to see whether a method's distributional and scale assumptions actually hold, catching violations a summary statistic hides.
A summary number can look perfectly healthy while the assumption underneath it is broken. Diagnostic Plot Review is the practice of looking — plotting the data and the model's residuals and reading, by eye, whether the distributional and scale assumptions the analysis relies on are actually honored. Where an audit checklist lists assumptions in words ahead of time, this mechanism confronts those assumptions with the fitted data: a quantile-quantile plot for normality, a residual-versus-fitted plot for constant variance and independence, a histogram or scale check for skew and censoring. Its defining move is that the graphic is the evidence — a heavy-tailed QQ plot or a funnel-shaped residual cloud is a direct sighting of an assumption violation, and it is also the signal that reopens the audit and forces a different method.
Example¶
An environmental agency fits an ordinary linear trend to two years of hourly fine-particulate readings from a roadside monitor to report whether pollution is declining. The slope is negative and the confidence interval looks tight. Before publishing, an analyst runs the plot review. The QQ plot of residuals bends sharply upward at both ends — the errors are heavy-tailed, not normal. The residual-versus-fitted plot shows a fan: variance grows on high-pollution days. And a simple time plot reveals that a wildfire week produced a cluster of extreme readings the linear fit is straining to accommodate.
None of this appeared in the slope or its standard error. The graphics did what the number could not: they showed that the interval's tidy width rests on a normal-errors assumption the data visibly break. That sighting is the trigger — it sends the analysis back to the audit and toward a distribution-free comparison and a robust summary, and it turns "the trend is significant" into "the trend is suggestive but the error model doesn't hold, so read the interval loosely."
How it works¶
The review is a fixed rotation of plots, each aimed at a specific assumption:
- Match plot to assumption. QQ plot → distributional form; residual-vs-fitted → homoscedasticity and mean structure; residual-vs-order or autocorrelation plot → independence; histogram or empirical CDF → skew, multimodality, censoring.
- Read for shape, not points. The signal is the pattern — a bend, a fan, a trend in residuals — not any single dot. Influential single points are noted but the structural verdict comes from the shape.
- Check the scale. Confirm the measurement scale the method presumes is the scale the data actually carry (interval where distances are meaningful, not ordinal dressed as interval).
- Trigger or clear. A clean set of plots clears the method to proceed; a violated one is logged as a review trigger that reopens the assumption audit and redirects method choice.
Tuning parameters¶
- Plot panel — which diagnostics are standard for this analysis. A wider panel catches more violation types but takes longer and tempts over-reading of noise.
- Reading threshold — how pronounced a pattern must be before it counts as a violation. A strict threshold avoids chasing every wiggle; a lax one over-rejects perfectly usable models.
- Formal backstop — eyeball only, or pair each plot with a numeric check (a formal test or an influence measure). Numbers curb subjective reading but reintroduce their own assumptions.
- Reviewer independence — self-review versus a second reader. A fresh reader catches the pattern the analyst is motivated not to see, at the cost of coordination.
When it helps, and when it misleads¶
Its strength is speed and honesty: a good residual plot exposes a broken assumption faster and more convincingly than any test, and it is the cheapest possible guard against reporting a precise-looking result built on a fit that never held. Anscombe's quartet — four datasets with identical means, variances, and regression lines but wildly different shapes — is the standing proof that you cannot trust summary statistics without looking at the picture.[1]
Its failure mode is subjectivity in both directions: an eager reviewer sees violations in random noise and rejects a fine model, while a motivated one squints past a real funnel. Small samples make every plot ambiguous; large ones make trivial, decision-irrelevant departures look alarming. The classic misuse is treating a clean plot as proof the assumption holds — absence of a visible pattern is weak evidence, not a certificate. The guarding discipline is to fix the plot panel and reading thresholds in advance (so the review is not improvised to suit the result) and to treat the plots as a trigger for method change, not as a licensing stamp.
How it implements the components¶
evidence_scale_alignment— the plots are precisely how the method is kept honest about what the data can express: a QQ or scale plot shows whether the distributional and measurement assumptions the procedure presumes match the evidence in hand.assumption_review_trigger— a violated diagnostic is the signal that reopens the audit; the review is the standing tripwire that fires when the data contradict a listed assumption.
It sees whether assumptions hold but does not enumerate or rank them. It does NOT implement assumption_audit or consequential_assumption_boundary — those belong to Assumption Audit Checklist, the paper-first ledger this review confronts with the fitted data.
Related¶
- Instantiates: Assumption-Light Inference — supplies the visual evidence that a listed assumption is or isn't honored.
- Consumes: Assumption Audit Checklist — the checklist names which assumptions to plot for; the review checks them.
- Sibling mechanisms: Assumption Audit Checklist · Model Comparison Table · Median-Based Summaries · Bootstrap-Like Checks · Nonparametric Tests · Permutation Tests · Rank-Based Methods · Robust Statistics
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Diagnostic Plot Review operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it reads fitted-data graphics to see whether a method's distributional and scale assumptions actually hold, catching violations a summary statistic hides.
Independent corroboration: The frozen evidence defines Diagnostic Plot Review as 'Reads fitted-data graphics to see whether a method's distributional and scale assumptions actually hold, catching violations a summary statistic hides', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Statistical model diagnostics cohered visual review of residual, quantile, scale, and distribution plots to test assumptions hidden by summaries.
Related originating lineages:
- Data Science & Analytics — Exploratory data analysis operationalized routine plotting before model acceptance.
Review resolution: Statistical model diagnostics cohered visual review of residual, quantile, scale, and distribution plots to test assumptions hidden by summaries. The retained alternate lineages materially shaped the mechanism's form.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Anscombe's quartet (Francis Anscombe, 1973) — four small datasets with near-identical means, variances, correlations, and fitted regression lines but radically different shapes when plotted, one dominated by a single outlier. It is the canonical demonstration that summary statistics can hide the very structure a plot makes obvious. withdrawn registry ↩