Weight Sensitivity Sweep¶
Analysis procedure — instantiates Objective Weighting Governance
Tests decision outcomes across plausible alternative weight sets.
A single weight set gives a single answer, and that answer can be an accident of the exact numbers chosen. Weight Sensitivity Sweep is the computational procedure that stress-tests a weighting rule by re-running the decision across many plausible alternative weight sets and recording how the outcomes move. Its defining idea is generative perturbation: it systematically varies the weights — one at a time, in correlated bundles, or by sampling a whole region of weight space — and produces the raw evidence of how the result responds. It does not decide whether the result is "stable enough" and does not render a verdict; it manufactures the data that such a judgment needs. The sweep is the engine; interpreting its output into a go/no-go on the ranking is a separate downstream job.
Example¶
A SaaS product team prioritizes its next-quarter roadmap with a scoring model over reach, impact, confidence, and effort. The current weights produce a ranked backlog, and leadership is about to commit. First, an analyst runs a Weight Sensitivity Sweep. Rather than trusting the one weight vector, the sweep samples 5,000 alternative weight sets drawn from ranges the team considers defensible — impact anywhere from 0.25 to 0.40, effort penalized more or less harshly — and re-scores the whole backlog under each. It uses a global sampling scheme rather than nudging one weight at a time, because the team suspects impact and confidence interact. The sweep's output is a distribution: for each feature, the share of weight sets under which it lands in the top ten. Feature A is top-ten in 98% of runs; feature G in 51%; feature K swings from rank 4 to rank 22 across the space. The sweep does not say what to do about K — it hands over the full response surface, from which the team's stability review will later read a verdict.
How it works¶
- Define the plausible region. The analyst bounds each weight to a range the organization would actually defend, so the sweep explores credible disagreement, not arbitrary numbers.
- Choose a sampling scheme. One-at-a-time perturbation is cheap and readable; global sampling across the joint space (e.g. quasi-random draws) catches interactions a one-at-a-time scan misses.
- Re-run and record. For each sampled weight set the full decision is recomputed and the outcome logged, building a response surface of how scores and orderings move.
- Emit raw response data. The output is the distribution of outcomes across weight space — deliberately un-judged — ready to be interpreted downstream.
Tuning parameters¶
- Perturbation range — how far each weight is allowed to vary. Wider ranges are more honest about uncertainty but can flag "fragility" that no one would actually defend.
- Sampling scheme — one-at-a-time versus global. OAT is fast and interpretable but blind to interactions; global sampling is thorough but costlier and harder to attribute.
- Sample count — how many weight sets are drawn. More samples smooth the response surface but cost compute.
- Joint vs. marginal variation — whether weights move independently or under realistic correlations. Correlated variation is more faithful when objectives aren't independent.
- Outcome metric logged — score, rank, or boundary crossing. Recording ranks and crossings (not just scores) keeps the sweep tied to decisions rather than magnitudes.
When it helps, and when it misleads¶
Its strength is that it replaces false confidence in one weight vector with an honest map of how the decision behaves across reasonable disagreement — the raw material for detecting weight-fragile rankings before they cause harm. Done with global sampling, it also exposes interaction effects that a naive check would miss.
Its failure mode is the one-at-a-time trap: varying weights individually, the classic shortcut, systematically understates fragility because it never explores the corners where two weights move together — a well-known limitation that global methods like Sobol sensitivity analysis[n1] were built to fix. A second misuse is drawing the "plausible" range too narrowly, so the rule looks robust only because it was barely perturbed. The guarding discipline is to justify the perturbation range explicitly, prefer joint over marginal variation when objectives interact, and remember the sweep only measures response — declaring the result acceptable is a judgment it deliberately leaves to others.
How it implements the components¶
weight_sensitivity_analysis— this procedure is the archetype's sensitivity analysis: systematically testing whether conclusions change under plausible alternative weights.objective_weight— it operates directly on the weights, treating them as the varied inputs whose perturbation drives the analysis.
It generates response data but does not judge it or convert it into a decision verdict: decision_impact_trace and legitimacy_rule — classifying winners as robust or fragile and ruling the ranking acceptable — belong to Ranking Stability Report; the sweep runs the experiment, its twin reads the result.
Related¶
- Instantiates: Objective Weighting Governance — supplies the perturbation engine that produces fragility evidence.
- Consumes: Weighted Scoring Model — it re-runs a concrete scoring model under each sampled weight set.
- Sibling mechanisms: Ranking Stability Report · Weighted Scoring Model · Multi-Criteria Decision Analysis · Weighted Sum Objective
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Weight Sensitivity Sweep operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it tests decision outcomes across plausible alternative weight sets.
Independent corroboration: The frozen evidence defines Weight Sensitivity Sweep as 'Tests decision outcomes across plausible alternative weight sets', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Universal
Rationale: OECD/JRC Handbook on Constructing Composite Indicators documents that composite-indicator practice requires explicit weights and sensitivity analysis because rankings can change with value judgments. This is direct, mechanism-specific evidence for statistics experimental design as the best-evidenced historical home of the operation—Tests decision outcomes across plausible alternative weight sets.—rather than evidence merely that the operation is useful there. The retained alternates record genuine adjacent lineages; later portability is represented separately by domain_reach=universal.
Related originating lineages:
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: tests decision outcomes across plausible alternative weight sets.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: tests decision outcomes across plausible alternative weight sets.
- Organizational & Management Science — Organizational Management supplies a historically relevant adjacent lineage or formative practice for the operation—Tests decision outcomes across plausible alternative weight sets.—but the adjudicated evidence more directly locates the defining lineage in statistics experimental design.
- Public Administration & Policy — Public administration, policy implementation, and program oversight supplies a parallel or contributing lineage for the mechanism's defining operation: tests decision outcomes across plausible alternative weight sets.
- Systems Thinking & Cybernetics — Systems science's feedback, boundaries, control, and regulation tradition contributes a separate formative lineage to the mechanism's weight sensitivity sweep logic.
Review resolution: The blind reviewers disagree on primary lineage (organizational_management versus statistics_experimental_design). The defining operation is: Tests decision outcomes across plausible alternative weight sets. The researched OECD/JRC Handbook on Constructing Composite Indicators establishes that composite-indicator practice requires explicit weights and sensitivity analysis because rankings can change with value judgments. That source therefore supports statistics experimental design as the historical origin. organizational management remains in the uncapped alternates where it contributes a formative practice, but application or governance is not itself proof of origin. origin_mode=single_lineage records lineage construction; domain_reach=universal separately records later applicability.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] Sobol sensitivity analysis is a variance-based, global method that apportions how much of a result's variation each input — and each interaction between inputs — is responsible for. It is the standard corrective to one-at-a-time analysis, which is why joint sampling is treated here as the more honest default. ↩