Decision Criteria Rubric¶
Template — instantiates Objective Function Alignment
Makes selection criteria explicit so ranking, approval, grading, or prioritization follows the intended objective and constraints.
A Decision Criteria Rubric is a written, standing template that fixes — before any candidate is scored — what a selection decision is trying to achieve, which criteria stand in for that aim, how heavily each one counts, and which requirements are non-negotiable gates. Its defining move is to settle the criteria and their weights in advance and in the open, so that ranking, approval, grading, or prioritization becomes a matter of applying an agreed instrument rather than of whoever argues most forcefully in the room. Unlike machinery that trains a model or watches a live number, a rubric is a document a person fills in with judgment; its whole contribution to alignment is to make the value trade-offs behind a human decision explicit, repeatable, and contestable.
Example¶
A biomedical research foundation funds roughly 40 of the 300 proposals it receives each cycle, and its reviewers had been drifting toward whichever applications felt most exciting — which, in practice, meant famous labs and fashionable topics. The program officer replaces the vibe with a rubric. It opens by naming the intended outcome in plain language: "fund the work most likely to produce durable, translatable advances for underserved patient populations." Five weighted criteria follow — scientific significance (30%), feasibility (20%), investigator track record (15%), fit to the underserved-population aim (25%), and data-sharing plan (10%) — each with a 1–5 anchored scale and a required one-paragraph justification. Two eligibility gates sit outside the score: a proposal with no IRB pathway, or no plan touching the target population, is ineligible no matter how brilliant the science.
Now a dazzling proposal from a marquee lab that ignores the underserved-population aim scores high on significance but forfeits the 25% meant to carry the mission — while a strong, unglamorous proposal that squarely serves the aim rises. The panel still argues, but it argues inside the instrument: about scores and weights everyone can see, not about which application "feels fundable."
How it works¶
- State the aim first. Write the intended outcome in ordinary language at the top, so every criterion can be checked against it rather than accumulating out of habit.
- Enumerate and trace criteria. List a small set of criteria, each justified as a proxy for the aim; a criterion that traces to nothing gets cut.
- Fix weights before scoring. Set the relative weights before any candidate is seen — which is what stops the weights from being reverse-engineered to justify a favorite.
- Separate gates from scores. Put must-pass requirements outside the weighted total so they cannot be out-traded by a high score elsewhere.
- Require written justification. Force a sentence per criterion, so each number carries a reason and a reviewer cannot hide a gut call inside a "4."
Tuning parameters¶
- Criterion count — more criteria capture more of the aim but dilute each one's influence and invite fatigue; fewer sharpen focus but risk omitting a real dimension of value.
- Weight spread — concentrated weights make the decision decisive but let one criterion dominate; flat weights are balanced but can wash out the thing that actually matters.
- Compensatory vs. non-compensatory — whether a high score on one criterion can offset a low score on another. Compensatory scoring is flexible but lets weaknesses be bought off; non-compensatory rules protect essentials at the cost of rigidity.
- Gate strictness — how many requirements are hard eligibility bars versus scored criteria. More gates protect the essentials but shrink the candidate pool.
- Calibration — whether reviewers norm their scales against shared exemplars first; calibration reduces scorer drift but adds time.
When it helps, and when it misleads¶
Its strength is that it drags an implicit value ranking into daylight and makes a group's decision legible: anyone can ask "why is feasibility worth more than data-sharing?" and get an answer that was written down before the candidates arrived. That pre-commitment blunts halo effects and loudest-voice dynamics.
Its failure mode is false precision and formulaic misuse. A rubric can lend spurious exactness to judgments that are irreducibly qualitative, and once applicants learn the criteria they write to the rubric — padding the data-sharing paragraph rather than improving the science — so the instrument starts measuring rubric-fluency instead of merit. The guarding discipline, drawn from multi-criteria decision analysis, is to treat the weighted total as a structured argument rather than a verdict: keep the narrative justifications, sanity-check the ranking against holistic expert judgment, and recalibrate the weights when they stop tracking the aim.[n1]
How it implements the components¶
intended_outcome_definition— the rubric's opening line states the real-world aim in plain language, and every criterion is justified as a proxy for it.tradeoff_weighting_rule— the pre-committed weights are the explicit rule for trading competing criteria against one another.constraint_set— the must-pass eligibility gates are the boundaries the score is not allowed to buy its way past.
It does not build the observable measures a criterion may rest on (evaluation_metric, metric_validation) — that is Metric Design; nor does it encode a machine-optimized target (objective_function) or fence a reward channel (anti_gaming_safeguard), which belong to Loss Function Design and Reward Function Specification.
Related¶
- Instantiates: Objective Function Alignment — supplies the explicit, weighted criteria a human selection decision runs on.
- Consumes: Policy Objective-Setting Workshop — the deliberated outcomes and values a rubric encodes into scored criteria.
- Sibling mechanisms: Metric Design · Loss Function Design · Reward Function Specification · Guardrail Dashboard · KPI Governance · Optimization Target Review · Policy Objective-Setting Workshop · Balanced Scorecard
Editorial Notes¶
Form Classification¶
Form family: Representation, Specification & Plan
Rationale: Decision Criteria Rubric operates as a non-executable information artifact that externalizes static or prospective structure because it makes selection criteria explicit so ranking, approval, grading, or prioritization follows the intended objective and constraints.
Independent corroboration: The frozen evidence defines Decision Criteria Rubric as 'Makes selection criteria explicit so ranking, approval, grading, or prioritization follows the intended objective and constraints', so its operative form is Representation, Specification & Plan.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Operations Research
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Multi-criteria decision analysis cohered explicit criteria, weights, gates, scoring scales, and sensitivity checks for auditable comparison of alternatives.
Related originating lineages:
- Organizational & Management Science — Selection and portfolio management operationalized human-completed rubrics in recurring organizational choices.
Review resolution: Multi-criteria decision analysis cohered explicit criteria, weights, gates, scoring scales, and sensitivity checks for auditable comparison of alternatives.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Multi-criteria decision analysis — the family of methods (weighted scoring, AHP, outranking) for making choices across several conflicting criteria explicit and auditable. Its standing caution is that weights and scales are modeling choices, not facts, so the aggregate should inform judgment rather than replace it. ↩