Source Attribution Training Set¶

Labeled training corpus — instantiates Use-Time Source Attribution Calibration

A curated corpus of real items whose true source class is already known, held as the gold reference that calibrates and teaches an attribution judgment — human or model.

A source-attribution judgment is only as good as the examples it was tuned on. Source Attribution Training Set is the artifact that supplies those examples: a curated corpus of real items — notes, claims, passages, traces — each carrying its verified source class as a gold label, assembled to be the reference standard against which an attributor is trained and measured. Its defining move is that it is a static, ground-truth artifact rather than a live judgment: it does not classify the next incoming item, it holds the settled cases whose true origin is already established, so that people and models can practice on them, be scored against them, and be recalibrated when they drift. Everything else in the archetype makes attribution decisions; this is the answer key those decisions are checked against.

Example¶

A team building an AI writing assistant wants it — and the humans reviewing it — to tag each drafted sentence by origin: retrieved from a document, user-supplied, inferred, or model-generated. To calibrate that, they build a training set. They sample a few hundred real passages drawn from the actual commingled work store, spanning the messy middle cases, not just clean ones. Each passage is hand-labeled with its true source class by tracing it back to the genuine record of where it came from. That labeled corpus becomes the fixed reference.

Now the set does its work. New reviewers label it blind; where their answers diverge from the gold labels, the gaps show exactly which classes they conflate — several keep tagging model-generated summaries as "retrieved," because both read fluently. The model's tagger is scored the same way, and every confirmed miss is fed back to sharpen the cues and prompts. The set itself never decides a live case; it is the yardstick that tells everyone else how well they are deciding.

How it works¶

Sample from real commingled use. Draw items from the actual store where sources are mixed — deliberately including the ambiguous and adversarial cases — so the set reflects the judgment's true operating distribution, not a tidy subset.
Label to the schema, from ground truth. Assign each item its source class using the domain's defined classes, and pin each label to verified provenance (a record, a custody trail, a confirmed origin) rather than to a labeler's guess.
Hold as the gold reference. Freeze the labeled corpus as the standard against which attributors are trained, scored, and compared — the fixed point that makes "accuracy" measurable at all.
Feed misses back. When a human or model attribution disagrees with a gold label and the gold label is upheld, route the confirmed error into cue design, prompts, and interface warnings — the set is the substrate the whole calibration loop turns on.

What distinguishes it is its stance: it is the reference data, not a live classifier, an analyzer, or a grader. Its value is entirely in the trustworthiness of its labels.

Tuning parameters¶

Sampling distribution — mirror live use faithfully versus over-weight the hard, confusable cases. Faithful sampling measures real-world accuracy; hard-case weighting trains discrimination but can flatter or scare depending on which way the mix is skewed.
Label provenance bar — how strong the evidence must be before an item's gold label is trusted. A high bar yields a smaller but sounder set; a low bar yields volume but risks baking labeler error into the answer key.
Set size versus curation depth — many lightly-checked items or fewer deeply-verified ones. Breadth improves coverage; depth protects the labels the whole loop depends on.
Refresh cadence — how often the set is re-sampled from current use. Frequent refresh tracks a drifting source mix; a frozen set is cheaper but silently ages.
Ambiguity handling — whether genuinely unclassifiable items are excluded, or kept with an explicit unknown gold label. Excluding them cleans the metrics but hides the cases attribution most often fails on.

When it helps, and when it misleads¶

Its strength is that it is the only thing that makes attribution quality measurable: without a corpus of known-source items, "our source tagging is good" is unfalsifiable. It exposes systematic conflations that no single judgment reveals, and it gives new people and new models a concrete, scored way to calibrate rather than absorbing attribution habits by osmosis.

Its failure modes are those of any gold set. If the labels are wrong, everything trained and scored against them inherits the error with false authority — a corrupt answer key is worse than none. If the set stops resembling live use, a judgment that scores well on it can fail in the field: distribution shift quietly turns yesterday's representative corpus into a misleading one.^[1] Its classic misuse is running backward — curating the set toward easy, unambiguous cases so the measured accuracy looks high, which optimizes the number instead of the judgment. The discipline that keeps it honest is to sample from real commingled use including the hard cases, to hold the label-provenance bar high, and to re-sample as the source mix drifts rather than defending a frozen set.

How it implements the components¶

Source Attribution Training Set fills the reference-data foundation of the archetype — the components that make attribution learnable and measurable rather than decided:

source_class_schema — every gold label is drawn from the domain's defined source classes, so the set operationalizes the schema as concrete exemplars.
commingled_item_inventory — its contents are sampled from the real mixed-source store, giving the archetype a bounded, enumerated inventory of items with settled origins.
attribution_feedback_loop — it is the substrate that loop runs on: the gold reference that scores attributors and against which confirmed errors are measured and fed back.

It does NOT apply the live use_time_attribution_rule (the Reality Monitoring Checklist does), compile the source_attribution_error_profile from live misses (the Source Confusion Matrix Review does), or attach a source_confidence_annotation to an item (the Source Attribution Confidence Rubric does).

Instantiates: Use-Time Source Attribution Calibration — this artifact is the ground-truth reference the archetype's judgments are trained and measured against.
Sibling mechanisms: Source Confusion Matrix Review · Reality Monitoring Checklist · Source Attribution Confidence Rubric · Memory Source Probe · Borrowed Idea Attribution Scan · Chain-of-Custody or Lineage Check · Generated Content Disclosure Gate · Hallucination Intrusion Triage · Observation Recheck or Replication · Provenance Lookup Before Publication · Source-Label Preserving Summary Template

Notes¶

The set and the Source Confusion Matrix Review are a matched pair: this artifact supplies the gold labels, and the review is what reads live attributions against them to find where the confusions cluster. Keeping the two separate is what lets the answer key be improved (better sampling, tighter labels) without re-running the whole error analysis, and vice versa.

References¶

[1] Distribution shift (dataset drift) — when the mix of cases a labeled set was built from stops matching the mix seen in live use, so a judgment that scores well on the set quietly stops performing in the field. It is why a source-attribution training set has to be re-sampled from real use rather than frozen once and trusted indefinitely. ↩