Skip to content

Typicality Rating Exercise

Elicitation method — instantiates Prototype-Centered Category Modeling

Has people rate how typical each example is of a category, turning intuition into a graded ranking that surfaces the clearest anchors and the fuzzy middle.

A Typicality Rating Exercise asks people how typical each example is of a category — usually a rating on a short scale, collected across several raters and aggregated. Its distinctive move is that it elicits the category's graded structure empirically from human judgment rather than assuming it, computing it, or drawing it. The question is deliberately graded — "how good an example is this?" — not binary — "is it in or out?" — and that is what makes internal structure appear: which examples are central (the anchors) and how membership grades down through a soft middle to a fringe. It produces the raw graded signal that a rubric later explains, a classifier consumes, and a map plots.

Example

A nutrition app is building a "breakfast food" category that resists a hard yes-or-no. It asks a panel to rate how typical each of two hundred foods is of "breakfast food" on a 1-to-7 scale. Oatmeal and cereal come back around 6.8 — the anchors. Eggs sit near 6.2, a bagel around 5.5, leftover pizza around 2.5 (eaten at breakfast, but rarely typical of it), a raw onion near 1.1. The aggregated ranking hands the app a graded membership scale and a clear anchor set at once, and reveals that "breakfast food" has a long, culturally variable middle rather than a clean boundary. This mirrors the classic finding that people rate a robin a more typical bird than a penguin even though both are fully, unquestionably birds.[1]

How it works

  • Ask for typicality, not membership. The prompt is "how good an example is this?", which surfaces graded internal structure; asking "is this in the category?" would collapse exactly the gradient you want to see.
  • Aggregate across raters. Averaging several raters cancels idiosyncrasy and — just as usefully — exposes disagreement, which flags the contested regions of the category.
  • Rank to find anchors and zones. The top of the ranking are the prototype anchors; the shape of the distribution defines the graded scale and its core, middle, and fringe zones.

Tuning parameters

  • Scale design — number of points, labels, and whether "not a member at all" is kept distinct from "an atypical member." Separating validity from typicality matters: a penguin is atypical yet fully a bird.
  • Rater panel — who rates, and how many. Experts, laypeople, or the actual downstream users each capture a different prototype; the panel choice quietly decides whose category you are modeling.
  • Aggregation — mean, median, or full distribution, and whether disagreement is reported and acted on rather than averaged away.
  • Item sampling — how the rated examples are chosen; a skewed item set drags the discovered center with it.
  • Instruction framing — "typical of," "a good example of," or "first thing that comes to mind" are subtly different constructs that yield different rankings.

When it helps, and when it misleads

Its strength is that it cheaply converts a fuzzy category into an evidence-based graded structure, discovers the actual anchors instead of a designer's guess, and exposes internal variation and contested zones that a binary label hides. Its central failure mode is the confusion of typicality with validity: the exercise ranks how central an example is, not whether it counts, and treating a low typicality score as low membership demotes legitimate-but-atypical members — the penguin problem, and a genuine fairness hazard when the atypical members are people. Ratings also inherit the raters' shared assumptions, so a homogeneous panel can encode a biased prototype as simply "what's typical." The classic misuse is reading the average typicality rating as the membership decision. The discipline that guards against this is to keep typicality and membership as separate questions, to diversify the panel, and to route the boundary decision to a policy or review panel rather than reading it straight off the ratings.

How it implements the components

Typicality Rating Exercise fills the components an elicitation can produce from human judgment:

  • prototype_anchor_set — the top-rated examples are the empirically identified anchors, discovered from raters rather than asserted by a designer.
  • graded_membership_scale — the aggregated ratings are the graded scale, with its core, middle, and fringe read off the distribution.

It does not render that structure spatially — the Prototype Embedding Map does — nor name the dimensions behind the ratings, which is the Similarity Dimension Rubric; the boundary cases belong to the Near-Miss Comparison Set.

Notes

Typicality rating and the sibling Card Sort or Example Sort are the two elicitation methods here, and they answer different questions: rating captures graded centrality (how good an example each case is), while sorting captures grouping and dimensions (which cases belong together, and along what features). They are often run together — a sort to find the structure, a rating to grade it.

References

[1] Eleanor Rosch's work on categorization found that categories have graded internal structure — some members are consistently rated more typical than others (a robin more typical a bird than a penguin) — even when membership itself is not in doubt. Typicality ratings are the classic instrument for eliciting that structure. ↩