Prototype A/B or Multivariate Test¶
Experimental method — instantiates Affordance Shaping
Puts two or more candidate shapings in front of real agents at once and lets their measured behaviour decide which one actually moves the target action.
Prototype A/B or Multivariate Test settles a shaping question by experiment rather than argument. It builds two or more versions of an affordance — layouts, wordings, defaults, orderings — exposes real agents to them at random, and measures which version best produces the behaviour you are after. Its defining commitment is controlled comparison: because assignment is randomised and the variants run at the same time on the same population, the difference in outcome can be attributed to the design rather than to season, novelty, or luck. Where a desire path shows what people do in the wild and a contextual inquiry shows why, this mechanism asks a sharper, narrower question — which of these designs wins — and answers it with evidence you can act on.
Example¶
A charity can't agree whether a single suggested donation amount or a three-tier ladder gets more people to actually give. Rather than debate it, they run an A/B test: incoming visitors are split at random between the two pages, everything else held constant, and the system records not clicks but completed donations per visitor. A multivariate arm also crosses the amount layout with two button colours, to check for interaction. After enough traffic to tell signal from noise, the three-tier page shows a modestly higher completion rate — perhaps ≈ a tenth more — and the button colour turns out not to matter.
The output is a measured, real-behaviour verdict — this layout converts better here, by a margin large enough not to be noise — that replaces a meeting-room opinion. Crucially it measures the thing that pays (completed gifts), not a proxy anyone could argue about.
How it works¶
- Randomise and run concurrently. Assign agents to variants at random and expose them at the same time, so environment and population are held constant and the design is the only thing that differs.
- Measure the real target, not a proxy. Instrument the actual behaviour the shaping is meant to change — the completed action — captured as outcome traces per variant.
- Go multivariate when factors interact. Cross several design factors at once to catch effects that appear only in combination, at the cost of needing more traffic.
- Wait for separation, then decide. Let it run until the signal parts from the noise, and treat the winning variant as the fit-tested one.
Tuning parameters¶
- Number of variants — two (A/B) versus many (multivariate); more variants explore more of the design space but split the population thinner and need more agents.
- Success metric — which behaviour counts as the win, and how far downstream it sits; a proximal proxy (a click) is fast but gameable, a distal outcome (a completed, retained action) is truthful but slow.
- Sample size / run length — how much evidence before deciding; too little invites false winners, too much wastes exposure on a losing variant.
- Segmentation — whether to read one average effect or per-population effects; segmenting catches a design that helps one group and hurts another.
When it helps, and when it misleads¶
Its strength is that it replaces taste and seniority with measured behaviour, and it is the only sibling that establishes causation — that this shaping, and not something else, moved the action. It is especially strong at choosing among options that are all individually plausible.
Its failure mode is that an A/B test optimises the variants you thought to build, so relentless local tuning climbs the nearest hill and can miss the better design that was never in the test — the local-optimum trap.[1] It is easily corrupted by peeking at results and stopping the moment a variant looks like it's winning, which manufactures false positives; and run backwards, it becomes a search for any metric that flatters the design already chosen. It also measures only what it instruments, so a variant that wins on the proxy can quietly lose on the thing that mattered. The discipline is to fix the metric and sample size before starting, to feed the candidates from real observation rather than the whiteboard, and to keep asking whether the winning number is the outcome you actually wanted.
How it implements the components¶
Prototype A/B or Multivariate Test fills the experimental-validation side of the archetype — the components that test a shaping against real behaviour:
feedback_trace_capture— it instruments and records the outcome traces of each variant, the raw evidence the comparison runs on.ecological_fit_probe— by exposing variants to real agents in the live setting, it probes which design actually fits behaviour, quantitatively and causally.
It defines neither the goal it measures against (goal_and_harm_path_definition) nor the agent model — those are Contextual Inquiry or Walkthrough — and it invents no candidate designs; the variants it compares come from siblings such as Friction Adjustment, Signifier Prototyping, and Safe Default or Preselected Path. Its qualitative, non-randomised counterpart is Usability or Field Test.
Related¶
- Instantiates: Affordance Shaping — it is the experimental judge that decides which candidate shaping actually moves behaviour.
- Consumes: Contextual Inquiry or Walkthrough supplies the success metric it scores against; the shaping siblings (Friction Adjustment, Signifier Prototyping, Safe Default or Preselected Path) supply the candidate variants it compares.
- Sibling mechanisms: Contextual Inquiry or Walkthrough · Friction Adjustment · Desire Path Observation · Usability or Field Test · Affordance Audit · Physical or Digital Keying · Signifier Prototyping · Safe Default or Preselected Path · Wayfinding Marker · Task and Capability Analysis · Robot Action-Space Mapping
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Prototype A/B or Multivariate Test operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it puts two or more candidate shapings in front of real agents at once and lets their measured behaviour decide which one actually moves the target action.
Independent corroboration: The frozen evidence defines Prototype A/B or Multivariate Test as 'Puts two or more candidate shapings in front of real agents at once and lets their measured behaviour decide which one actually moves the target action', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Human-Computer Interaction
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Testing alternative prototypes with users is a characteristic HCI and usability-design practice.
Related originating lineages:
- Engineering & Design — Iterative prototyping supplied the candidate-artifact development lineage.
- Statistics & Experimental Design — Randomized A/B and multivariate comparison supplied the inferential design.
Review resolution: Both blind reviewers agree on human_computer_interaction as the primary origin. Explicit reconciliation resolves encyclopedia_synthesis_disagreement. The merged alternate lineages retain only domains the reviewers identified as materially formative; domain_reach=multi_domain records later applicability separately from origin breadth.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Birkett, A. "Local Maximum: What It Is, and How to Get Over It in A/B Testing". CXL (2022). Explains how iterative A/B testing can reach a local maximum and miss larger gains available only through innovative tests or redesign. registry ↩