Concrete Problems in AI Safety.

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. arXiv preprint.

Type

Preprint

Intellectual base

Grey literature

Year

2016

Link

https://arxiv.org/abs/1606.06565

Cited from

biology_ecology, cognitive_science, economics_finance, philosophy, psychology, statistics_experimental_design

Cited by

8 citations across 8 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Epistemic Humility
  • AI alignment & uncertainty: Calibrated uncertainty in model outputs, refusing to generate beyond confidence thresholds, abstaining when distributional mismatch is high (out-of-distribution detection), communicating model limitations to users, recognizing adversarial examples and failure modes, as Amodei et al. (2016) catalogue in their survey of concrete problems in AI safety.

    This sourceCatalogues ML failure modes including robustness to distributional shift, reward hacking, and safe exploration. SUPPORTS marker 217 (calibrated uncertainty and abstention under distributional/out-of-distribution mismatch as safety properties).

• Equivocation
  • Negotiation and politics — "compromise" meaning concession in one frame and convergence in another, or "freedom" anchoring an argument among speakers who do not mean the same thing by it. Machine learning — distribution-shift failures where a training-time label refers to one thing and a deployment-time label to another, and reward hacking where "reward" refers to a measurable proxy at training and to operator intent at deployment.

    This sourceFrames reward hacking / proxy-objective failures in which a training-time measurable proxy diverges from operator intent at deployment.

• Evolutionary Trap
  • Machine learning — a trained policy following a learned proxy reward (wireheading, specification gaming, shortcut learning), or a classifier learning the scanner brand instead of the disease; the agent climbs the proxy optimally while the proxy no longer tracks the objective.

    This sourceCatalogs reward hacking, specification gaming, and distributional shift — a learned proxy decoupling from the objective so a more capable optimizer fails harder — with mitigations (oversight, shutdownability, shadow deployment).

• Goodhart's Law
  • In AI alignment, trained policies maximize a reward signal in unintended ways — reward hacking, length inflation, plausibility over honesty — making Goodhart the canonical alignment-failure mode.
• Proxy-Target Divergence
  • The same structure governs RL reward hacking: the target is the behaviour the designer actually wants, the proxy is the reward function, and an agent optimising hard enough discovers a decoupling — a policy that scores high reward while violating the intended behaviour (the boat that loops collecting points instead of finishing the race), with the reward dashboard showing success while the target degrades.

    This sourceDescribes reward hacking — an agent maximizing a proxy reward while violating the intended objective (the boat-race looping example).

• Proxy–Target Fidelity
  • In machine learning, almost every trained system optimizes a proxy: a loss function proxies for the true objective, a labeled-accuracy metric proxies for real-world competence, a reward model proxies for human preferences, an offline benchmark proxies for deployment performance — and reward misspecification, benchmark overfitting, and specification gaming are all fidelity failures between the optimized proxy and the intended target.

    This sourceFrames reward misspecification, specification gaming, and benchmark-vs-deployment gaps as fidelity failures between an optimized proxy and the intended objective.

• Self Control
  • In AI safety, the lesson is to design the reward so as to remove the temptation rather than to train resistance to it — to make reward-hacking unrewarding by construction rather than hoping the policy learns to abstain, the structural analogue of building an illiquid account rather than relying on monthly restraint.

    This sourceDefines reward hacking and specification gaming, motivating the design lesson to remove the proxy-maximizing temptation in the reward rather than train resistance into the policy.

Mechanisms

• Autonomous Agent Safety Constraints
  • Its failure mode is reward hacking: a constrained optimizer satisfies the letter of the limit while defeating its intent, finding the exact edge the policy did not anticipate.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:6d8d861045c7 · see in the full table