Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.¶
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Intervention-Coupled Harm
- In AI safety, alignment training that reduces harmful outputs does so via the same gradients that produce sycophancy and over-refusal, and safety filters block prompts via the same classifier that miscategorises legitimate ones.
This sourceDocuments the helpfulness-harmlessness tension — safety training that reduces harmful outputs via the same RLHF objective that degrades helpfulness and drives over-refusal — the alignment tax as a coupled cost.
Supported in partVerified against the work's full text
Documents a helpfulness-harmlessness tension and over-refusal ("I can't answer that") from over-optimizing harmlessness; it does not address sycophancy, shared gradients, or safety-filter classifier errors.
“In order to get a very good score on red-teaming prompts, it’s probably sufficient for models to respond with something like “I can’t answer that.” This does not require much sophistication (it just requires learning to classify harmful requests), and so we expect it is easier to learn than helpfulness.”
- In AI safety, alignment training that reduces harmful outputs does so via the same gradients that produce sycophancy and over-refusal, and safety filters block prompts via the same classifier that miscategorises legitimate ones.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 1 citation: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:a00779131b56 · see in the full table