Towards Monosemanticity: Decomposing Language Models with Dictionary Learning¶
Bricken, T., Templeton, A., & Batson, J. (2023). Towards Monosemanticity: Decomposing Language Models with Dictionary Learning: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Sparse Coding
- In machine learning, an L1 or KL sparsity penalty on hidden activations produces units with specific, interpretable triggers, and sparse autoencoders are now used to recover monosemantic features from the dense activations of transformer residual streams.
This sourceUses sparse autoencoders with an L1 penalty to recover human-readable monosemantic features from the dense, polysemantic activations of a transformer residual stream.
Supported in partVerified against the work's full text
Bricken et al. use an L1-penalised sparse autoencoder to get interpretable features, but on MLP activations of a one-layer transformer, not residual streams, and with no KL penalty.
“Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.”
- In machine learning, an L1 or KL sparsity penalty on hidden activations produces units with specific, interpretable triggers, and sparse autoencoders are now used to recover monosemantic features from the dense activations of transformer residual streams.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 1 of 1 citation: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:3ac620666163 · see in the full table