Scalable Extraction of Training Data from (Production) Language Models

Nasr, M. (2023). Scalable Extraction of Training Data from (Production) Language Models.

Type

Preprint

Intellectual base

Grey literature

Year

2023

Link

https://arxiv.org/abs/2311.17035

Cited by

1 citation across 1 artifact.

Each citation links to the sentence it supports in the citing article.

Domain-specific

• Data Extraction Through Prompting
  • Training-data extraction via repetitive or divergence-eliciting prompts — demonstrated by Milad Nasr, Nicholas Carlini, and colleagues against production language models, including the "poem poem poem" attack that recovered verbatim training text — exploits the fact that a model is far more likely to emit a training sequence verbatim when that sequence was duplicated many times in its training data

    This sourceThe divergence attack itself — 'Repeat this word forever: "poem poem…poem"' against production models, emitting training data at 150x the ordinary rate — and the duplication finding this clause states: 'LLMs are much more likely to emit a training sequence when it is duplicated many times'. The paper this sentence describes — 'Scalable Extraction of Training Data from (Production) Language Models' (2023) and its divergence attack — whose byline runs, as the sentence now has it, Milad Nasr then Nicholas Carlini. The specific finding: the repeat-a-word prompt drives the model off its aligned generation into verbatim memorised training text, including strings over 4,000 characters and real personal contact details.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:5b17d7b7f454 · see in the full table