Data Extraction Through Prompting¶

The AI-security failure mode in which an adversary crafts inputs that make a deployed language model emit private content — training text, system prompts, retrieved documents — by exploiting the legitimate inference interface, whose breadth exceeds the access-control envelope around the data.

Core Idea¶

Data extraction through prompting is the AI-security failure mode in which an adversary crafts inputs to a deployed language model that cause it to emit information the operator intended to remain private — verbatim training-set text, memorised personally identifiable information, system-prompt instructions, the contents of retrieval-augmented documents, or upstream API credentials — by exploiting the model's legitimate inference interface rather than by breaching storage systems or model weights. The structural feature that makes this possible is the absence of an enforceable boundary inside the model between what it has internalised or been instructed with and what it is willing to output: the same generation mechanism that produces helpful responses can, under adversarially designed inputs, produce emissions of content the operator intended to withhold. Named techniques instantiate this structure in different ways. Training-data extraction via repetitive or divergence-eliciting prompts — demonstrated by Nicholas Carlini and colleagues against production language models, including the "poem poem poem" attack that recovered verbatim training text — exploits the fact that frequently repeated training samples are more easily reconstructed from model activations. System-prompt leakage via role-play, summarisation, or recursive instruction attacks exploits the model's tendency to treat system-context text as content it can reason about and paraphrase. Prompt injection embeds a "reveal" or "ignore previous instructions" directive inside retrieved documents, web pages, or tool outputs that the model processes as authoritative context, causing it to follow the injected instruction rather than the operator's original instructions. The shared structure across all variants is that the query interface is broader in what it can be made to emit than the access-control envelope the operator has placed around the data: the model's capacity to generate responses conditioned on its full context — training data, system prompt, retrieved documents — is what creates the exfiltration surface, and that capacity cannot be selectively disabled for adversarial inputs without degrading legitimate function.

Structural Signature¶

Sig role-phrases:

the deployed model — the trained model that has internalised training data, system instructions, or retrieved content the operator intends to keep private
the query interface — the legitimate inference channel accepting adversary-crafted prompts and returning adversary-readable responses
the access-control envelope — the boundary the operator has drawn around the data, narrower than what the interface can be induced to emit
the no-internal-boundary fact — the structural absence of any enforceable line inside the model between what it is permitted to know and what it is permitted to say
the crafted prompt — the adversarial input exploiting divergence/repetition, role-play, recursion, or injected instructions to elicit protected content
the emission — the protected content (verbatim training text, PII, system prompt, retrieved documents, credentials) carried back out through the sanctioned output channel
the query-chaining move — the attacker triangulating protected content across turns, since each query reveals more than intended
the structural irreducibility — the gap being uncloseable because the conditioning capacity that creates it is the same capacity the helpful behaviour depends on
the defense locus — output filtering, retrieval-source gating, and training-data sanitisation (not the refusal-and-policy layer where jailbreak defenses live)

What It Is Not¶

Not a storage breach. No system is broken into: the weights, the vector store, and the API credentials remain intact, and the leak occurs through the model's ordinary, sanctioned generation at the inference interface. Modeling the threat as a storage-security problem — encrypt, lock down, gate — misses where the disclosure actually happens, which is the query interface emitting content the operator meant to withhold.
Not a patchable bug. The gap between what the model is permitted to know and what it can be induced to say is structural, not a defect awaiting a fix. The capacity to condition responses on training data, system prompt, and retrieved context is the same capacity the helpful behaviour depends on, so it cannot be selectively disabled for adversarial inputs without degrading legitimate function — "switch it off for attackers" is infeasible by construction, not merely unimplemented.
Not jailbreaking. Despite the frequent conflation, this is about information disclosure, not the bypass of a behaviour policy; a jailbreak can be the vehicle for extraction but is a different failure mode with a different defense locus. Hardening the refusal-and-policy layer where jailbreak defenses live leaves the exfiltration surface — output filtering, retrieval-source gating, training-data sanitisation — untouched.
Not closable by an output blocklist. Because the space of crafted inputs is open-ended, any enumerated list of known techniques (divergence prompts, role-play, "ignore previous instructions") will be evaded by novel inputs; the honest answer to "what can the model be made to say about everything it has access to?" is essentially anything, modulo cost. The surface can be narrowed but not closed, so a defense premised on exhausting the attack catalogue is structurally incomplete.
Not the model choosing to leak. The disclosure is not a misbehaving intent or an alignment lapse the model elected; it follows from the absence of an enforceable internal boundary between internalised/instructed content and permitted output. Reading it as the model "deciding" to reveal secrets mislocates the cause in the model's disposition rather than in the structural fact that its generation channel is broader than the access-control envelope around its data.

Scope of Application¶

Data extraction through prompting lives across the subfields of AI security and red-teaming; its reach is within LLM-substrate information-disclosure security, bounded by the crafted-prompt techniques and the in-a-generation-model fact that there is no internal boundary between what the model may know and may say. The legitimate-output-channel-leaks-protected-information shape it instances travels under side_channel_attack (with access_control/trust_boundary/query_interface); a hardware timing leak or a moderation-policy leak is that shape in another substrate — not a "prompting" attack — and stays out of the map.

LLM red-teaming — training-data-extraction benchmarks (Carlini et al., the poem-poem-poem divergence attack) and the OWASP LLM Top-10 "Sensitive Information Disclosure" and "Prompt Injection" categories.
Deployed-product security — customer-support assistants leaking internal documents, employee chatbots leaking salary data, and RAG systems leaking vector-store contents under crafted prompts.
Agent and tool-use security — prompt injection in retrieved content tricking an agent with file or API access into exfiltrating those resources to an attacker channel.
Model-IP protection — recovery of system prompts or behavioural cloning of proprietary models through the inference interface revealing more than intended.
Regulated-deployment compliance — guarding against PHI extraction under HIPAA, MNPI leakage in finance, and classified-content elicitation in defense, where the same interface-versus-envelope gap is the threat.

Clarity¶

Naming this failure mode relocates the data-privacy question from where security engineers instinctively look. The reflex is to treat private data as a storage problem — encrypt the weights, lock down the vector store, gate the API credentials — but extraction through prompting shows the leak occurs at the inference interface, through the model's ordinary, sanctioned generation behaviour, with no storage system breached. The clarifying move is to make explicit that the access-control envelope an operator has drawn around the data is narrower than the set of things the query interface can be induced to emit, and that this gap is structural rather than a bug to be patched: the very capacity to condition responses on training data, system prompt, and retrieved context is what the helpful behaviour depends on, so it cannot be selectively switched off for adversarial inputs without degrading legitimate function. The sharper question a defender can now ask is not "is this data stored securely?" but "what can the model be made to say about everything it has access to?" — and the honest answer reframes the threat model entirely.

The concept also earns its keep by separating extraction from the sibling failure modes it is constantly conflated with, because the diagnostic boundary dictates where the defense must live. It is not jailbreaking, which bypasses a behaviour policy; extraction is about information disclosure, even though a jailbreak can be the vehicle for it. It is not data poisoning, which corrupts training inputs to alter future behaviour; extraction recovers inputs after the fact. It is not membership inference, which only establishes whether a record was in the training set; extraction reproduces the record itself. Holding these apart tells the practitioner that extraction defenses belong at output filtering, retrieval-source gating, and training-data sanitisation — not at the refusal-training and policy-classification layer where jailbreak defenses live — so that an operator who hardens only the refusal layer has left the exfiltration surface open.

Manages Complexity¶

The attack surface, enumerated, is unbounded and ever-growing: divergence prompts that recover verbatim training text, role-play and summarisation tricks that leak a system prompt, recursive "ignore previous instructions" directives, injected commands buried in retrieved documents or tool outputs, and an open-ended space of future crafted inputs no blocklist can exhaust — each a separate exploit with its own mechanics. Naming data extraction through prompting collapses that catalogue onto a single structural invariant: the query interface is broader in what it can be induced to emit than the access-control envelope drawn around the data, because the model has no internal boundary between what it is permitted to know and what it is permitted to say. The defender then stops chasing each technique and tracks one gap, asking one diagnostic question — "what can the model be made to say about everything it has access to?" — whose answer (essentially anything, modulo cost) is read straight off the invariant rather than re-derived per exploit. That same collapse fixes the other load-bearing parameters cheaply. It locates the leak: not at storage (weights, vector store, credentials) but at the inference interface, so encryption and lockdown are seen to miss it before they are tried. It tells the defender the gap is structural, not a patchable bug, because the conditioning capacity that creates it is the same one the helpful behaviour needs — so "disable it for adversarial inputs" is recognised as infeasible up front. And it sorts the defense by a clean boundary against the sibling failure modes it is confused with (jailbreak = policy bypass, poisoning = corrupting inputs, membership inference = mere presence), which places extraction defenses at output filtering, retrieval-source gating, and training-data sanitisation rather than the refusal layer. An unbounded zoo of prompts and a tangle of adjacent threats reduce to one interface-versus-envelope gap, one diagnostic, and a fixed defense locus — from which the threat model, its irreducibility, and where to defend all follow.

Abstract Reasoning¶

Within AI security and red-teaming the concept licenses reasoning moves that all run on the gap between the access-control envelope and the breadth of the query interface.

Diagnostic — when private content surfaces, infer that the leak is at the inference interface, and classify which failure mode produced it. The signature move resists the storage reflex: confronted with a model emitting a verbatim training passage, a salary figure, or its own system prompt, the analyst reasons FROM "no storage system was breached — the weights, vector store, and credentials are intact" TO "the disclosure occurred through ordinary sanctioned generation, conditioned on context the operator meant to withhold." A second diagnostic move reads the emission's character back to a technique: a divergence-eliciting or repetitive prompt that recovers exact training text implicates training-data extraction (and points to a frequently repeated, more-easily-reconstructed sample); a role-play or summarisation prompt that paraphrases hidden instructions implicates system-prompt leakage; a "reveal" or "ignore previous instructions" directive followed from inside a retrieved document implicates prompt injection. The reasoning is FROM the observed leak and the input that caused it TO the interface as the locus and the specific variant at work — which a generic "data was exposed" verdict cannot supply.

Interventionist — place the defense at the locus the gap dictates, and predict what each control will and will not close. Because the leak lives at the inference interface, the move is to prescribe controls there and predict their reach: output filtering, retrieval-source gating, and training-data sanitisation are the levers, and the analyst reasons FROM "the exfiltration surface is the model's conditioning on training data, system prompt, and retrieved context" TO "harden these output and input-context layers, not the refusal-and-policy layer where jailbreak defenses live." The sharper interventionist prediction is negative: reasoning FROM "the conditioning capacity that creates the gap is the same capacity the helpful behaviour depends on" TO "it cannot be selectively disabled for adversarial inputs without degrading legitimate function," so the analyst predicts that any 'patch' aiming to switch off the capacity will fail, and that an operator who hardens only the refusal layer leaves the exfiltration surface open. The move is FROM the structural locus TO a defense placement plus a forecast of its limits.

Boundary-drawing — separate extraction from its sibling failure modes, and permitted-to-know from permitted-to-say. A first boundary move holds extraction distinct from the threats it is constantly conflated with, because the distinction dictates where the defense must live: it is not jailbreaking (a behaviour-policy bypass, though a jailbreak can be the vehicle for disclosure), not data poisoning (which corrupts inputs to alter future behaviour, whereas extraction recovers inputs after the fact), not membership inference (which only establishes whether a record was present, whereas extraction reproduces the record itself). The analyst reasons FROM the signature of the attack TO its category TO the matching defense locus. A second, deeper boundary move names the structural fact the model lacks: there is no enforceable internal boundary between what the model is permitted to know and what it is permitted to say, so the analyst reasons FROM "the access-control envelope is narrower than what the query interface can be induced to emit" TO "this gap is structural, not a bug" — refusing to model the problem as a closable defect.

Predictive — anticipate query-chaining and the residual surface no blocklist exhausts. A forward move predicts adversary behaviour across turns: because the interface reveals more than intended per query, an attacker can chain queries to triangulate protected content, so the analyst forecasts multi-query exfiltration rather than single-shot leaks and sizes defenses accordingly. A second predictive move concerns coverage: because the space of crafted inputs is open-ended, the analyst predicts that any enumerated blocklist of known techniques will be evaded by novel inputs, and that the honest answer to "what can the model be made to say about everything it has access to?" is essentially anything, modulo cost — so the defender reasons FROM the unbounded input space TO the conclusion that the surface can be narrowed but not closed.

Knowledge Transfer¶

Within AI security and red-teaming the concept transfers as mechanism, intact. The interface-versus-envelope gap, the storage-reflex-refusing diagnostic, the emission-character-to-technique classification, the defense-locus prescription (output filtering, retrieval-source gating, training-data sanitisation rather than the refusal layer), and the negative prediction (the conditioning capacity cannot be selectively disabled) all carry without translation across the home substrate: LLM red-teaming (training-data-extraction benchmarks, the OWASP LLM Top-10 categories), deployed-product security (support assistants leaking internal docs, employee chatbots leaking salary data, RAG systems leaking vector-store contents), agent and tool-use security (prompt injection in retrieved content exfiltrating file or API resources), model-IP protection, and regulated-deployment compliance (PHI under HIPAA, MNPI in finance, classified content in defense). Across these the data and the deployment change but the structure, the diagnostic, and the defense placement do not; the home domain is LLM-substrate information-disclosure security as a whole.

Beyond the LLM substrate the right reading is the shared abstract mechanism, not "data extraction through prompting" travelling. Stripped of model vocabulary the pattern is side_channel_attack: a legitimate output channel of a system reveals information the access-control model intended to protect, decomposed as side-channel + query-interface-as-attack-surface + a trust-boundary violation between intended and hostile use (access_control, trust_boundary, query_interface). That pattern genuinely recurs as co-instances across substrates with no prompting in them: timing, power-analysis, cache-timing, and electromagnetic-emanation attacks in hardware security; traffic analysis in networking; conversation patterning in social engineering; query patterning that defeats differential privacy; a biometric system leaking template information through false-accept patterns; a content-moderation system leaking its policy through which queries get blocked; a routing system leaking map data through its path choices. All are side-channel extraction — and crucially, none of them would be called "extraction through prompting." That is exactly the tell that the LLM case is a new substrate for an old pattern, not a structure that itself ports: what travels is side-channel reasoning, and the cross-domain lesson should carry side_channel_attack (with access_control/trust_boundary/query_interface), not the prompt-specific construct. The home-bound cargo is everything LLM-particular: the crafted-prompt techniques (divergence/repetition like the poem-poem-poem attack, role-play and summarisation system-prompt leakage, recursive "ignore previous instructions," prompt injection inside retrieved context), the structural fact realized in a generation model that there is no internal boundary between what it is permitted to know and to say, and the specific defense surfaces tuned to that substrate. Naming a hardware timing leak or a moderation-policy leak a "prompting" attack would be analogy; the honest move is to treat LLM extraction as the worked AI-substrate instance of side_channel_attack and carry that prime across domains. See Structural Core vs. Domain Accent.

Examples¶

Canonical¶

The defining demonstration is the divergence attack reported by Nicholas Carlini, Milad Nasr, and colleagues in their 2023 work on scalable training-data extraction from production language models. They found that instructing a deployed chatbot to repeat a single word endlessly — the "poem poem poem…" prompt — eventually caused the model to diverge from the repetition and instead emit long stretches of verbatim memorized training text, including personally identifiable information such as real email addresses and phone numbers. No weights were stolen and no database was breached; the private content came straight out through the ordinary chat interface, because frequently repeated training samples are the most easily reconstructed from the model's activations.

Mapped back: The chatbot is the deployed model holding memorized training text; the repeat-forever string is the crafted prompt sent through the query interface. The recovered PII is the emission, carried out the sanctioned output channel — a direct display of the no-internal-boundary fact: nothing enforced a line between what the model had internalized and what it was willing to say.

Applied / In Practice¶

System-prompt leakage against a live product is the textbook applied case. When Microsoft launched its Bing chat assistant in February 2023, a Stanford student, Kevin Liu, entered a prompt asking it to ignore prior instructions and print the text at the start of its document; the assistant obligingly revealed its confidential system prompt, including its internal codename "Sydney" and the operator's hidden behavioral rules. The instructions the operator had placed inside the model's context — meant to steer it invisibly — were treated by the model as ordinary text it could reason about and repeat back.

Mapped back: Bing's hidden configuration is inside the access-control envelope the operator drew, but Liu's "ignore previous instructions" crafted prompt reached past it through the query interface. The leaked "Sydney" rules are the emission, and the incident illustrates the structural irreducibility: the same context-conditioning that lets the assistant follow its brief is what let it disclose the brief.

Structural Tensions¶

T1: Helpful conditioning versus exfiltration surface (the capacity that serves is the capacity that leaks). The whole failure mode rests on a single dual-use fact: the model's ability to condition responses on its full context — training data, system prompt, retrieved documents — is exactly what makes it helpful and exactly what creates the leak. There is no separate "good" generation channel and "bad" generation channel to pry apart; the same mechanism that follows the operator's brief is what discloses the brief. This is why the gap is structural rather than a patchable bug: "switch off the conditioning for adversarial inputs" would degrade legitimate function, because the adversarial and legitimate uses run through the identical capability. The tension is that hardening against extraction and preserving usefulness pull on the same lever in opposite directions, so every real defense is a partial trade of capability for containment. Diagnostic: Can the control being proposed here distinguish adversarial from legitimate conditioning, or does it necessarily blunt the helpful generation it is trying to protect?

T2: Storage locus versus inference locus (a correct security instinct aimed at the wrong surface). The engineer's reflex is to treat private data as a storage problem — encrypt the weights, lock the vector store, gate the credentials — and that instinct is not wrong in general; those controls genuinely protect against breach. But extraction through prompting occurs with every storage system intact, at the inference interface, through sanctioned generation. The tension is that a defender can do storage security impeccably and remain fully exposed, because the leak never touches storage — yet abandoning storage hardening would open the breach threats it does defend. The concept's contribution is to relocate this threat to the interface without implying the storage controls were pointless for their own threats. Aim everything at storage and the exfiltration surface stays open; treat storage as irrelevant and you drop defenses against actual breaches. Diagnostic: Did this disclosure require breaking into a storage system, or did it come out through ordinary generation — and is the proposed control placed where the leak actually happened?

T3: Narrowable versus uncloseable (investing in filters that are structurally incomplete). Because the space of crafted inputs is open-ended, the honest answer to "what can the model be made to say about everything it has access to?" is anything, modulo cost — any enumerated blocklist of known techniques (divergence prompts, role-play, "ignore previous instructions") will be evaded by novel inputs. Yet output filtering, retrieval-source gating, and training-data sanitisation genuinely narrow the surface and raise attacker cost, so they are worth building. The tension is that the defender must invest seriously in controls while knowing they cannot close the gap, and must resist both the illusion that the catalogue can be exhausted and the fatalism that says partial defense is worthless. A defense premised on completing the blocklist is structurally doomed; a posture that abandons filtering because it is incomplete forfeits the cost-raising that is the only available win. Diagnostic: Is this defense being sold as closing the exfiltration surface, or as narrowing it and raising cost — and is the residual, uncloseable surface being budgeted for?

T4: Extraction versus its sibling failure modes (a boundary that dictates defense placement yet blurs at the vehicle). Holding extraction distinct from jailbreaking, data poisoning, and membership inference is load-bearing precisely because the category dictates where the defense lives — extraction defenses belong at output filtering, retrieval gating, and training-data sanitisation, not the refusal-and-policy layer where jailbreak defenses sit. An operator who hardens only refusal leaves the exfiltration surface open. But the boundary is not clean in practice: a jailbreak can be the vehicle that carries out a disclosure, so the same incident can be both a policy bypass and an extraction. The tension is that over-separating the failure modes misses their real interaction (the jailbreak-delivered leak), while over-merging them routes extraction defenses to the wrong layer entirely. Diagnostic: Is the disclosure here a matter of information leaving (extraction, defend at the output/context layers) or of policy being bypassed (jailbreak, defend at refusal) — and is a bypass merely the delivery mechanism for a leak?

T5: Structural absence versus model disposition (where to locate the cause). It is tempting to read the model as choosing to leak — an alignment lapse, a misbehaving intent the refusal training should have caught. The concept insists the opposite: the disclosure follows from the absence of an enforceable internal boundary between what the model may know and what it may say, not from a disposition it elected. This attribution is consequential because it redirects the defense: read as disposition, the fix is more refusal training and policy classification (the jailbreak layer); read as structure, the fix is at the output and context surfaces. The tension is that the behaviour genuinely looks like the model deciding to reveal secrets, and refusal training does reduce some leaks, so the dispositional framing is seductive and partially effective — yet treating it as the primary cause mislocates the problem in the model's character rather than its architecture. Diagnostic: Is this leak best explained by the model lacking an internal know/say boundary (structural — defend at the interface), or by a refusal it should have produced and did not (dispositional — defend at policy)?

T6: Autonomy versus reduction (an LLM-specific failure mode or the AI-substrate instance of side-channel attack). Data extraction through prompting is a richly-specified, canonically-demonstrated LLM security concept, with its own attack catalogue (the poem-poem-poem divergence attack, system-prompt leakage, prompt injection) and its own defense surfaces. Yet stripped of model vocabulary it is side_channel_attack: a legitimate output channel of a system reveals information the access-control model meant to protect (with access_control, trust_boundary, and query_interface). That pattern recurs across substrates with no prompting in them — timing and cache attacks in hardware, traffic analysis in networking, conversation patterning in social engineering, policy leakage in content moderation — and none of them would be called "extraction through prompting," which is exactly the tell that the LLM case is a new substrate for an old pattern, not a structure that itself ports. The tension is between a construct specific enough to name the prompt-craft techniques and the recognition that its transferable core already belongs to side-channel reasoning. Diagnostic: Resolve toward side_channel_attack (with access_control/trust_boundary/query_interface) when carrying the lesson to hardware, networking, or moderation substrates; toward data extraction through prompting itself when the attack surface is a generation model conditioned on protected context.

Structural–Framed Character¶

Data extraction through prompting sits at the mixed point of the spectrum — closely paralleling cross-site scripting: a genuine, observer-free emission mechanism that lives inside a human-engineered artifact and is defined relative to a human-set privacy policy. The five criteria split.

On evaluative weight it carries a mild framing: "failure mode" and "attack" mark the emission as a defect, but only relative to what the operator intended to remain private — a designed security objective, not a fact of nature. This is lighter than a moral verdict and grounded in a concrete mechanism, but it is not the pure neutrality of feedback. On human_practice_bound it patterns partly structural and partly framed: the emission mechanism runs observer-free — the model generates conditioned output whether or not anyone is judging — so it is not practice-bound the way a code smell's verdict is; yet the substrate it runs in (a deployed LLM, its inference interface, the access-control envelope) is a built environment, and the "protected" status is set by an operator. Its institutional_origin is accordingly pronounced: the model, the query interface, and the very notion of an access-control envelope are artifacts of AI-systems engineering, and the concept is named within an AI-security discourse (OWASP LLM Top-10, red-teaming) — the trust model against which "disclosure" is defined is a human design. On vocab_travels it scores low: deployed model, query interface, system prompt, RAG, crafted prompt, and prompt injection are pinned to the LLM substrate and rename off it. And import_vs_recognize is bimodal in the entry's own sharp terms — within LLM security the failure is recognized intact, while its cross-substrate cousins (timing, cache, traffic-analysis, moderation-policy leaks) are carried by the parent as co-instances, and — the entry's own tell — none of them would be called "extraction through prompting," so the LLM case is a new substrate for an old pattern, not a structure that itself ports.

The one portable structural skeleton is side-channel extraction — a legitimate output channel of a system reveals information the access-control model intended to protect, because the interface's expressive breadth exceeds the envelope drawn around the data. This skeleton decomposes into side_channel_attack with access_control, trust_boundary, and query_interface. It is genuinely substrate-spanning and recurs across hardware, networking, and moderation systems, which tempts a structural reading. But it does not pull data extraction off the mixed point, because that side-channel structure is exactly what the entry instantiates from its parent side_channel_attack, not what makes "data extraction through prompting" itself travel: the cross-domain reach belongs to the parent, while the crafted-prompt techniques (divergence/repetition, role-play system-prompt leakage, prompt injection), the in-a-generation-model no-internal-boundary fact, and the LLM-tuned defense surfaces — the distinctive layer — stay home. Its character: a real, observer-free information-emission mechanism carrying a defect verdict relative to a designed privacy policy and running inside a wholly engineered LLM substrate, structural only in the side-channel skeleton it borrows from its parent and mixed in every distinctively prompt-and-model accent.

Structural Core vs. Domain Accent¶

This section decides why data extraction through prompting is a domain-specific abstraction and not a prime, and it carries the case for its domain-specificity — there is no separate section for that.

What is skeletal (could lift toward a cross-domain prime). Strip the LLM substrate away and a thin relational structure survives: a legitimate output channel of a system reveals information the access-control model intended to protect, because the interface's expressive breadth exceeds the envelope drawn around the data — and the breadth cannot be selectively disabled for hostile use without disabling legitimate use. The portable pieces are abstract — an access-control envelope narrower than what the query interface can be induced to emit, a trust-boundary violation between intended and hostile use, disclosure through the sanctioned channel rather than a storage breach, and a residual surface that can be narrowed but not closed. That side-channel-extraction skeleton is genuinely substrate-portable: it recurs, as the entry stresses, as co-instances with no prompting in them — timing, power-analysis, cache-timing, and electromagnetic-emanation attacks in hardware; traffic analysis in networking; conversation patterning in social engineering; query patterning that defeats differential privacy; a content-moderation system leaking its policy through which queries get blocked. That recurrence is mechanism, which is why the entry decomposes the failure into its parent — but it is the core data extraction shares, not what makes it data extraction through prompting.

What is domain-bound. Almost everything that makes the concept extraction through prompting in particular is LLM-security furniture and none of it survives extraction intact: the crafted-prompt techniques (divergence/repetition like the poem-poem-poem attack, role-play and summarisation system-prompt leakage, recursive "ignore previous instructions," prompt injection inside retrieved context); the structural fact realized in a generation model that there is no internal boundary between what it is permitted to know and to say; the substrate objects (deployed model, inference interface, system prompt, RAG vector store); the LLM-tuned defense surfaces (output filtering, retrieval-source gating, training-data sanitisation); and the empirical cases — the poem-poem-poem PII recovery, the Bing "Sydney" system-prompt leak. These are the worked vocabulary, techniques, and cases the discipline actually studies. The decisive test is the entry's own tell: the cross-substrate cousins — a hardware timing leak, a moderation-policy leak — would never be called "extraction through prompting," which is exactly the sign that the LLM case is a new substrate for an old pattern; remove the generation model and its crafted-prompt interface and what remains is bare side-channel extraction, a looser and more general thing.

Why this does not clear the prime bar. A prime is a relational structure whose vocabulary travels and whose cross-domain transfer is recognition of the same mechanism, not analogy. Data extraction's transfer is bimodal. Within AI security the mechanism travels intact — the interface-versus-envelope gap, the storage-reflex-refusing diagnostic, the emission-character-to-technique classification, the defense-locus prescription, and the negative prediction (the conditioning capacity cannot be selectively disabled) all keep their meaning across LLM red-teaming, deployed-product security, agent and tool-use security, model-IP protection, and regulated-deployment compliance, because each is the same LLM-substrate information-disclosure threat with different data and deployment. Beyond the LLM substrate, naming a hardware timing leak or a moderation-policy leak a "prompting" attack is analogy — the named construct does not port. And when the bare structural lesson is needed cross-domain, it is already carried, in more general form, by the parent data extraction instantiates: a legitimate output channel leaking access-controlled information is side_channel_attack, decomposing into access_control (the envelope drawn around the data), trust_boundary (violated between intended and hostile use), and query_interface (the attack surface). The cross-domain reach belongs to that parent; "data extraction through prompting," as named, is the worked AI-substrate instance of it, carrying prompt-and-model baggage that does not and should not travel.

Relationships to Other Abstractions¶

Current abstraction Data Extraction Through Prompting Domain-specific

Parents (2) — more general patterns this builds on

Data Extraction Through Prompting is a kind of Input Manipulation Attack Domain-specific

Data extraction through prompting is input manipulation specialized to a deployed language model whose crafted live input elicits protected content through its ordinary generation channel.
Data Extraction Through Prompting presupposes Minimum-Necessary Disclosure Prime

Prompt-mediated extraction presupposes a legitimate disclosure budget that the model's response exceeds because protected surplus was not projected away before the observable output channel.

Hierarchy paths (4) — routes to 4 parentless roots

Data Extraction Through Prompting → Input Manipulation Attack → Adversarial Boundary Navigation

Show alternative paths (3)

Not to Be Confused With¶

Jailbreaking. The constantly-conflated sibling: bypassing a model's behaviour policy to make it produce disallowed content (violence, hate, disallowed instructions). Extraction is about information disclosure, not policy bypass — and the distinction dictates defense placement: jailbreak defenses live at the refusal-and-policy layer, extraction defenses at output filtering, retrieval-source gating, and training-data sanitisation. A jailbreak can be the vehicle for an extraction, but they are different failure modes. Tell: is the harm the model saying something against policy (jailbreak) or private content leaving through the interface (extraction)?
Prompt injection. A delivery mechanism — smuggling instructions into content the model treats as authoritative (a retrieved document, a tool output) so it follows the attacker rather than the operator. It is one vector that can drive extraction, but injection can also cause non-disclosure harms (making an agent take unauthorized actions, corrupt outputs). Tell: is the referent the technique of injecting authoritative-looking instructions (prompt injection, a means) or the outcome of protected content being emitted (extraction, an end)? Injection is how; extraction is what.
Data poisoning attack. Corrupting the training inputs to alter the model's future behaviour (planting a backdoor, degrading accuracy). Extraction runs the opposite direction in time — it recovers content already internalised, changing nothing about the model. Tell: does the attack inject bad data to shape what the model will do (poisoning) or pull existing data out of a deployed model (extraction)? One writes, the other reads.
Membership inference attack. Establishes only whether a specific record was in the training set — a yes/no presence signal, often used to argue privacy leakage. Extraction goes further: it reproduces the record itself, verbatim. Tell: does the attack conclude "this datum was present" (membership inference) or actually emit the datum's contents (extraction)? Presence versus reproduction.
Model inversion / model extraction (stealing). Related recovery attacks: model inversion statistically reconstructs representative training inputs from outputs/confidences; model extraction clones a proprietary model's behaviour or weights through queries. Data extraction through prompting instead elicits verbatim protected content (training text, system prompt, retrieved docs) via crafted natural-language prompts. Tell: is the target a statistical reconstruction or a cloned model (inversion/stealing) or exact protected strings emitted through the chat interface (extraction through prompting)?
The side-channel-attack parent (umbrella). The substrate-neutral pattern the entry instantiates — a legitimate output channel reveals information the access-control model meant to protect, because the interface's breadth exceeds the envelope around the data — carried by side_channel_attack with access_control, trust_boundary, and query_interface. Timing, cache, traffic-analysis, and moderation-policy leaks are co-instances, and none would be called "extraction through prompting." Tell: the parent travels to any substrate with a leaking legitimate channel; "data extraction through prompting," treated more fully in a later section, is the LLM-substrate instance with its crafted-prompt techniques.

Neighborhood in Abstraction Space¶

Data Extraction Through Prompting sits in a crowded region of the domain-specific corpus (10^th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.

Family — Unclustered & Miscellaneous (309 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-07-12