Attention (machine learning)

A neural-network mechanism that computes context-dependent weights over representations and combines them so each output can focus selectively on relevant inputs.

Version

v2 · 2026-09-19 · History

Domain-specific #

3362

Domain group

Applied Sciences & Engineering

Origin domain

Computer Science & Software Engineering

Subdomains

Machine Learning, Neural Attention → Computer Science & Software Engineering

Core Idea

Attention maps a query and key–value set to a weighted combination of values based on query–key compatibility. Scores compare each query with keys, normalization converts scores to weights, and weighted value sums transmit relevant information; multiple heads learn different relations. The abstraction is therefore identified by a declared carrier, a transformation or constraint over that carrier, and an invariant that tells an analyst whether the named structure is genuinely present.

The load-bearing residual is not the broad topic of machine learning. It is content-addressed differentiable routing among representations. That residual remains recognizable when examples, notation, scale, or implementation change, but it disappears if the carrier is mistyped, the condition that masking, normalization and value aggregation align with the declared attention form and tensor axes fails, a neighboring object is substituted, or notation and topical resemblance replace the constitutive test.

How would you explain it like I'm…

Take More From Best Matches

Imagine you have a question and a row of boxes, each with a label on the front and something inside. You check how well each label matches your question. Then you take a little bit from every box, but much more from the boxes whose labels match best, and mix it together into your answer.

Question, Label, Blend

In machine learning, attention is a way for a computer program to decide how much each piece of information should matter for another piece. Each piece makes a question, called a query, and every piece also has a label, called a key, and some information, called a value. The program checks how well the query matches each key and turns those match scores into weights that add up to one. Then it blends the values together using those weights, so better matches contribute more. Programs often do this several ways at once, called multiple heads, so they can notice different kinds of connections.

Content-Based Weighted Mixing

Attention is a mechanism in machine learning that computes, for each query, a weighted combination of values, where the weights depend on how compatible the query is with a set of keys. Each query is compared with every key to produce scores, the scores are normalized, commonly with softmax, into weights, and the values are summed with those weights. The result is that information flows according to content: what gets passed along depends on matches, not on fixed positions. Masking can block some query–key pairs, for example so a model cannot look at future words. Multi-head attention runs several of these computations in parallel, each learning a different kind of relation. Because every step is differentiable, the whole mechanism can be trained with gradient descent, unlike a hard lookup that picks a single item.

 

Attention maps a query and a set of key–value pairs to a weighted combination of the values, with weights derived from query–key compatibility. Concretely, compatibility scores are computed between each query and all keys (for example, scaled dot products), a normalization such as softmax converts them into weights, and a weighted sum of values yields the output. This implements content-addressed, differentiable routing among representations: information moves according to learned similarity rather than fixed wiring, and gradients flow through the soft weights. Multi-head attention applies several learned projections in parallel so that different heads can capture different relations, and the head outputs are combined. Masks restrict which query–key pairs may contribute, as in causal or padding masks. Correctly identifying an attention operation requires that masking, normalization and value aggregation be consistent with the declared form, such as self- versus cross-attention, and with the tensor axes over which weights are normalized and values summed.

Scope of Application

Attention (machine learning) belongs to machine learning and is useful where the analyst can specify queries, keys and values, similarity scores, scaling and mask, softmax or alternative normalization, weighted aggregation, heads, sequence positions and learned parameters, then evaluate masking, normalization and value aggregation align with the declared attention form and tensor axes. The scope is broad within that domain but bounded by the need for masking, normalization and value aggregation align with the declared attention form and tensor axes. The entry records a descriptive analytical identity; practical use requires the governing domain's evidence, standards, and safety obligations.

Clarity

The abstraction clarifies a crowded vocabulary by making masking, normalization and value aggregation align with the declared attention form and tensor axes the center of the account. A claim should name the carrier, the governing operation or relation, the applicable assumptions, and the recognition test. A bare label is insufficient because the name Attention (machine learning) can be used for a formal identity, an implementation, or a neighboring result unless carrier and convention are stated.

Manages Complexity

Without the abstraction, an analyst must reason directly over many local details: the carrier roles, admissibility assumptions, competing conventions, derived invariants, boundary cases, and proof or validation obligations specific to Attention (machine learning). Attention (machine learning) compresses them into the roles in the structural signature. That compression permits comparison across instances without erasing the variables that determine validity. It also exposes which details may be varied safely and which are constitutive.

Abstract Reasoning

1. Identify the carrier. State what the elements, states, objects, or observations are: queries, keys and values, similarity scores, scaling and mask, softmax or alternative normalization, weighted aggregation, heads, sequence positions and learned parameters. Reject examples whose alleged carrier belongs to a different problem.
2. Lock the constitutive rule. Express masking, normalization and value aggregation align with the declared attention form and tensor axes independently of one notation or implementation.

Knowledge Transfer

Knowledge transfers strongly among subfields of machine learning because they reuse queries, keys and values, similarity scores, scaling and mask, softmax or alternative normalization, weighted aggregation, heads, sequence positions and learned parameters, Scores compare each query with keys, normalization converts scores to weights, and weighted value sums transmit relevant information; multiple heads learn different relations., and type the carrier, state every parameter and convention in the definition, test that masking, normalization and value aggregation align with the declared attention form and tensor axes, compare the nearest accepted identity, and report counterexamples, uncertainty, and limiting cases.

Relationships to Other Abstractions

Current abstraction Attention (machine learning) Domain-specific

Parents (1) — outgoing typed relations

• Attention (machine learning) is a kind of Selection Prime

  Attention selects and weights relevant representations for each query.

Hierarchy path (1) — routes to 1 parentless root

• Attention (machine learning) → is a kind of Selection

Neighborhood in Abstraction Space

Attention (machine learning) sits in a crowded region of the domain-specific corpus (20^th percentile for distinctiveness): several abstractions share nearly its structure, so a description that fits it tends to fit its neighbors too.

Family — Machine Learning Models & Representations (25 abstractions)

Nearest neighbors

• Ensemble learning — 0.91
• Hidden layer — 0.91
• Lazy learning — 0.90
• Neural Turing machine — 0.90
• Deep learning — 0.90

Computed from structural-signature embeddings · 2026-10-25