Deep Learning Architectures & Scaling

← Back to Domain-Specific Families

Abstractions about neural architectures, representation learning, ensembles, generative models, transfer, compression, topology search, and model scaling.

16 abstractions in this family — domain-specific abstractions that sit near one another in structural-signature space (k-means over structural-signature embeddings). Each is shown with its short description.

• Attention (machine learning) — A neural-network mechanism that computes context-dependent weights over representations and combines them so each output can focus selectively on relevant inputs.
• Deep belief network — A multilayer generative model whose top layers form an undirected associative model and lower layers form directed latent-variable connections.
• Deep learning — Learn task-relevant hierarchical representations with multilayer parameterized neural networks trained end to end by optimization over data, enabling complex prediction and generation at the cost of opacity, data dependence, and compute.
• Ensemble learning — A machine-learning strategy combining predictions from multiple models so their complementary errors yield a stronger aggregate predictor.
• Evolutionary acquisition of neural topologies — An evolutionary reinforcement-learning method that jointly evolves artificial-neural-network structure and weights, using structural mutation and evolution-strategy parameter optimization.
• Evolutionary data mining — A family of data-mining methods that use evolutionary search to evolve rules, feature sets, model structures, parameters, or pipelines under a data-dependent fitness function.
• Generative adversarial network — A generative-learning framework in which a generator and discriminator are trained in opposition so generated samples approach a data distribution.
• Hidden layer — A neural-network layer situated between inputs and outputs whose learned nonlinear transformations construct intermediate representations used by later layers.
• MobileNet — A family of convolutional neural-network architectures designed for efficient vision inference through depthwise-separable convolutions and later mobile-oriented bottleneck, activation, attention, and scaling designs.
• Mode collapse — A generative-model failure in which the learned output distribution represents too few modes and produces markedly reduced diversity.
• Model compression — The reduction of a trained machine-learning model’s storage, memory or computation while preserving an explicitly tolerated level of task performance.
• Neural scaling law — An empirical relation describing how neural-network loss or capability changes with model size, data, training compute or inference compute.
• Neural Turing machine — A differentiable recurrent architecture coupling a neural controller to an addressable external memory.
• Optimistic knowledge gradient — A sequential sampling policy for costly crowdsourced labeling that scores an item by an optimistic estimate of how one more label could improve the final classification decision.
• Text-to-image model — A generative machine-learning model that conditions image synthesis on a natural-language description.
• Transfer learning — A machine-learning strategy that reuses representations, parameters or examples learned in a source task or domain to improve learning in a related target task.