Skip to content

Selective Parameter Freezing

Isolation method — instantiates Context-Keyed Representation Switching

Write-protects the parameters that encode one context's map so that learning a different context cannot overwrite them, drawing the isolation boundary in parameter space.

When several context-maps live in one parameter set — a genuinely shared substrate — updating for context B can silently degrade context A's map, the failure known as catastrophic forgetting. Selective Parameter Freezing pins the parameters that carry A's map (and the shared-invariant core underneath all contexts) so that B's update is only allowed to move the parameters that are A-neutral. Its distinguishing idea is where the isolation boundary is drawn: not by giving each map its own address or its own snapshot, but by making the protected weights read-only during another context's update. The wall is in parameter space, and it is enforced at the moment of learning.

Example

A shared perception network must handle both daytime and night driving on the same weights. Fine-tuning on a fresh batch of night data reliably wrecks daytime performance — the night gradients drag the shared weights toward night and the daytime map is forgotten. Selective Parameter Freezing identifies the units that daytime performance depends on, and the low-level feature backbone common to both, and freezes them: their gradients are zeroed for the night update. Only a subset of higher-layer parameters is left free to adapt. The result is night performance that improves while daytime holds, because the daytime-critical and shared-invariant weights were write-protected exactly while the interfering update ran.

How it works

  • Identify which parameters encode the map(s) to protect, which form the shared-invariant core, and which are freely adaptable.
  • Freeze the protected and invariant parameters — zero their gradients or mark them read-only — for the duration of the other context's update.
  • Update only the unfrozen subset.
  • Refreeze along the new boundary as protection shifts to a different context.

Tuning parameters

  • Freeze fraction — how much of the network is frozen. More frozen means stronger protection and less capacity to learn the new context — the plasticity–stability trade-off in one dial.
  • Freeze granularity — whole layers versus individual units or weights. Finer preserves capacity but is far harder to target correctly.
  • Hard versus soft freeze — a strict read-only freeze, or a penalty that merely resists moving protected weights. Soft allows negotiated change; hard guarantees zero drift.
  • Boundary source — how "protected" is decided: measured parameter importance, task masks, or explicit layer assignment.

When it helps, and when it misleads

Its strength is that it lets several context-maps coexist on one substrate without keeping a full copy per context, directly countering catastrophic forgetting[1] and protecting the shared-invariant layer every context relies on. Its failure modes are all about the boundary: freeze too much and the new context cannot be learned at all (it underfits); freeze the wrong parameters and you protect the unimportant while the important ones drift; and a hard freeze cannot accommodate a context that legitimately needs to revise shared structure. The classic misuse is freezing broadly as a blunt "don't break anything" reflex, which starves the model of the plasticity it needs and hides that the substrate was over-shared to begin with. The discipline is to base the freeze boundary on measured parameter importance for each protected context, and to revisit it — a static freeze fossilizes yesterday's partition of the maps.

How it implements the components

Selective Parameter Freezing fills the on-substrate isolation subset — keeping shared-parameter maps from overwriting each other:

  • representation_isolation_boundary — freezing is the boundary, drawn in parameter space: frozen weights are the wall one context's update cannot cross into another's map.
  • shared_invariant_layer — it explicitly protects the invariant core valid across all contexts by freezing it while only the context-specific parameters adapt.

It does not keep maps apart by naming or addressing them separately (that's the Context-Tagged Namespace Partition), preserve a full snapshot for later restore (that's Per-Context Model Checkpoint), or gate the promotion of an updated map (that's Shadow-Map Evaluation).

Notes

Parameter freezing, Context-Tagged Namespace Partition (isolation by address), and Shared Backbone with Context Adapters (isolation by module) are three ways to stop maps colliding on a shared substrate. Freezing is the right tool specifically when the maps must share one parameter set and cannot be cleanly separated into namespaces or modules — it is the finest-grained and the most delicate of the three to target.

References

[1] Catastrophic forgetting and its correctives — including Elastic Weight Consolidation, a soft-freeze method that penalizes moving parameters deemed important to earlier tasks. Freezing is the hard-boundary form of the same idea. ↩