HOPE: Functional Capacity, Progressive Encoding, and a Better Stability–Plasticity Boundary

Fine-tuning is usually framed as an optimization problem: choose an objective, decide what to unfreeze, and update the model until the target metric improves.

That framing hides a harder systems question:

Which parts of a trained network are load-bearing knowledge, which parts are redundant copies, and which parts can safely become capacity for a new task?

Hilbert Operator for Progressive Encoding (HOPE), by Hossein Mobahi and Peter L. Bartlett, answers with functional geometry rather than parameter magnitude. It lifts each neuron into a Hilbert space, treats the neuron as a rank-1 Hilbert–Schmidt operator, and uses projection distortion to compare pruning, neuron merging, and residual-block eviction under one metric.

The paper then turns that capacity ordering into DEFT — Dispersed Elastic Fine-Tuning: consolidate redundant source features, freeze a universal core, route learning into peripheral slack, and sever the paths through which changing slack could corrupt the core.

The reported proof-of-concept is strong enough to study and narrow enough not to oversell. On the paper’s CIFAR-100→SVHN transfer protocol, DEFT reaches 89.79% target accuracy, 52.14% source retention, and an H-Score of 65.82, the best balance among five reported methods. But this is not yet evidence for transformer or LLM-scale deployment.

Why parameter magnitude is the wrong abstraction

Magnitude pruning assumes small weights or channels are less important. Deep networks make that assumption fragile because their parameterizations are not unique.

Within-neuron and cross-layer scale symmetries can rescale one component and compensate in another without materially changing the implemented function. Raw norms can therefore reflect optimization history more than functional importance.

HOPE changes the atomic unit. A neuron includes:

• its incoming weights;
• Batch Normalization statistics and affine parameters;
• its nonlinear activation;
• and its outgoing weight vector.

Let the input side define a scalar activation function and the output side define a direction in the next layer. Their tensor product is a rank-1 operator. In this representation, the neuron is no longer “a list of numbers.” It is a vector-valued function with measurable energy and geometric relationships to other neurons.

The Hilbert inner product captures functional correlation. The induced norm becomes functional capacity. Two neurons with different raw scales can still be recognized as functionally aligned; two similarly sized neurons can be recognized as behaviorally different.

For builders, this is the first important lesson:

Select compression and plasticity boundaries using the function implemented by a unit, not only the coordinates used to parameterize it.

The data-free path is analytical, but conditional

HOPE needs a distribution over neuron inputs to compute expected activation products. Repeatedly passing a dataset through the network would defeat the paper’s data-free goal.

For networks with Batch Normalization, the checkpoint already contains moving means and variances. HOPE applies the maximum-entropy principle to those constraints, yielding a Gaussian surrogate for local pre-activations. The resulting kernel integrals can be evaluated analytically.

This avoids real data, synthetic image generation, and repeated activation matching.

The condition matters. The framework is data-free in the presence of BN statistics. For architectures without BN, the authors propose a one-time calibration pass over a small batch to collect pre-activation statistics. That is a reasonable extension path, not an empirical result on modern transformers.

Do not translate “data-free on BN networks” into “drop-in data-free compression for every architecture.”

One geometry, three compression actions

HOPE places three reductions in the same action space:

1. Pruning: project one neuron to zero.
2. Merging: approximate a pair of correlated neurons with one optimal rank-1 parent.
3. Block eviction: drive an entire residual pathway to zero so the skip path becomes the identity mapping.

Each candidate action receives a projection-distortion cost. The controller then divides that cost by the initial parameter footprint released by the action.

The use of the initial footprint is subtle. If the controller used the live footprint, pruning one channel would shrink the apparent future savings of adjacent operations. The greedy search could repel itself from contiguous structural reductions and leave a fragmented model. Freezing the footprint surrogate restores the item-independence assumption used by the continuous knapsack relaxation.

The action rule is therefore conceptually simple:

Execute the currently admissible operation with the lowest distortion per initially measured parameter removed.

Only one action is executed before costs are reconsidered. This receding-horizon loop prevents a long hypothetical plan from compounding stale state estimates.

The scan is not free. With roughly N neurons per layer, pairwise merge candidates dominate at O(L·N²) across L layers. Cached candidate costs are O(1) to query, and a successful merge needs an O(N) localized cache update. That may be practical for the convolutional proof-of-concept; it is an obvious scaling question for wide modern networks.

DEFT turns capacity ordering into a plasticity policy

HOPE’s progressive encoding produces pruning costs that rank neurons by their functional importance. DEFT converts that ranking into a binary elasticity map:

• high-capacity neurons become the frozen Universal Core;
• lower-capacity neurons become Peripheral Slack and may update.

This alone would still waste capacity. A source feature can be fragmented across several correlated neurons. A static threshold would freeze all copies, treating redundancy as foundational knowledge.

DEFT first merges correlated core features into a single parent, releasing the other child slots as plastic capacity.

Figure 4 is reproduced from the paper and only cropped/converted for web display.

The second mechanism is a structural mask. If a plastic neuron in layer l feeds a frozen core neuron in layer l+1, that cross-connection is set to zero at initialization and prevented from updating. Core-to-core and slack-to-slack paths remain.

This gives DEFT two different protections:

• Static control: the initialization change is bounded by the capacity threshold and projection errors.
• Dynamic decoupling: drifting slack signals cannot enter the frozen downstream core through a severed edge.

That distinction is useful for agent and model infrastructure. “Frozen parameters” are not a sufficient safety boundary if changing upstream activations can still perturb their inputs. Protection must include the information-flow edges, not just update permissions on nodes.

What Table 2 actually shows

The transfer experiment pretrains source models on tasks built from four CIFAR-100 superclasses, yielding 20 fine-grained categories per source task. The models are then adapted to all ten SVHN digit classes. The final table averages 4 trials with 5 tasks each, or 20 source→target scenarios.

Table 2 is reproduced from the paper and only cropped/converted. Values and ± terms are preserved verbatim.

The trade-off is sharp:

• Full FT: 94.09 target, 7.52 source retention, 13.88 H-Score.
• Head-Only: 36.11 target, 63.13 source retention, 45.79 H-Score.
• EWC: 93.94 target, 6.74 source retention, 12.54 H-Score.
• PEFT / BN-Tuning: 81.91 target, 5.44 source retention, 10.18 H-Score.
• DEFT: 89.79 target, 52.14 source retention, 65.82 H-Score.

DEFT is not the best target learner and not the best source preserver. It is the strongest reported compromise under the harmonic-mean objective.

There is a small numerical typo in the prose above Table 2: one sentence assigns 93.94% to Full FT, while the table and the following sentence assign Full FT 94.09%; 93.94% belongs to EWC. This article uses the table values.

The theoretical guarantee is not perfect retention

The paper proves bounds around the structural intervention and argues that the frozen core receives zero dynamic interference from severed slack-to-core paths.

That can sound stronger than the empirical result. Source retention for DEFT is 52.14%, not 100%.

There is no contradiction. Dynamic gradient isolation does not erase:

• the static shock introduced when cross-connections are removed;
• projection error from merging;
• changes in the usable end-to-end computation around the frozen core;
• or mismatch between the source task metric and the local capacity surrogate.

For system designers, the takeaway is precise:

A no-update guarantee protects one channel of interference. It does not prove end-to-end behavioral invariance.

Always pair local structural guarantees with task-level regression evaluation.

What I would borrow for agent systems

HOPE is a neural-network paper, not an agent-harness paper. Still, its control ideas transfer cleanly.

1. Measure function, not artifact size

The size of a memory file, tool trace, prompt section, or skill document does not reveal its operational importance. Evaluate what changes when it is removed, merged, or isolated.

2. Merge redundancy before allocating new capacity

Agent memory often accumulates several paraphrases of the same rule. Freezing all of them wastes context and can create inconsistent policy copies. Consolidate correlated knowledge into one canonical parent before granting more writable memory.

3. Protect edges, not only nodes

Marking one component “read-only” is insufficient if untrusted or rapidly changing components can still inject state into it. Structural masks have an agent analogue: explicit information-flow policies that prevent experimental memory, speculative findings, or low-trust tool output from mutating the stable core.

4. Optimize distortion per capacity released

Deletion choices should consider both damage and resource gain. Removing a large context block that slightly affects quality may be better than removing a tiny block with the same distortion. Conversely, a visually large artifact may release little real runtime capacity.

5. Re-evaluate after every material change

HOPE’s receding-horizon strategy is safer than committing to a long compression plan computed from the initial state. Agent systems should do the same: change one governed component, rerun checks, update beliefs, then choose the next modification.

Limits before production adoption

The paper explicitly positions the experiments as proof-of-concept, and the boundary is important:

• Model compression is illustrated on Keras ResNet-50 pretrained on ImageNet.
• Transfer is evaluated on CIFAR-100→SVHN, not a long sequence of evolving domains.
• Convolutional adaptation assumes spatial stationarity from globally averaged BN variance.
• Networks without BN need calibration data.
• Pairwise merges create an O(L·N²) scan.
• DEFT introduces a percentile choice for the core/slack split.
• There is no transformer, LLM, multimodal, or production-latency study.
• The paper does not establish broad statistical superiority across architectures and datasets.

The compression plot reports better accuracy–density behavior than three magnitude-based structured baselines, but the paper does not provide an exhaustive large-scale benchmark. The safest conclusion is that the geometry is coherent and the early evidence is promising.

Builder takeaway

HOPE’s most valuable contribution is not a claim that Hilbert space has solved catastrophic forgetting.

It is a cleaner decomposition of the problem:

• represent units by function;
• measure capacity geometrically;
• merge duplicated knowledge;
• expose plastic slack;
• isolate stable core pathways;
• choose each reduction by damage per capacity released;
• and re-evaluate after every step.

That is a useful control architecture even before the method reaches LLM scale.

The deeper idea is that learned knowledge may not correspond to individual parameters at all. It may live in a functional subspace distributed across many redundant coordinates. If so, safe adaptation begins by identifying the subspace—not by guessing which weights look small.

Sources

• Hossein Mobahi and Peter L. Bartlett, Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks, arXiv:2607.21366v1, submitted 23 July 2026.
• Figure 4 and Table 2 are taken directly from the paper and only cropped/converted for display.

Image lane: OpenAI native primary.