Take Only What You Need:
Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning

Rank is the capacity lever for the plasticity–stability trade-off

Before proposing a method, we run a controlled probe: train CLIP with LoRA at different placements and ranks, and measure both new-task adaptation (plasticity) and retention of pre-trained capability (stability). Rank and placement turn out to be the knobs that trade one against the other. Each takeaway below is stacked with its evidence.

Takeaway 1 · Placement

LoRA placement is a plasticity–stability lever: no single placement or rank simultaneously optimizes both across tasks.

Takeaway 2 · Rank

Rank governs the balance: higher rank favors plasticity, lower rank favors stability. For a given level of adaptation, forgetting is minimized at a moderate-rank sweet spot.

Takeaway 3 · No universal rank

The sweet-spot rank is not universal: its location varies systematically by module and by downstream task — so the right method must allocate rank adaptively.

Method

CoDyRA is a capacity-controlled method: it makes the per-location rank a learnable, task-adaptive quantity and shrinks it with a sparsity penalty — distinct from model complementation (freeze the backbone, add external modules) and reference-constrained methods (constrain each update with stored data or prior-task projections).

Rank-weighted LoRA update

For module m at task t, the update is a sum of rank-1 components, each scaled by a learnable importance weight w_i; the number of non-zero weights upper-bounds the effective rank:

Rank minimization as the regularizer

The objective adds an \(\ell_1\) penalty on the importance weights — the supervised term drives plasticity, the sparsity term minimizes rank and regularizes forgetting:

Because \(\ell_1\) is non-smooth, weights take a proximal (soft-thresholding) step after each gradient step, with a threshold \(\kappa_t\) ramping from zero to \(\kappa_{\max}=0.005\):

Why lower rank means less forgetting

A first-order bound on forgetting \(F(\theta_t)\) grows with the update size, itself bounded by its rank \(\rho\) (Prop. 1):

Both terms shrink as \(\rho\) decreases, and \(\Delta\mathbf{W}\) occupies a subspace of dimension at most \(\rho\). So rank minimization behaves as an implicit forgetting regularizer — tightening the bound along both magnitude and subspace dimension, protecting pre-trained and prior-task knowledge alike.

The bound in practice: bounded parameter shift

This is not just theory. By minimizing rank, CoDyRA directly minimizes the update magnitude \(\lVert\Delta\mathbf{W}\rVert_F\): per task it produces the smallest update — below even rank-1 LoRA — despite a moderate effective rank, and over the stream the cumulative drift \(\lVert\theta_t-\theta_0\rVert_F\) stays low while fixed-rank LoRA blows up near-linearly. A smaller parameter shift is exactly the mechanism the bound predicts for less forgetting — direct evidence that rank minimization acts as the regularizer.

Experiments

5-shot MTIL — CLIP continual learning

Per-domain Transfer / Average / Last accuracy (%), grouped by method family. CoDyRA (capacity-controlled) is the only method that surpasses the zero-shot Transfer ceiling (70.1) while also leading on Last (78.0).

X-TAIL — cross-domain, task-agnostic CLIP CL

Transfer / Average / Last accuracy (%) on the 10-domain X-TAIL benchmark (per-domain tables in the paper).

TRACE — LLM continual learning

Overall Performance (OP, ↑) and Forgetting (Fgt, ↓) over 8 sequential tasks (mean of 3 runs; ±std in the paper). Across all three backbones CoDyRA gives the highest OP and the lowest forgetting.

Efficiency & forgetting

CoDyRA trains the fewest parameters and, because the pruned update merges into the backbone, adds no inference-time parameters or memory — while attaining the lowest Forgetting on X-TAIL.

BibTeX

@inproceedings{lu2026take,
  title     = {{Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learning}},
  author    = {{Lu, Haodong and Zhao, Chongyang and Xue, Minhui and Yao, Lina and Moore, Kristen and Gong, Dong}},
  booktitle = {{Findings of the Association for Computational Linguistics: EMNLP 2026}},
  year      = {{2026}},
  url       = {{https://openreview.net/forum?id=TmGgXWnedq}}
}