ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience

Abstract

Setting

Online Agentic Test-Time Training: learn from one attempt per task

At deployment time, an agent meets a one-pass stream of related tasks. OaTTT trains the LLM's weights on the agent's own execution trajectories, from one attempt per task. We ask:

Can a deployed agent turn its own verified execution trajectories, one per task, into persistent weight updates that improve its success on subsequent tasks?

Existing parametric adaptation also updates the LLM's weights. Some test-time training methods adapt them within a single episode and reset them afterward. Others need pre-deployment training of the adaptation behavior or several attempts per task, and RL post-training optimizes the policy before deployment, typically with many attempts per task or through single-attempt RL with a learned critic. None of them adapts a long-horizon agent's weights at test time, in a single pass over the task stream with one attempt per task, while carrying updates across tasks.

OaTTT via direct imitation or reinforcement of experience

Directly imitating or reinforcing the generated tokens of a single attempt

A straightforward way to perform OaTTT is to directly imitate or reinforce the agent's own interaction trajectories. This is fragile, because each task yields only one attempt by the current policy. We formulate five such baselines in two families. Every method uses LoRA fast weights on the frozen base, carried across episodes, and all five learn only from the tokens the agent generated.

Scroll sideways or tap to open the figure at full size.

In OaTTT, we aim to enable single-attempt, per-sample online learning (batch size 1) of LLM weights, analogous to how in-context adaptation agents evolve. In these baselines, each update relies on a single trajectory, so its sample-specific noise is not averaged out. Because later attempts come from the updated policy, imitating or reinforcing the generated tokens can amplify this noise into overfitting, policy drift, and eventual degradation. ASCENT addresses this by distilling experience with an on-policy self-distillation (OPSD) objective, using next-token distributions from an anchored self-teacher conditioned on hindsight from verified trajectories as targets.

Method

ASCENT self-distills verified experience for stable online test-time training

ASCENT stably distills verified experience from the hindsight-informed predictions of the frozen initial model into LoRA fast weights, evolving the agent during OaTTT. When an attempt is verified, the frozen model reads privileged information built from the filtered trajectory and gives full next-token distributions at the agent's own student prefixes. Failed attempts skip the update. The updated agent produces better trajectories, enabling online self-evolution along the stream.

Scroll sideways or tap to open the figure at full size.

What the teacher sees

Results

ASCENT consistently improves long-horizon task success

At both scales, ASCENT improves exact success over the base by more than 22 points on ALFWorld and WebShop and exceeds the strongest online comparator. It also has the highest WebShop mean score and outperforms every offline in-context adaptation method on ALFWorld. ASCENT outperforms TT-OPSD, our test-time adaptation of OPSD, which uses the same self-distillation with a teacher that sees only the current turn. This shows the benefit of hindsight from the complete verified trajectory.

These consistent gains show that cross-episode parameter updates can extract reusable behavior from the same sparse environment verification available to offline and online in-context adaptation methods.

Setup. Qwen3.5-4B and Qwen3.5-9B. The base, all compared methods, and ASCENT use the same ReAct template and greedy decoding. Online methods follow the OaTTT protocol on the same task order: one pass over the stream, a single scored attempt per task with a 50-turn budget, and each task scored before its trajectory can update the method. They receive the same environment verification signal. Offline baselines instead build memories, skills, or prompts from the same fixed-size training split before evaluation. They fall outside OaTTT and serve as reference points. The direct imitation and policy-gradient baselines are left out of these tables because they severely degrade under OaTTT (Figure 1). Main results are mean ± std over 3 independent runs.

ALFWorld seen

WebShop

Cross-episode gains extend to difficult tasks

The base rarely solves Heat or Cool tasks (SR 6.2 and 8.0 at 4B, 0.0 and 8.0 at 9B), whereas ASCENT improves both task families at each scale (20.8 and 54.7 at 4B, 37.5 and 48.0 at 9B). Its gain over the base is also much larger on later 4B tasks, while the base remains flat. ASCENT derives this improvement from its own verified experience during deployment, without a stronger teacher or a pre-deployment adaptation stage.

Tables 1 and 2 show that ASCENT reduces capped mean turns per episode at both scales on both benchmarks. On ALFWorld tasks that both the un-evolved base model and ASCENT solve, its trajectories are shorter too. Episode time, which includes the attempt and the LoRA update after verified successes, is modestly higher than the base on ALFWorld (4B 97 → 115 s, 9B 115 → 133 s) and lower on WebShop (4B 87 → 76 s, 9B 166 → 164 s).

Transfer

The evolved policy transfers and continues learning under scene shift

On ALFWorld unseen (134 tasks in held-out rooms and layouts), ASCENT leads the online comparators in both frozen transfer and continued adaptation at both scales. At 4B, continued OaTTT, which resumes online updates, substantially exceeds frozen transfer (+15.9 points). At 9B, continuation can preserve the already strong transfer result. These results show that seen-stream experience is retained in the evolved policy and that the self-evolution protocol remains effective.

Frozen transfer Continued adaptation ASCENT

Analysis

Experience internalized in the weights completes a long-horizon task

On the task “put a cool egg in microwave”, the base and five in-context adaptation variants exhaust the 50-turn budget, whereas ASCENT completes it in 20 turns. The in-context variants fail either by retrieving irrelevant or contradictory experience or by not following relevant guidance. ASCENT retrieves no memory. It carries LoRA updates from two earlier cool-then-place tasks, internalizing that same-family experience in its weights, and transfers it across objects and destinations. Across whole streams, retrieving available same-type prior experience does not reliably lead to task completion (paper Appendix C.3).

Scroll sideways or tap to open the figure at full size.

Interaction efficiency

In Figure 5a, ASCENT's turn savings over the base grow later in the stream. They also persist on shared successes, the tasks that both the un-evolved base model and ASCENT solve. ASCENT uses fewer turns than the base on 64% (4B) and 67% (9B) of them (paper Table 8), which supports a gain in interaction efficiency beyond the change in the success/failure mix.

In-context and parametric co-evolution Preliminary

ASCENT stores experience in fast weights, whereas in-context adaptation stores it in the context. We combine ASCENT with ACE and MemP on ALFWorld seen and find that the two are complementary: adding ASCENT improves both in-context methods at both scales. Whether retrieved memory also helps on top of ASCENT depends on the model. On the smaller 4B model, ASCENT alone performs best, so the extra retrieved context may distract a weaker model, in line with its preference for shorter privileged information (Table 3 below). On the relatively stronger 9B model, both combinations exceed ASCENT alone, maybe because the model can use the retrieved memory and its evolved weights together. Co-evolving context and weights is thus promising for stronger agents.

Ablations

Generally reasoning is the key component of the privileged information, and gains hold across divergences

Distillation divergence

ASCENT still improves on the base at both scales when reverse KL or Jensen–Shannon (JS) divergence replaces forward KL. Reverse KL trails forward KL at both scales, most at 9B. JS matches reverse KL at 4B but exceeds forward KL at 9B. Forward KL favors coverage of the teacher distribution, whereas reverse KL is mode-seeking.

Citation

BibTeX

@misc{lu2026ascent,
  title         = {{ASCENT}: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience},
  author        = {Lu, Haodong and Gong, Dong},
  year          = {2026},
  eprint        = {2610.05303},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2610.05303}
}