ACPACP AI Research
⌕ZH
Deep ResearchHeat 88Quality 93

TEMPO: Scaling Test-Time Training via Critic Recalibration for Large Reasoning Models

TEMPO introduces a semi-supervised reinforcement learning framework designed to scale test-time training for large reasoning models. By executing an alternating Expectation-Maximization loop, TEMPO resolves reward drift and diversity collapse inherent in self-rewarding baselines.

AITest-Time TrainingReinforcement LearningLLMReasoning

Core Takeaway

TEMPO is a semi-supervised reinforcement learning framework designed to achieve scalable test-time training (TTT) for large reasoning models (LRMs) [1]. By replacing ungrounded self-reward loops with an alternating Expectation-Maximization (EM) optimization cycle, TEMPO effectively overcomes reward drift and diversity collapse bottlenecks [1][2].

Concept Background

Standard reasoning models during inference often rely on static Chain-of-Thought prompting or self-rewarding TTT approaches such as TTRL. However, conventional self-rewarding frameworks frequently plateau because reward models drift without ground-truth grounding, while actor policies collapse into repetitive output patterns [1]. TEMPO introduces a semi-supervised formulation that continuously stabilizes the model's self-generated reward signals [1][2].

Technical Principles

TEMPO formulates test-time training as an alternating EM loop that anchors the model's reward evaluator:

  • E-step (Critic Recalibration): Periodically updates a value critic model using verifiable rewards derived from a small labeled anchor dataset [1].
  • M-step (Policy Refinement): Updates actor policy parameters on unlabeled test instances using advantage estimates computed by the recalibrated critic [1][2].

Key Evolution

Benchmark evaluations on AIME 2024 demonstrate significant performance enhancements:

  • OLMo3-7B: Pass@1 accuracy increased from 33.0% to 51.1% (+18.1 percentage points) [1].
  • Qwen3-14B: Pass@1 accuracy improved from 42.3% to 65.8% (+23.5 percentage points) [1][2].

Unlike self-rewarding baselines that saturate early, TEMPO sustains performance improvements and preserves pass@k solution diversity across more than 350 training steps [1].

Practical Value

TEMPO demonstrates robust generalization beyond mathematics to broader non-math reasoning domains, including STEM challenges and logical puzzles [1][2]. It establishes a promising paradigm for adaptive inference-time learning across complex reasoning tasks.

Risks and Limits

  • Anchor Data Requirement: Depends on access to a small, high-quality labeled dataset for periodic critic recalibration [1].
  • Computational Overhead: Dynamically updating weights during inference incurs substantial computational delay and overhead compared to static CoT prompting [1][2].
  • Hyperparameter Sensitivity: The alternating EM loop can be sensitive to hyperparameter choices when deployed on highly out-of-distribution tasks [1].

Sources

  1. TEMPO: Scaling Test-time Training for Large Reasoning Models - arXiv (2026-04-21)
  2. TEMPO: Scaling Test-time Training for Large Reasoning Models - Hugging Face (2026-04-22)