Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Vladislav Beliaev Independent Researcher belyaev.vladislav.nw@gmail.com thinkdense.ai

摘要 / AbstractReinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.

目录 / Outline

1 Introduction p.1
Contributions. p.2
2 Related Work p.3
3 Preliminaries: GRPO and the Unscored Trace p.3
4 Method: Agon p.4
4.1 Why a second model p.4
4.2 Instantiating the two policies p.5
4.3 The draft-and-challenge step p.6
4.4 Reward: correctness plus a conversion bonus p.6
4.5 Compute parity p.8
5 Experiments p.9
5.1 Main results p.10
5.2 Scaling and generality p.10
5.3 A second domain: competitive-programming code p.11
5.4 Ablations p.11
5.5 Analysis: why traces shorten p.12
5.6 An optional density lever (auxiliary study) p.13
6 Conclusion p.13

引言

当前基于可验证奖励的强化学习（如 GRPO）仅根据最终答案给予奖励，不评估推理过程。在困难问题上，这促使模型生成更长但推理质量未提升的轨迹（过度思考），因为增加文本是提高期望奖励的廉价方式。Agon 提出一种竞争性跨模型 RL 方法，让两个模型互为评分者：轮流起草和挑战对方，通过谁解题更优来竞争，从而在训练中隐式地评估推理质量，无需过程标签或奖励模型。

方法

如何在没有过程标签的情况下为推理轨迹提供隐式质量信号？ → 引入竞争性跨模型 RL 框架 Agon：训练两个策略（A 和 B），在一轮中，一个作为起草者生成解法，另一个作为挑战者阅读对方的推理总结后解题，并通过竞争性奖励（正确性加转化奖励）鼓励击败对手；角色每步轮换。两个策略互为评分者，隐式评估推理。

§4p.3Agon trains two policies, A and B, head-to-head on the same problems. The design has four parts: a draft-and-challenge rollout that routes one model's reasoning into the other's context ... a competitive reward that grades the challenger against the drafter ...

Figure 3（第 5 页）：Why a second model. $ \mathbf{Left} $: a single policy optimized on its own rollouts re-audits with the same biases that caused its errors, so self-play saturates [Huang et al., 2024]. $ \mathbf{Right} $: Agon pairs two divergent policies; each grades the other with different blind spots, each is rewarded for winning, and because both are optimized, the grader improves with the graded.

Figure 5（第 7 页）：Peek and outscore. A conceptual view aggregated over one rotation cycle: within a single step the exchange is one-directional (Figure 2); roles swap on the next step, so each model both reads and is read, on different problems. While solving, the challenger reads the rival's worked solution (the summary it writes after its private reasoning, final answer withheld) and writes against it, catching its errors (a sign error in eq. 3), pruning its dead ends, and borrowing its good moves; the annotations are what the reader of the summary exploits, not messages exchanged between the models. Neither solution is labeled, yet by rewarding whoever out-solves the other, the rival's reasoning is graded implicitly, a signal that outcome-only RL does not use.

Figure 6（第 8 页）：Illustrative exchange (hand-constructed for clarity). The drafter drops a sign, distrusts its own (correct-magnitude) result, and spirals into re-exploration; its post-reasoning summary recaps the flawed derivation (but does not contain the boxed answer). The challenger sees only that summary (raw thinking trace dropped, final answer withheld as a precaution), pinpoints the flawed step, re-drives it, and answers correctly in a fraction of the tokens. The solution summary carries the derivation, which is what the challenger exploits.

Table 1（第 9 页）：The ablation matrix. Each cell is a training configuration; everything but the two factors is held fixed. Self-refinement is a single-model control for exchange; the untrained MoA at inference is a zero-training Mixture-of-Agents control.

如何从单一基础模型高效产生两个行为不同的策略？ → 采用双 LoRA 适配器架构：在一个冻结的基础模型上分别训练两个低秩适配器（$\pi_A = \text{base} + \Delta_A$, $\pi_B = \text{base} + \Delta_B$）。通过不同的初始化（A 零初始化，B 小高斯噪声）和交替更新流（起草vs挑战）来维持发散，无需全量双模型。

§4.2p.5We instantiate both as LoRA adapters [Hu et al., 2022] over a single frozen base model ... The second policy costs one extra rank-16 adapter (~10M parameters, ~2% of the base, plus its optimizer state) ... We maintain divergence by (i) different initialization (B starts with small Gaussian noise around the zero matrix; A uses the standard LoRA zero-init) and (ii) different update streams induced by role rotation.

Figure 4（第 6 页）：Designed divergence (schematic of the target regime, not a measurement). Two copies of one model (left) cover the same problems and fail on the same ones, so a second copy adds little. A pair that is equally strong but behaviorally different (right) has complementary error profiles, which is what makes cross-model grading pay off. Our dual-adapter instantiation starts near the left picture (same base, noise-level adapter difference) and relies on divergent initialization and rotated update streams to move toward the right during training; we do not measure how far it gets.

如何设计竞争性奖励以避免梯度消失并使模型关注困难对手样本？ → 使用转化奖励 $c(b_i)(1-c(a_i))$，其中 $c(y) \in \{0,1\}$ 为正确性。该奖励在对手失败时给予挑战者更高权重，且要求组内对手上下文多样化（每个挑战者 rollout 对应不同 draft），以保证组相对优势标准化后仍有梯度信号。共享对手或简单 margin 奖励（$c(b_i)-c(a_i)$）会因动作独立常数项在标准化后归零。

结果

DeepMath-hard 留出集 pass@1 (%)：Agon 达到 $61\%$ pass@1，显著优于零点 ($23\%$)、普通 GRPO ($30\%$)、协同交换 ($46\%$) 和自精炼 ($32\%$) 等基线。在匹配生成预算下，Agon 比普通 GRPO 提高 $+31$ pp，比未经训练的 MoA 级联（$34\%$）提高 $+27$ pp。

对比方法	数值	引用
Zero-shot	23
Vanilla GRPO (baseline)	30
Self-refinement (control)	32
GRPO two-pass self-cascade (control)	35
MoA [Wang et al., 2024] (no train)	34
Competitive, shared opponent (single-pass)	32
Cooperative exchange	46
Agon (competition + exchange)	61

§5.1p.10Information exchange alone (cooperative exchange: 46) already improves over vanilla GRPO (30) ... adding competition (Agon: 61) improves further over cooperation ... The Agon cascade reaches roughly twice vanilla GRPO's held-out pass@1.

Figure 1（第 2 页）：From answer-grading to rival-grading. On a hard-math held-out set, vanilla GRPO lifts a Qwen3 model only modestly over zero-shot, and an untrained Mixture-of-Agents pass (MoA, no train), two models cross-refining at inference, adds a small bump. Training the same two-model setup competitively (Agon) reaches roughly $ 2\times $ GRPO's and $ 1.8\times $ the untrained MoA's pass@1 at the same two-pass inference budget (Table 3). Intermediate ablations appear in Section 5.

Table 3（第 11 页）：Main comparison on Qwen3-0.6B, DeepMath-hard held-out (%), at a matched training generation budget; inference passes per method are noted in the footnote. Single training run per method; 95% Clopper-Pearson confidence intervals over the 300-problem held-out set are shown in Table 4. Values rounded to nearest integer.

Table 4p.1195% Clopper-Pearson confidence intervals over the 300-problem held-out set (single training run per method).

Table 5p.12Model scaling and family generality: held-out pass@1 (%; trained cells: single training run, cascade protocol of Table 3, rounded to the nearest point; zero-shot: single evaluation).

GSM8K pass@1 (%)：Agon 在 GSM8K 上达到 $75\%$，优于零点 ($62\%$) 和普通 GRPO ($68\%$)。

对比方法	数值	引用
Zero-shot	62
Vanilla GRPO	68
Agon	75

MATH-500 pass@1 (%)：Agon 在 MATH-500 上达到 $64\%$，优于零点 ($45\%$) 和普通 GRPO ($52\%$)。

对比方法	数值	引用
Zero-shot	45
Vanilla GRPO	52
Agon	64

CodeContests (EASY) pass@1 (%)：在 CodeContests EASY 难度上，Agon 达到 $34\%$ pass@1，高于普通 GRPO ($24\%$) 和协同交换 ($29\%$)，表明方法可迁移至编程领域。

对比方法	数值	引用
Zero-shot	18
Vanilla GRPO	24
Cooperative exchange	29
Agon	34

平均最终阶段轨迹长度 (tokens)：Agon 的挑战者阶段平均长度为 $3.5$k tokens，远短于普通 GRPO 的 $8.1$k、协同交换的 $5.1$k 和未训练 MoA 的 $6.9$k，显示出推理密度的提升。

对比方法	数值	引用
Zero-shot	6.1k
Vanilla GRPO (baseline)	8.1k
Self-refinement (control)	7.9k
GRPO two-pass self-cascade	8.0k
MoA [Wang et al., 2024] (no train)	6.9k
Competitive, shared opponent	7.4k
Cooperative exchange	5.1k
Agon	3.5k

§5.1p.11Agon ... avg. len 3.5k; Vanilla GRPO 8.1k; Cooperative exchange 5.1k.

Table 8p.14Auxiliary density lever on Qwen3-0.6B (pass@1, same cascade protocol). The length tiebreak (Section 4.4) shortens traces at near-constant accuracy. Not part of the headline recipe.

Figure 7（第 12 页）：(a) Agon sits in the top-left, higher accuracy at shorter traces (lengths are final-stage completions), while GRPO buys length for little accuracy and the untrained MoA adds only a small bump. The arrow is illustrative. (b) Held-out pass@1 at evaluation checkpoints over training. Each point is a single sampled evaluation of a single training run (binomial noise $\sim \pm 5.5$ pp at 300 problems); curves are not averaged over seeds. Self-refinement plateaus just above GRPO, cooperative exchange climbs higher, and competition+exchange (Agon) separates clearly. Agon is still improving at the end of the budget; we stop at the matched budget rather than at convergence.

Figure 15p.12(a) accuracy vs. trace length

Figure 16（第 12 页）：(b) over training

Figure 17p.12mean trace length (k tokens, final stage)

Figure 18（第 12 页）：training problems processed (2N rollouts per problem for every method)

实现

Agon 在冻结的基础模型上训练两个低秩适配器（A 和 B）。训练每步执行起草-挑战流程：起草者从纯问题提示生成 $N$ 个 rollout 并获得普通 GRPO 更新；挑战者阅读每个对手的推理总结（不含最终答案）后生成 $N$ 个对应 rollout，并通过竞争性奖励更新。角色每优化步轮换，因此两个适配器交替学习独立解题和基于对手解题。推理时使用相同的两级级联：一个适配器起草，另一个阅读起草总结后给出最终答案。

**Figure 2**（第 5 页）：One Agon step. The drafter A produces N solutions from the plain prompt and receives a vanilla GRPO update on them (its standalone stream); the challenger B reads each opponent's post-reasoning solution summary in-context (final answer withheld), produces one paired rollout per opponent (N total), and receives the competitive reward, which uses the paired draft's outcome for the conversion bonus. Both adapters are updated every step. Roles rotate every optimizer step, so both models train in both streams. A and B are two LoRA adapters over one frozen base.

双 LoRA 策略对 (Dual-LoRA Policy Pair) — 提供两个行为有别的策略，用于起草和挑战。
设计：基于同一冻结基础模型，分别加入秩为 $16$ 的 LoRA 适配器 $\Delta_A$ 和 $\Delta_B$。A 采用标准零初始化，B 初始化为小高斯噪声；两者更新流不同以维持发散。
原理：参数效率高（额外开销约 $2\%$），通过初始化和梯度差异实现策略分歧。

起草-挑战训练步 (Draft-and-Challenge Step) — 实现双边信息交换和不对称训练。
设计：每步指定一个起草者和一个挑战者。起草者从题面生成 $N$ 个解 $\{a_i\}$，用普通 GRPO 更新；挑战者接收每个对手的推理总结（丢去原始思考链，隐藏最终答案），生成 $N$ 个配对回答 $\{b_i\}$，并计算竞争性奖励。两种更新的优势在各自组内计算。角色每步互换。
原理：起草者保持独立解题能力，挑战者学习利用对手信息并适应困难上下文；轮换使两个模型均获得两种梯度。

竞争性奖励 (Competitive Reward) — 为挑战者提供激励，使其击败对手。
设计：奖励形式为 $R(b_i) = 2c(b_i) + c(b_i)(1-c(a_i))$（加上格式项），其中 $c(y)$ 是正确性。在对手失败时正确解题获得更高奖励（3 vs 2）。要求组内对手上下文不同以保证梯度方向。
原理：转化奖励 $c(b_i)(1-c(a_i))$ 将正确性权重与对手难度耦合，既奖励正确性又鼓励在对手失败处解题。与简单 margin 或共享对手相比，它在组相对标准化后仍能提供梯度。

Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

引言

相关工作

方法

结果

实现