跳转至

文章背景与核心概要

带可验证奖励的强化学习（RLVR）高度依赖组采样（group sampling）来估计优势并稳定策略更新。然而，由于计算资源的限制，实际应用中的组大小（group sizes）通常较小，这导致训练循环中的有限 rollout 集合只能强化那些在采样中显式捕获的正确行为。当组大小受限时，更新往往会遗漏那些罕见的正确轨迹，从而无意中将模型的概率质量（probability mass）集中在更常见、易于采样的解上。

为了解决这一痛点，本文作者推导出了提示词局部尾部遗漏事件（prompt-local tail-miss events）作为组大小函数的概率，展示了其非单调行为，并在此基础上引入了 F-GRPO。受 Focal Loss 的启发，F-GRPO 采用了一种难度感知（difficulty-aware）的缩放系数，用于对高成功率的采样组进行权重降权。

在分类模拟、单解迷宫测试以及基于 Qwen2.5-7B（在 \(N=8\) 条件下）的大语言模型实验中进行的实证测试表明，该方法在不需要增加组大小或额外计算开销的前提下，能够带来一致的性能提升（例如，在 GRPO、DAPO 和 CISPO 框架下均能提高数学任务的 pass@256 指标）。

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

arXiv: 2602.06717 [cs.LG]
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Authors: Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov
Submission History: Submitted on 6 Feb 2026; last revised 2 September 2026 (v3)

Executive Summary

带可验证奖励的强化学习（RLVR）严重依赖组采样来估计优势并稳定策略更新。然而，由于计算限制通常排除了非常大的组，训练只能在有限的 rollout 集合下进行，这只能强化它们所暴露的正确行为。

当组大小在实践中受限时： * 更新经常会错过罕见的正确轨迹，同时保留混合奖励。 * 这种动态会无意中将模型的概率质量集中在更常见、易于采样的解上。

为了解决这个问题，作者推导了提示词局部尾部遗漏事件作为组大小函数的概率——展示了非单调行为——并引入了 F-GRPO。受 Focal Loss 的启发，F-GRPO 利用难度感知缩放系数来降低对高成功率采样组的更新权重。跨分类模拟、单解迷宫测试以及 LLM 实验（在 \(N=8\) 下使用 Qwen2.5-7B）的实证测试表明，该方法在无需增加组大小或额外计算开销的情况下，实现了持续的性能提升（例如，在 GRPO、DAPO 和 CISPO 框架中提高了数学 pass@256 指标）。

Reinforcement Learning with Verifiable Rewards (RLVR) heavily relies on group sampling to estimate advantages and stabilize policy updates. However, because computational constraints often limit group sizes, training loops run with finite rollout sets that can only reinforce the correct behaviors explicitly captured within those samples.

When group sizes are practically restricted: * Updates can frequently miss rare-correct trajectories while retaining mixed rewards. * This dynamic inadvertently concentrates the model's probability mass on more common, easily sampled solutions.

To resolve this, the authors derive the probability of prompt-local tail-miss events as a function of group size—demonstrating non-monotonic behavior—and introduce F-GRPO. Inspired by Focal Loss, F-GRPO utilizes a difficulty-aware scaling coefficient to down-weight updates on high-success sampled groups. Empirical testing across categorical simulations, single-solution maze tests, and LLM experiments (using Qwen2.5-7B at \(N=8\)) show consistent performance boosts (e.g., raising math pass@256 across GRPO, DAPO, and CISPO frameworks) without requiring increased group sizes or extra computational overhead.

Abstract

带可验证奖励的强化学习（RLVR）通常基于组采样来估计优势并稳定策略更新。在实践中，计算限制往往排除了非常大的组，因此训练在有限的 rollout 集合下进行，这只能强化其暴露的常规正确行为。在实际的组大小下，更新可能会错过罕见的正确轨迹，同时包含混合奖励，从而将概率集中在更常见的采样解上。

我们推导了此类提示词局部尾部遗漏事件作为组大小函数的概率，展示了非单调行为，并在分类抽象中描述了未采样正确质量如何在总正确质量增长的同时反而缩小。受此分析的启发，我们提出了一种受 Focal Loss 启发、具备难度感知的缩放系数，它能够降低对高成功率采样组的更新权重。

从经验来看，分类模拟说明了分类设置中的相同效果，Maze 提供了一个单解测试，而 LLM 实验包括具有代表性的 GRPO 组大小扫描，以及跨 GRPO、DAPO 和 CISPO 的固定 \(N\) 迁移。在 \(N=8\) 的 Qwen2.5-7B 上，我们的方法将平均数学 pass@256 从 \(64.1 \to 70.3\) (GRPO)、\(69.3 \to 72.5\) (DAPO) 以及 \(73.2 \to 76.8\) (CISPO) 进行了提升；OOD pass@256 在所有三种情况下也得到了改善，且没有增加组大小或计算成本。

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions.

We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups.

Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-\(N\) transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at \(N=8\), our method improves average math pass@256 from \(64.1 \to 70.3\) (GRPO), \(69.3 \to 72.5\) (DAPO), and \(73.2 \to 76.8\) (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.