跳转至

文章背景与核心概要

带可验证奖励的强化学习（RLVR）显著提升了多模态大模型（MLLMs）的准确性，但这些性能增益往往较为脆弱。简单的提示词释义或模板修改就可能导致性能下降，从而在医疗视觉问答（VQA）等高风险应用场景中带来严重的可靠性风险。

本文指出了标准强化学习目标函数中的两个核心缺陷：1. 格式与内容的混淆（Format-Content Conflation）：二元验证器无法区分事实错误的答案与仅存在格式错误的答案；2. 分布狭窄性（Distributional Narrowness）：训练分布仅覆盖了真实世界提示词极小的一部分，导致模型在未见过的测试提示词下表现出不可预测的行为。为了解决这些漏洞，作者提出了提示词不变强化学习（Prompt-Invariant RLVR, PIRL），它引入了动态三元奖励以及基于嵌入空间对抗者的稠密一致性正则化器。在严格的压力测试和动态评估下，PIRL展现出了卓越的泛化鲁棒性，其性能降幅保持在 \(\le 1\%\)，而 GRPO 的降幅约为 3%。

Improving Generalization Robustness of Multimodal RLVR

arXiv: 2608.08802 [cs.AI]
Submitted: August 9, 2026
Authors: Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You

arXiv: 2608.08802 [cs.AI]
Submitted: August 9, 2026
Authors: Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You

📌 Executive Summary

带可验证奖励的强化学习（RLVR）显著提升了多模态大模型的准确性，但这些增益往往很脆弱：简单地对问题进行释义或更改提示词模板就会导致性能下降，这对医疗 VQA 等高风险场景下的可靠部署提出了挑战。

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA.

本文将此追溯到标准 RL 目标的两个问题。首先，二元验证器将格式与内容混淆，因此奖励信号无法将错误答案与格式错误的答案区分开来。其次，训练分布仅覆盖了模型在部署时可能遇到的真实世界提示词的一小部分，因此在训练分布上表现良好的策略在测试时遇到未见过的提示词时可能会有不同的表现。这两种失效都呼唤一种强大的后训练方法，帮助策略覆盖更广泛的语义等效提示词分布，我们确定了有助于实现这一目标的两个措施：在奖励中将格式与语义分离，以及在具有等效语义的受扰动提示词之间应用策略不变性。因此，我们提出了提示词不变 RLVR（PIRL），它由动态三元奖励和基于嵌入空间对抗者的稠密一致性正则化器组成。在压力测试下，PIRL 在基准测试上的平均准确率降幅仅为 \(\le 1\%\)，而 GRPO 的降幅约为 3%。在动态评估中，PIRL 也实现了最小的性能降幅。

We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only \(\le 1\%\), where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.

📖 Abstract

📖 Abstract

(注：此处摘要内容与前文 Executive Summary 中的英文引文对应，已在上文翻译中融合呈现。)

🛠️ Key Components of PIRL

  • 动态三元奖励：将格式正确性与语义正确性解耦，为策略模型提供更清晰的反馈循环。
  • 嵌入空间对抗者：充当一致性正则化器，在语义等效但受到扰动的提示词之间强制实现策略不变性。
  • Dynamic Trinary Reward: Decouples format correctness from semantic correctness to deliver clearer feedback loops for the policy model.
  • Embedding-Space Adversary: Acts as a consistency regularizer enforcing policy invariance across semantically equivalent yet perturbed prompts.

📊 Evaluation & Results

  • 压力测试：PIRL 将基准测试准确率的下降控制在 \(\le 1\%\) 以内，优于遭遇约 \(3\%\) 下降的标准 GRPO。
  • 动态评估：在不断变化的评估条件下，在所有测试方法中保持了最稳定的性能表现。
  • Stress Testing: PIRL limits benchmark accuracy drops to \(\le 1\%\), outperforming standard GRPO, which suffers around a \(3\%\) drop.
  • Dynamic Evaluation: Maintains the most stable performance profile among tested methods under changing evaluation conditions.