跳转至

文章背景与核心概要

传统的语言模型后训练方法（如监督微调和强化学习）通常将推理能力直接内化到模型的权重中，这导致了一个黑盒式的架构：推理路径无法被直接检查、逐步验证或轻易迁移至其他模型。为了解决这一痛点，本文引入了一种名为 PLVR（带可验证奖励的程序学习） 的全新后训练框架，它将推理过程外化为由确定性原语和神经原语组成的显式程序。

通过引入“符号反向传播”机制，PLVR 将信用分配（credit assignment）视为一种形式化推导，而非传统的统计估计。实验结果表明，在 LiveCodeBench v6 和 Tau2Bench 基准测试中，采用 PLVR 的 30B 基础模型表现优于强化学习基线，甚至能够比肩规模大一个数量级的最前沿模型，同时大幅降低了对高昂特定任务微调数据的依赖。

Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs

Authors: Vishvesh Bhat
Published: August 28, 2026
Primary Subject: Artificial Intelligence (cs.AI)
arXiv ID: 2608.28421 | DOI: 10.48550/arXiv.2608.28421

Executive Summary

传统的大语言模型（LLM）后训练方法——如监督微调（SFT）和强化学习（RL）——将推理能力直接内部化到模型权重中。这就形成了一种黑盒设置，其中的推理路径无法被直接检查、逐步验证或轻松迁移到其他模型。

Traditional post-training methods for Large Language Models (LLMs)—such as supervised finetuning (SFT) and reinforcement learning (RL)—internalize reasoning capabilities directly into the model's weights. This creates a black-box setup where reasoning paths cannot be directly inspected, checked step-by-step, or easily transferred to other models.

本文引入了 PLVR（Program Learning with Verifiable Rewards，带可验证奖励的程序学习），这是一个崭新的后训练框架，它将推理过程外化为由确定性和神经原语组成的显式程序。通过使用一种称为 符号反向传播（symbolic backpropagation） 的机制，PLVR 将信用分配视为形式化推导，而不是估计。在 LiveCodeBench v6 和 Tau2Bench 上的测试表明，PLVR 使 30B 基础模型能够超越强化学习基线，并匹敌规模大得多的前沿模型，同时显著减少了对昂贵特定任务微调数据的需求。

This paper introduces PLVR (Program Learning with Verifiable Rewards), a novel post-training framework that externalizes reasoning into explicit programs composed of deterministic and neural primitives. Using a mechanism called symbolic backpropagation, PLVR treats credit assignment as a formal derivation rather than an estimation. Tested on LiveCodeBench v6 and Tau2Bench, PLVR enables 30B base models to outperform reinforcement learning baselines and match frontier models significantly larger in scale, all while drastically reducing the need for costly task-specific finetuning data.

Key Concepts & Methodology

  • 外化推理： PLVR 没有将逻辑困在模型权重中，而是构建了由原语库构建的显式程序。
  • 符号反向传播：
  • 每个程序层都带有类型化的本体（typed ontology）。
  • 在输出层针对真实标签（ground truth）计算损失。
  • 要求的输入本体通过原语签名的类型推断向后传播。
  • 这充当了传统神经网络链式法则的精确模拟，使信用分配成为严格的数学推导。
  • 密集奖励： 标准的带可验证奖励的强化学习（RLVR）仅验证最终结果，而 PLVR 生成逐层合同判定（per-step contract verdicts），在整个程序上提供密集的结构化反馈。
  • Externalized Reasoning: Instead of trapping logic inside model weights, PLVR constructs explicit programs built from primitive libraries.
  • Symbolic Backpropagation:
  • Each program layer carries a typed ontology.
  • A loss is computed at the output layer against the ground truth.
  • Required input ontologies are propagated backward via type inference over primitive signatures.
  • This serves as an exact analogue to the traditional neural network chain rule, where credit assignment becomes a rigorous mathematical derivation.
  • Dense Rewards: While standard Reinforcement Learning with Verifiable Rewards (RLVR) only verifies the terminal outcome, PLVR generates per-step contract verdicts that provide dense structural feedback over the entire program.

Experimental Results & Performance

  • 匹配预算基准测试： 在 LiveCodeBench v6 和 Tau2Bench 上，使用 PLVR 增强的 30B 基础模型比标准强化学习方法的平均分高出 27.8分。
  • 缩放优势： 经 PLVR 增强的 30B 模型甚至比大一个数量级的前沿大语言模型高出 13.6分。
  • 数据效率： 单个原语库可以服务于多个基准测试。训练新任务的边际成本降至仅需 100 个程序搜索样本，零新增微调数据。
  • 消融实验洞察： 将损失引导搜索替换为在相同类型可采纳空间上的统一采样（在相等的计算预算下），导致中位程序深度从 65.6 急剧下降到 17.5，这证实了反向传播——而不仅仅是底层类型系统本身——是性能的关键驱动力。
  • Matched-Budget Benchmarks: On LiveCodeBench v6 and Tau2Bench, 30B base models augmented with PLVR outperformed standard reinforcement learning approaches by 27.8 points on average.
  • Scaling Advantage: PLVR-enhanced 30B models even outperformed frontier LLMs an order of magnitude larger by 13.6 points.
  • Data Efficiency: A single primitive library can service multiple benchmarks. The marginal cost of training a new task is reduced to just 100 examples of program search, requiring zero new finetuning data.
  • Ablation Insight: Replacing loss-guided search with uniform sampling over the same type-admissible space (at an equal compute budget) caused the median program depth to collapse from 65.6 down to 17.5, confirming that the backward pass—rather than the underlying type system alone—is the key driver of performance.

Resources & Access

  • 查看论文： arXiv:2608.28421 | PDF 视图
  • 代码与工件： 作者发布了符号反向传播库以及一致性检查器，以方便适配外部原语库。
  • View Paper: arXiv:2608.28421 | PDF View
  • Code & Artifacts: The authors have released the symbolic backpropagation library alongside a conformance checker to facilitate adaptation to external primitive libraries.