跳转至

文章背景与核心概要

在当前的文本到图像生成领域，高效的生成过程通常需要将基于强化学习（RL）的奖励对齐与少步蒸馏技术结合起来。然而，传统的做法往往将这两个步骤串行执行，这不仅显著增加了训练成本，还存在模型压缩过程中丢失来之不易的奖励增益的风险。

为了解决这一痛点，本文作者提出了一种RL原生视角：扩散强化学习本质上能够生成带有奖励评分的有限步轨迹，其中间状态并非只是用完即弃的采样副产物，而是天然且极具价值的蒸馏监督信号源。基于此，研究团队推出了REST（奖励增强评分轨迹蒸馏）框架，通过优势调制蒸馏（AMD）机制，在无需额外图像采样或对抗训练的情况下，实现了少步无分类器引导（CFG-free）推理，其性能达到甚至超越了40步的RL教师模型。

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

arXiv ID: arXiv:2608.09226
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Authors: Yuhan Li, Fangao Zeng, Sicong Kang, Mengfei Xu, Hao Zhou, Wei Li, Pipei Huang, Bingbing Ni
Submission Date: August 10, 2026

📌 Summary

Efficient text-to-image generation typically requires combining reinforcement learning (RL)-based reward alignment with few-step distillation. However, these two procedures are usually executed sequentially, which significantly increases training costs and risks losing hard-earned reward gains during model compression.

To overcome this, the authors propose an RL-native perspective: diffusion RL inherently generates reward-scored finite-step trajectories, where intermediate states serve as a natural, highly valuable source of distillation supervision rather than just disposable sampling byproducts.

Key Contributions:

  • REST (Reward-Enhanced Scored-Trajectory Distillation): A novel, lightweight, single-stage RL-distillation co-training framework that attaches a decoupled student model to an arbitrary RL teacher. The student learns segment-wise from the teacher's evolving rollout trajectories without modifying the original teacher optimization.
  • Advantage-Modulated Distillation (AMD): A mechanism introduced to prevent uniform imitation from preserving undesirable low-reward behaviors. AMD transforms rollout advantages into signed weights over a base distillation loss, effectively reinforcing preferred trajectories while gently repelling the student away from low-reward ones.
  • Plug-and-Play Efficiency: The framework requires no extra image rollouts, no separate distillation datasets, and no adversarial training.
  • Superior Performance: Experiments across compositional generation, visual text rendering, and human-preference alignment demonstrate that REST achieves few-step CFG-free inference matching or exceeding its 40-step RL teacher. Notably, it improves the DrawBench PickScore over RTDMD by 0.82 while using only one-fifth of the training iterations, keeping additional training overhead under 25% compared to pure RL.
  • REST（奖励增强评分轨迹蒸馏）： 一种新颖、轻量级的单阶段 RL-蒸馏协同训练框架，它将一个解耦的学生模型附加到任意的 RL 教师模型上。学生模型通过分段的方式向教师不断演进的 rollout 轨迹学习，同时不修改原有的教师优化过程。
  • 优势调制蒸馏（AMD）： 引入该机制是为了防止均匀模仿保留不良的低奖励行为。AMD 将 rollout 优势转化为基础蒸馏损失上的带符号权重，从而有效地强化偏好的轨迹，同时温和地将学生模型推开低奖励轨迹。
  • 即插即用的效率： 该框架不需要额外的图像 rollout、单独的蒸馏数据集，也不需要对抗训练。
  • 卓越的性能： 在组合生成、视觉文本渲染和人类偏好对齐等方面的实验表明，REST 实现了少步无 CFG 推理，其性能达到甚至超越了 40 步的 RL 教师模型。值得注意的是，它在仅使用 五分之一 训练迭代次数的情况下，将 DrawBench PickScore 相比 RTDMD 提升了 0.82，并且与纯 RL 相比，额外的训练开销保持在 25% 以下。

License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International