通过 100 步 GRPO 微调 350M 模型以获得更好的结构化输出

文章背景与核心概要

本文介绍了一种低成本且公开可复现的方案，旨在显著提升小型语言模型（SLM）的结构化输出合规性。通过利用 TRL 库通过群相对策略优化（GRPO）对 LFM2.5-350M 模型进行微调，并在 IFStruct 基准测试上进行评估，模型性能在仅 100 个训练步骤（约 500 个样本）内实现了从 22.6% 到 29.7% 的跃升。

该工作流非常轻量，足以在免费级别的 Colab 或 Kaggle GPU 上运行。它展示了如何通过任务特定的奖励塑造（Reward Shaping），弥补小型模型在严格遵循结构化输出方面的差距，从而使其成为边缘设备和受限部署环境下的可行替代方案。

Executive Summary

This guide outlines a budget-friendly, public recipe for dramatically improving the structured-output compliance of a small language model. By fine-tuning LFM2.5-350M using Group Relative Policy Optimization (GRPO) via the TRL library and evaluating it on the IFStruct benchmark, performance increases from 22.6% to 29.7% in just 100 training steps (~500 samples). This workflow is light enough to run on a free-tier Colab or Kaggle GPU.

本指南概述了一种低成本、公开的方案，可大幅提高小型语言模型的结构化输出合规性。通过使用 TRL 库通过群相对策略优化（GRPO）对 LFM2.5-350M 进行微调，并在 IFStruct 基准测试上进行评估，在仅 100 个训练步骤（约 500 个样本）中，性能从 22.6% 提升至 29.7%。该工作流足够轻量，可以在免费版的 Colab 或 Kaggle GPU 上运行。

Introduction

Structured output generation is vital for integrating LLMs into automated systems, requiring models to reliably return valid, parseable schemas. However, smaller models often struggle with strict schema adherence.

Note: The pipeline detailed here is designed to demonstrate how task-specific fine-tuning on smaller models can elevate compliance to match much larger models, rather than replicating the exact training stack of the official IFStruct RL release.

结构化输出生成对于将大语言模型（LLM）集成到自动化系统中至关重要，它要求模型能够可靠地返回有效且可解析的架构（Schema）。然而，较小的模型往往难以严格遵守架构。

注：此处详细介绍的流程旨在演示如何通过对较小模型进行特定任务的微调，将合规性提升至与更大模型相媲美的水平，而不是复制官方 IFStruct RL 发布的精确训练技术栈。

Prerequisites

The process is divided into two distinct environments: 1. Fine-tuning: Performed on a cloud GPU (e.g., Google Colab / Kaggle free tiers). 2. Evaluation: Performed locally using llama.cpp to run an OpenAI-compatible server that interfaces with the IFStruct evaluator.

Setting up Tooling

Install uv for Python management and llama.cpp for model serving:

brew install llama.cpp
llama-server --version

前提条件

该过程分为两个不同的环境： 1. 微调： 在云 GPU 上执行（例如 Google Colab / Kaggle 免费层）。 2. 评估： 在本地使用 llama.cpp 执行，以运行一个与 IFStruct 评估器对接的兼容 OpenAI 的服务器。

设置工具

安装用于 Python 管理的 uv 和用于模型服务的 llama.cpp：
brew install llama.cpp
llama-server --version

IFStruct Evaluation on LFM2.5-350M (Base Model)

Before fine-tuning, we evaluate the baseline LFM2.5-350M model using the IFStruct benchmark.

Clone the benchmark repository:

git clone https://github.com/Liquid4All/ifstruct.git

Serve the base model locally using the BF16 GGUF variant (LiquidAI/LFM2.5-350M-GGUF):

llama-server \
  -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --alias LiquidAI/LFM2.5-350M \
  --host 127.0.0.1 \
  --port 8080

Run the benchmark evaluation across 2,000 samples:

uv run ifstruct-eval \
  --model LiquidAI/LFM2.5-350M \
  --base-url http://localhost:8080/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-llamacpp-base.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v

Baseline Results: * Overall: 452/2000 passed (22.6%) * JSON: 18.0% passed * YAML: 27.2% passed

LFM2.5-350M（基准模型）的 IFStruct 评估

在微调之前，我们使用 IFStruct 基准测试评估基准 LFM2.5-350M 模型。

克隆基准测试仓库：
git clone https://github.com/Liquid4All/ifstruct.git
使用 BF16 GGUF 变体（LiquidAI/LFM2.5-350M-GGUF）在本地部署基础模型：
llama-server \
  -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --alias LiquidAI/LFM2.5-350M \
  --host 127.0.0.1 \
  --port 8080
运行跨 2,000 个样本的基准评估：
uv run ifstruct-eval \
  --model LiquidAI/LFM2.5-350M \
  --base-url http://localhost:8080/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-llamacpp-base.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v
基线结果： * 总体： 452/2000 通过（22.6%） * JSON： 18.0% 通过 * YAML： 27.2% 通过

GRPO Fine-tuning with TRL on Structured Outputs

The complete training script is available in the accompanying notebook.

1. Training Data & Augmentation

We leverage nvidia/Nemotron-RL-instruction_following-structured_outputs, pairing prompts with target JSON schemas and field counts. To match the diversity of the IFStruct evaluation: * 40% of prompts include an appended instruction to return outputs inside a fenced code block. * 20% are restructured into top-level array tasks to enforce bare-list and item-count compliance.

2. Model and LoRA Configuration

We target the LFM-specific module names for adapter placement (~6M trainable parameters):

from peft import LoraConfig

lora_config = LoraConfig(
    r=16, 
    lora_alpha=32, 
    bias="none", 
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)

3. Reward Functions

Completions are scored on a [0, 1] scale across three core metrics: * json_format_reward: Evaluates parseability and format correctness (fenced vs. raw). * field_count_reward: Scores top-level field count accuracy. * schema_validation_reward: Validates outputs against the target JSON schema.

Weights are set via reward_weights=[1.0, 0.5, 2.0].

4. Training Execution

We configure GRPO for a 16 GB GPU constraint:

from trl import GRPOConfig

training_args = GRPOConfig(
    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
    learning_rate=5e-5,
    max_steps=100,
    warmup_steps=10,
    num_generations=8,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    steps_per_generation=2,
    max_completion_length=1024,
    mask_truncated_completions=False,
    temperature=1.1,
    beta=0.01,
    reward_weights=[1.0, 0.5, 2.0],
    logging_steps=1,
    save_steps=100,
)

5. Merging and Saving Weights

MERGED_DIR = f"{training_args.output_dir}-merged"

merged_model = trainer.model.merge_and_unload()
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)

使用 TRL 在结构化输出上进行 GRPO 微调

完整的训练脚本可在配套的笔记本中查看。

1. 训练数据与数据增强

我们利用 nvidia/Nemotron-RL-instruction_following-structured_outputs，将提示词与目标 JSON 架构和字段计数进行配对。为了匹配 IFStruct 评估的多样性： * 40% 的提示词包含附加指令，要求将输出返回到围栏代码块（fenced code block）中。 * 20% 被重构为顶级数组任务，以强制执行纯列表（bare-list）和项数合规性。

2. 模型与 LoRA 配置

我们针对 LFM 特定的模块名称进行适配器放置（约 600 万个可训练参数）：
from peft import LoraConfig

lora_config = LoraConfig(
    r=16, 
    lora_alpha=32, 
    bias="none", 
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)
3. 奖励函数

完成的内容在三个核心指标上按 [0, 1] 的标度进行评分： * json_format_reward：评估可解析性和格式正确性（带围栏与原始）。 * field_count_reward：对顶级字段计数准确性进行评分。 * schema_validation_reward：根据目标 JSON 架构验证输出。

权重通过 reward_weights=[1.0, 0.5, 2.0] 设置。

4. 训练执行

我们针对 16 GB GPU 的限制配置 GRPO：
from trl import GRPOConfig

training_args = GRPOConfig(
    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
    learning_rate=5e-5,
    max_steps=100,
    warmup_steps=10,
    num_generations=8,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    steps_per_generation=2,
    max_completion_length=1024,
    mask_truncated_completions=False,
    temperature=1.1,
    beta=0.01,
    reward_weights=[1.0, 0.5, 2.0],
    logging_steps=1,
    save_steps=100,
)
5. 合并并保存权重
MERGED_DIR = f"{training_args.output_dir}-merged"

merged_model = trainer.model.merge_and_unload()
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)

IFStruct Evaluation on GRPO-Tuned LFM2.5-350M

After fine-tuning, the weights are converted into a BF16 GGUF file for local serving via llama.cpp:

git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install ./llama.cpp/gguf-py

mkdir -p models
python llama.cpp/convert_hf_to_gguf.py \
  PATH_TO_YOUR_MERGED_MODEL \
  --outfile ./models/lfm25-350m-grpo-bf16.gguf \
  --outtype bf16

# Serve the fine-tuned model
llama-server \
  -m ./models/lfm25-350m-grpo-bf16.gguf \
  --alias lfm25-350m-grpo-structured-output \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8081

Run the benchmark evaluation again:

uv run ifstruct-eval \
  --model lfm25-350m-grpo-structured-output \
  --base-url http://localhost:8081/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-grpo.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v

对 GRPO 微调后的 LFM2.5-350M 进行 IFStruct 评估

微调完成后，将权重转换为 BF16 GGUF 文件，以便通过 llama.cpp 在本地提供服务：

git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install ./llama.cpp/gguf-py

mkdir -p models
python llama.cpp/convert_hf_to_gguf.py \
  PATH_TO_YOUR_MERGED_MODEL \
  --outfile ./models/lfm25-350m-grpo-bf16.gguf \
  --outtype bf16

# 部署微调后的模型
llama-server \
  -m ./models/lfm25-350m-grpo-bf16.gguf \
  --alias lfm25-350m-grpo-structured-output \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8081

再次运行基准测试评估：

uv run ifstruct-eval \
  --model lfm25-350m-grpo-structured-output \
  --base-url http://localhost:8081/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-grpo.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v

Results & Comparison

IFStruct Group	Base Model	GRPO-Tuned Model	\(\Delta\)
Overall	22.6%	29.7%	+7.1%
JSON	18.0%	31.9%	+13.9%
YAML	27.2%	27.5%	+0.3%
Wrapper Key	28.5%	29.7%	+1.2%
Bare List	16.6%	29.7%	+13.1%

The fine-tuning run successfully targets and fixes JSON structure generation, lifting overall accuracy by nearly 7 absolute points and jumping JSON compliance by almost 14%.

结果与对比

IFStruct 组别基准模型 GRPO 微调模型 \(\Delta\)

总体 (Overall) 22.6% 29.7% +7.1%

JSON 18.0% 31.9% +13.9%

YAML 27.2% 27.5% +0.3%

包装键 (Wrapper Key) 28.5% 29.7% +1.2%

纯列表 (Bare List) 16.6% 29.7% +13.1%

这次微调运行成功针对并修复了 JSON 结构生成问题，将总体准确率提升了近 7 个绝对百分点，并将 JSON 合规性提高了近 14%。

Conclusion

A lightweight reinforcement learning pass using GRPO over ~500 samples and 100 steps can significantly enhance a 350M parameter model's structured-output execution. Task-specific reward shaping bridges the performance gap, making small models viable alternatives for edge and constrained deployments.

结论

通过在大约 500 个样本和 100 个步骤上使用 GRPO 进行轻量级强化学习传递，可以显著增强 350M 参数模型的结构化输出执行能力。特定任务的奖励塑造弥补了性能差距，使小型模型成为边缘部署和受限部署的可行替代方案。

IFStruct 组别	基准模型	GRPO 微调模型	\(\Delta\)
总体 (Overall)	22.6%	29.7%	+7.1%
JSON	18.0%	31.9%	+13.9%
YAML	27.2%	27.5%	+0.3%
包装键 (Wrapper Key)	28.5%	29.7%	+1.2%
纯列表 (Bare List)	16.6%	29.7%	+13.1%