跳转至

文章背景与核心概要

在大语言模型（LLM）的推理过程中，实时分类对于安全过滤、行为分析以及模型监控至关重要。然而，传统方法往往在效率和准确性之间陷入权衡困境：隐状态探测器速度虽快，但缺乏上下文感知或计算成本高昂；而专用分类模型（如 Llama Guard、Qwen Guard）虽然准确率高，却带来了沉重的计算负担。

本文引入了一种名为“投机探测”（Speculative Probing）的新方法，它巧妙地将近期大语言模型中的投机解码模块重新用作序列分类器。通过在投机解码流水线的目标序列中附加一个训练好的软提示（soft prompt），由于键值（KV）缓存已经驻留在 GPU 内存中，分类过程几乎不产生额外的计算开销。

在四个分类任务和四个模型（Qwen3.5-4B、9B、27B 和 MiniCPM4.1-8B）上的评估表明，这些紧凑的探测器在性能上 consistently 持续超越 zero-shot 的 GPT-5.4-mini，并且在无需完整调用大语言模型的情况下，能够媲美甚至超过专门的 8B 安全分类器（如 Qwen3Guard-Gen-8B 和 Llama-Guard-3-8B）。

Speculative Probing: LLM Monitoring at Speculative-Decoding Cost

arXiv: 2608.28099 [cs.AI]
Authors: Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
Submitted: 28 August 2026
License: Creative Commons Attribution 4.0 license icon

Speculative Probing: LLM Monitoring at Speculative-Decoding Cost

arXiv: 2608.28099 [cs.AI]
Authors: Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
Submitted: 28 August 2026
License: Creative Commons Attribution 4.0 license icon

Summary

Real-time classification during language model inference is essential for safety filtering, behavioral analysis, and model monitoring. However, traditional approaches force an efficiency-accuracy trade-off: * Hidden-state probes are fast, but they either lack context awareness (operating on a single vector) or incur massive computational costs (pooling hidden states across all tokens). * Dedicated classifier models (e.g., Llama Guard, Qwen Guard, LLM-as-judge) offer high accuracy at a heavy computational price.

This paper introduces Speculative Probing, a method that repurposes the speculative-decoding module in recent Large Language Models (LLMs) to function as a sequence classifier. By appending a trained soft prompt to the target sequence within a speculative-decoding pipeline, classification adds virtually no overhead since the Key-Value (KV) cache is already resident in GPU memory.

Evaluated across four classification tasks and four models (Qwen3.5-4B, 9B, 27B, and MiniCPM4.1-8B), these compact probes consistently outperform zero-shot GPT-5.4-mini and match or exceed specialized 8B safety classifiers (Qwen3Guard-Gen-8B, Llama-Guard-3-8B) without requiring a full LLM invocation.

Summary

Real-time classification during language model inference is essential for safety filtering, behavioral analysis, and model monitoring. However, traditional approaches force an efficiency-accuracy trade-off: * Hidden-state probes are fast, but they either lack context awareness (operating on a single vector) or incur massive computational costs (pooling hidden states across all tokens). * Dedicated classifier models (e.g., Llama Guard, Qwen Guard, LLM-as-judge) offer high accuracy at a heavy computational price.

This paper introduces Speculative Probing, a method that repurposes the speculative-decoding module in recent Large Language Models (LLMs) to function as a sequence classifier. By appending a trained soft prompt to the target sequence within a speculative-decoding pipeline, classification adds virtually no overhead since the Key-Value (KV) cache is already resident in GPU memory.

Evaluated across four classification tasks and four models (Qwen3.5-4B, 9B, 27B, and MiniCPM4.1-8B), these compact probes consistently outperform zero-shot GPT-5.4-mini and match or exceed specialized 8B safety classifiers (Qwen3Guard-Gen-8B, Llama-Guard-3-8B) without requiring a full LLM invocation.