跳转至

基于 Qwen3-TTS 的声音克隆与定制化语音合成实战

Custom Voices With Qwen3-TTS

文章背景与核心概要

在如今以大语言模型 (Large Language Model, LLM) 为代表的生成式 AI 浪潮中，语音合成技术正从“机械念稿”向“个性化情感表达”飞速演进。传统文本转语音 (Text-to-Speech, TTS) 工具往往发音千篇一律、语调生硬，难以满足独立游戏、互动剧本或个性化内容创作对独特声音气质的要求。知名开源开发者 Armin Ronacher 在为自己和孩子们动手开发的地牢探险游戏制作配音时，深入挖掘了通义千问开源语音模型 Qwen3-TTS 的底层潜力。他发现，借助说话人向量嵌入 (Speaker Embeddings) 的数学特性，开发者不仅能用一段数十秒的日常演讲音频实现高保真声音克隆，更能跨模型架构组合 Base 与 CustomVoice 模型实现“带情绪指令的声音克隆”，甚至在高维向量空间中对声音的性别与风格特征进行代数加减运算。本文完整梳理了这一兼具趣味与实用价值的技术实战，为开发者探索灵活、低成本的定制化语音生成提供了绝佳范例。

内容概述

Summary

在这篇文章中，我们将探索如何利用 Qwen3-TTS 为我和孩子们自己动手制作的地牢探险游戏打造独一无二的定制旁白配音。市面上标准的 AI 语音听起来往往平淡机械、千人一声，而 Qwen3 则支持通过特征向量嵌入 (Embeddings) 实现高精度的声音克隆。通过巧妙组合 Qwen3 旗下的不同模型变体 (Base 基础版、CustomVoice 定制版以及 VoiceDesign 声音设计版)，开发者不仅能够融合不同声音的特征向量，还能为克隆出的声音附加文字风格指令，甚至通过向量运算 (例如平滑调节声音的性别特征) 创造出极具辨识度、非标准化的特色旁白风格。

This post explores how to use Qwen3-TTS to create unique, custom voice-overs for a home-built dungeon crawler game with kids. While standard AI voices sound generic, Qwen3 supports voice cloning via embeddings. By combining different Qwen3 models (Base, CustomVoice, and VoiceDesign), developers can blend voice embeddings, apply text instructions to cloned voices, and even perform vector arithmetic (like gender shifting) to create distinct, non-standard narration styles.

Qwen3 带来了什么？

What Qwen Offers

在当前的开源生态中，Qwen3 并非唯一的文本转语音 (Text-to-Speech, TTS) 模型——在项目起步时，我们也曾尝试过 Kokoro——但 Qwen3 之所以脱颖而出，核心在于它对原生声音克隆提供了出色支持。

Qwen3 is not the only Text-to-Speech (TTS) model available—we initially experimented with Kokoro—but Qwen3 stands out because it supports native voice cloning.

在工程实践中，Qwen3 官方提供了三种功能各异的模型变体： * 1.7B-Base (以及 0.6B 版)：仅凭数秒的参考音频片段，即可精准克隆对应声线。 * 1.7B-CustomVoice：内置了 9 种预设发音人，并支持丰富的语气与风格提示词指令。 * 1.7B-VoiceDesign：能够纯粹根据一段自然语言文字描述，凭空“设计”并生成全新的声音。

In practice, Qwen3 ships with three distinct model variants: * 1.7B-Base (and 0.6B): Clones a voice from a few seconds of reference audio. * 1.7B-CustomVoice: Comes with nine built-in speakers and style instructions. * 1.7B-VoiceDesign: Invents an entirely new voice from a text description.

Base 基础模型既可以通过上下文学习 (In-Context Learning, ICL) 的方式进行克隆，也可以单纯依赖提取出的说话人特征向量 (Speaker Embeddings) 来完成。然而，官方目前并没有提供公开文档或标准接口，来将克隆后的声音与直接的风格提示词指令 (例如要求它“带着悲伤的语气说话”或“轻声耳语”) 结合起来。现阶段官方唯一推荐的替代方案，要么是在单个发音人数据集上对 Base 模型进行监督微调 (Supervised Fine-Tuning, SFT)，要么先用 VoiceDesign 根据描述生成一段音频样本，再将其喂给 Base 模型进行克隆。

The base model clones either in-context or via speaker embeddings alone. However, there is no official, documented way to combine a cloned voice with direct style instructions (such as telling it to "speak sadly" or "whisper"). Currently, the only official alternatives are fine-tuning the Base model on a single speaker or generating a clip via VoiceDesign and cloning it with Base.

动手实操体验

Playing With It

在本次实验中，我截取了自己在 PyAI Conf 技术大会演讲中的一段 28 秒录音。这段音频截取自两次自然停顿之间，并使用 Whisper 完成了转录对齐。

For this experiment, I used a 28-second clip of my talk at PyAI Conf, sliced between two pauses, and transcribed it using Whisper.

技术大会现场的演讲录音显然远远达不到专业录音棚的音质标准，但它恰恰印证了一个极其关键的结论：你根本不需要一尘不染的完美音质或漫长的样本数据，就能为某个人创造出高度逼真、极具说服力的数字声音克隆。

Conference talk recordings are far from studio quality, but they prove an important point: you don't need pristine audio or long samples to create a convincing digital clone of someone's voice.

不仅如此，你还可以将真实录制的音频样本与通过 VoiceDesign 凭空设计出的虚拟声音进行混合。借助 Qwen 官方的 Python 工具包 qwen-tts (pip install qwen-tts，版本为 0.1.1)，我们只需几行代码就能根据文本描述凭空塑造出一段全新声音：

You can blend a recorded sample with another voice generated via VoiceDesign. Using Qwen’s Python package qwen-tts (pip install qwen-tts, version 0.1.1), a designed voice can be created like this:

import torch
from qwen_tts import Qwen3TTSModel

kw = dict(device_map="mps", dtype=torch.bfloat16, attn_implementation="sdpa")
design = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign", **kw)
wavs, sr = design.generate_voice_design(
    text=open("voices/transcript.txt").read(), language="English",
    instruct="A man in his forties with a warm, natural, medium-low voice, speaking "
             "clearly and calmly at an easy conversational pace, like someone reading "
             "aloud to a friend.",
)

💬 交互式演示：请访问原博文试听音频。

💬 [原文引用 / Original Quote]: Interactive demo: see the post on the website.

声音嵌入向量

Voice Embeddings

Base 基础模型会提取出说话人特征向量嵌入——这是由经过训练的说话人编码器输出的一个 2048 维向量。声音越相似，该向量中的数值分布就越接近，这种特性也使得这些向量在基础的说话人识别与音频切分任务中非常实用。

The Base model captures a speaker embedding—a 2048-dimensional vector produced by a trained speaker encoder. Similar voices yield similar numbers, which also makes these vectors useful for basic speaker detection and audio segmentation.

正因为这些嵌入本质上是纯粹的数值向量，我们就可以对它们进行向量代数运算。例如，计算两个声音向量之间的加权平均值，就能融合出一个介于两者之间、兼具二者特质的“混血”新声音：

Because these embeddings are simple numerical vectors, you can perform vector arithmetic on them. For instance, calculating a weighted average between two embeddings creates a hybrid voice sitting midway between the two sources:

base = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-Base", **kw)

def fingerprint(path):
    prompt = base.create_voice_clone_prompt(ref_audio=path, x_vector_only_mode=True)
    return prompt[0].ref_spk_embedding

def blend(a, b, t):
    a, b = a.float(), b.float()
    mixed = (1 - t) * a + t * b
    return (mixed / mixed.norm() * ((1 - t) * a.norm() + t * b.norm())).to(torch.bfloat16)

armin, designed = fingerprint("voices/armin.flac"), fingerprint("voices/designed.flac")

💬 交互式演示：请访问原博文试听音频。

💬 [原文引用 / Original Quote]: Interactive demo: see the post on the website.

如果你对合成出来的这种混合新声音感到满意，还可以将其“烘焙”保存为一个标准的声音参考文件：

If you like a newly synthesized voice blend, you can bake it back into a standard reference file:

import soundfile as sf

mixed = blend(armin, designed, 0.5)
audio, rate = say(open("voices/transcript.txt").read(), mixed)  # say() is shown further down
sf.write("voices/final.flac", audio, rate)
final = fingerprint("voices/final.flac")

说话人识别准确率

Speaker Recognition Accuracy

在针对长达 20 分钟的播客节目 (State of Agentic Coding) 进行说话人识别测试时，如果以大会现场的演讲音频片段作为参考基准，识别完全失败了。然而，如果改用该播客节目本身开头的 20 秒自我介绍作为参考，识别准确率则达到了 98%。相比之下，像 ECAPA 这样专门针对说话人验证训练的专用模型在跨领域表现上要优秀得多，即使直接使用大会演讲片段作为参考，也能以 97% 的准确率成功识别出目标说话人。

When testing Qwen3's embeddings for speaker recognition against a 20-minute podcast episode (State of Agentic Coding), using a conference clip as the reference failed entirely. However, using a 20-second self-introduction from the podcast itself resulted in 98% accuracy. Dedicated verification models like ECAPA performed much better out-of-domain, successfully identifying the speaker from the conference clip 97% of the time.

💬 交互式演示：请访问原博文试听音频。

💬 [原文引用 / Original Quote]: Interactive demo: see the post on the website.

带指令控制的克隆声音

Instructed Cloned Voices

虽然特征向量嵌入在声音克隆上表现出色，但我们该如何给克隆出来的声音附加语气或风格指令呢？

While embeddings excel at voice cloning, how do we apply style instructions to them?

Base 模型与 CustomVoice 模型在底层都使用输入的特征向量来控制说话人的声线。因此，只要将通过 Base 模型提取出的说话人特征向量 (x-vector) 直接喂给 CustomVoice 模型架构，我们就能在克隆声音的同时，利用自然语言指令对其发音风格进行精细调控：

Both the Base and CustomVoice models manipulate speakers using an input embedding vector. By taking the x-vector extracted via the Base model and feeding it into the CustomVoice model's architecture, we can leverage instructions on a cloned voice:

cv = Qwen3TTSModel.from_pretrained("Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", **kw)

def say(text, xvec, instruct=None, language="English"):
    ids = cv._tokenize_texts([cv._build_assistant_text(text)])
    told = [cv._tokenize_texts([cv._build_instruct_text(instruct)])[0]
            if instruct else None]
    voice = {"ref_code": [None], "ref_spk_embedding": [xvec],
             "x_vector_only_mode": [True], "icl_mode": [False]}
    codes, _ = cv.model.generate(
        input_ids=ids, instruct_ids=told, voice_clone_prompt=voice,
        languages=[language], non_streaming_mode=True,
        **cv._merge_generate_kwargs(temperature=0.7, subtalker_temperature=0.7),
    )
    wavs, rate = cv.model.speech_tokenizer.decode([{"audio_codes": codes[0]}])
    return wavs[0], rate

wav, rate = say("I really thought this would work.", armin,
                "Speak in a sad, sorrowful tone, voice low and heavy, on the verge of tears.")

💬 交互式演示：请访问原博文试听音频。

💬 [原文引用 / Original Quote]: Interactive demo: see the post on the website.

在克隆出来的声音上施加风格指令，实际效果出人意料地好——尽管其稳定程度稍逊于模型内置的预设发音人——但这已经足以胜任为电子游戏生成定制化角色配音的绝佳工具。

Instructions on cloned voices work surprisingly well—though less reliably than on built-in speakers—making them ideal for generating bespoke video game voice-overs.

更多有趣的向量数学玩法

Other Fun Math

因为每一个声音都被表示为一个 2048 维向量空间中的点，所以我们可以对其进行各种高阶空间几何变换，例如直接操作声音的性别属性。

Because voices are represented as points in a 2048-dimensional vector space, we can perform advanced spatial transformations, such as manipulating gender attributes.

通过使用 VoiceDesign 分别生成 14 个不同的男性声音与 14 个女性声音，我们可以计算出一条平均的“性别向量轴”，并沿着这条轴平移任意目标声音的特征向量：

By using VoiceDesign to generate 14 distinct male and female voices, we can calculate an average gender vector axis and shift any target voice along it:

women = torch.stack([fingerprint(f"design/f{i}.wav") for i in range(14)]).float()
men = torch.stack([fingerprint(f"design/m{i}.wav") for i in range(14)]).float()
middle = (women.mean(0) + men.mean(0)) / 2
axis = women.mean(0) - men.mean(0)
axis = axis / axis.norm()

def move(xvec, t):
    """Put the voice at position t along the gender axis, keep the rest."""
    x = xvec.float()
    return (x + (t - (x - middle) @ axis) * axis).to(torch.bfloat16)

💬 交互式演示：请访问原博文试听音频。

💬 [原文引用 / Original Quote]: Interactive demo: see the post on the website.

希望这篇技术概述能为你探索 Qwen3-TTS 与声音向量嵌入带来一些富有创意的灵感。请务必将这项技术用于善意与创造性的领域，切勿滥用。

Hopefully, this overview inspires some creative ideas for playing around with Qwen3 TTS and voice embeddings. Use it for good, not evil.