文章背景与核心概要

手语视频合成对于改善听障人士的沟通至关重要，但由于手语动作涉及复杂的面部表情和精细的手部姿态，传统生成模型在捕捉局部细节和稳定训练方面往往面临挑战。为此，本技术报告提出了一种新颖的基于损失引导的多专家生成对抗网络（GAN）框架，旨在实现高质量、高效率的手语视频生成。

该架构的核心创新在于引入了全局、手部和头部三个专精判别器，分别引导生成器中对应的专家分支关注特定的视觉区域；同时提出了“联合损失（United Loss）”共识机制，有效稳定了多判别器系统在训练初期的混乱动态。此外，每个分支均采用具有可学习AdaptiveFeatureFusion的双路径卷积-Transformer设计，兼顾了卷积的稳定性和窗口自注意力机制的细节表现。系统在自定义的156GB数据集上进行了评估，变体模型展现出极高的视觉质量和硬件效率，可在消费级显卡上流畅运行，并已在2025年香港前沿科技峰会上进行展示。

基于损失引导多专家GAN的手语视频合成 (Sign Language Video Synthesis via Loss-Guided Multi-Expert GANs)

摘要 (Summary)

本初步技术报告介绍了一种利用损失引导的多专家生成对抗网络（GAN）进行手语视频合成的新颖框架，旨在改善听障人士的沟通体验。

This preliminary technical report introduces a novel framework for sign language video synthesis utilizing a loss-guided multi-expert Generative Adversarial Network (GAN) designed to improve communication for individuals with hearing impairments.

该架构具有三个专精判别器（全局、手部和头部），它们引导生成器中对应的专家分支指向不同的视觉区域。为了稳定早期训练，引入了“联合损失”共识机制。此外，每个分支都采用带有可学习的 AdaptiveFeatureFusion 的双路径卷积-Transformer设计。该系统在包含156GB的自定义数据集上进行了评估，展现出高视觉质量和高效率，可在消费级硬件上运行，并曾在2025年香港前沿科技峰会上展出。

The architecture features three specialized discriminators (global, hand, and head) that guide corresponding expert branches in the generator toward distinct visual regions. To stabilize early-phase training, a "United Loss" consensus mechanism is introduced. Furthermore, each branch adopts a dual-pathway convolutional-transformer design with learnable AdaptiveFeatureFusion. Evaluated on a custom 156GB dataset, the system demonstrates high visual quality and efficiency, running on consumer-grade hardware, and was showcased at the 2025 Hong Kong Frontier Technology Summit.

文档元数据 (Document Metadata)

摘要正文 (Abstract)

本初步技术报告提出了一种使用损失引导的多专家生成对抗网络（GAN）进行手语视频合成的框架，以增强听障人士的沟通能力。

This preliminary technical report presents a framework for sign language video synthesis using a loss-guided multi-expert Generative Adversarial Network (GAN) to enhance communication for individuals with hearing impairments.

三个专精判别器——全局判别器、手部判别器和头部判别器——各自引导生成器中的对应专家分支流向不同的视觉区域，从而在没有显式多样性损失的情况下实现隐式特征专业化。为了稳定这个多判别器系统（其早期训练否则会表现出混乱的动态），我们引入了一个联合损失共识机制，该机制以10%的权重将每个判别器正则化至集成平均值。

Three specialized discriminators -- global, hand, and head -- each guide a corresponding expert branch in the generator toward a distinct visual region, enabling implicit feature specialization without explicit diversity losses. To stabilize this multi-discriminator system, whose early-phase training otherwise exhibits chaotic dynamics, we introduce a United Loss consensus mechanism that regularizes each discriminator toward the ensemble average at a 10% weight.

此外，每个分支采用具有可学习 AdaptiveFeatureFusion 的双路径卷积-Transformer设计，平衡了卷积的稳定性和窗口自注意力的细节。生成器使用交替的三模式调度（判别器、整体生成、分支专精生成）进行训练。

Each branch further adopts a dual-pathway convolutional-transformer design with learnable AdaptiveFeatureFusion, balancing the stability of convolutions against the detail of windowed self-attention. The generator is trained using an alternating three-mode schedule (discriminator, holistic generation, branch-specialized generation).

在一个移除了简单和重复样本的经过滤测试集的自定义156GB数据集上，我们的 0.2B参数 变体实现了 29.8 PSNR，而 1.3B参数 变体实现了 30.7 PSNR，推理显存占用分别为 1.5 GB 和 8 GB，从而能够在消费级硬件上进行部署。由于单GPU上2-3个月的训练周期，完整的消融实验仍在进行中。该系统已在2025年香港前沿科技峰会上展出。

On a custom 156GB dataset with a filtered test set that removes easy and repetitive samples, our 0.2B-parameter variant achieves 29.8 PSNR and the 1.3B-parameter variant achieves 30.7 PSNR, with inference VRAM footprints of 1.5 GB and 8 GB respectively, enabling deployment on consumer-grade hardware. Full ablation studies remain ongoing due to the 2-3 month training cycle on a single GPU. The system was showcased at the 2025 Hong Kong Frontier Technology Summit.

核心架构亮点 (Key Architecture Highlights)

• 多专家判别器： 专门针对全局、手部和头部区域，以捕捉细粒度的手语动作。
• Multi-Expert Discriminators: Targets global, hand, and head regions specifically to capture fine-grained sign language movements.
• 联合损失共识： 通过将每个判别器正则化至集成平均值来稳定训练动态。
• United Loss Consensus: Stabilizes training dynamics by regularizing each discriminator toward the ensemble average.
• 双路径设计： 结合了卷积稳定性和带有可学习 AdaptiveFeatureFusion 的窗口自注意力Transformer。
• Dual-Pathway Design: Combines convolutional stability with windowed self-attention transformers using learnable AdaptiveFeatureFusion.
• 资源高效性： 在低显存需求（1.5 GB 至 8 GB）下实现高保真度（高达 30.7 PSNR），非常适合消费级硬件。
• Resource Efficiency: Achieves high fidelity (up to 30.7 PSNR) with low VRAM requirements (1.5 GB to 8 GB), making it suitable for consumer hardware.

全文与访问链接 (Full-Text & Access Links)

• 查看许可证 (知识共享署名 4.0)
• View License (Creative Commons Attribution 4.0)
• 在 arXiv 上查看 PDF
• View PDF on arXiv
• 实验性 HTML 版本
• Experimental HTML Version
• TeX 源代码
• TeX Source Code

引用与参考文献 (Citation & References)

• BibTeX 格式： 可通过 arXiv abs 页面 获取。
• BibTeX Format: Available via the arXiv abs page.
• 外部索引：
• External Indices:
• Google 学术 (Google Scholar)
  • Google Scholar
• Semantic Scholar
  • Semantic Scholar
• NASA ADS
  • NASA ADS