跳转至

文章背景与核心概要

3D空间推理是多模态大语言模型（MLLM）理解和交互物理世界的核心能力，但现有模型在精确几何测量、视角转换（自我中心与他者中心视角转换）以及细粒度视觉定位方面表现依然不可靠。传统的解决方案通常依赖于昂贵的大规模微调或专用的3D几何编码器，这导致系统严重绑定于高昂的监督成本和特定的模型架构。

为此，本文介绍了 GraFT，这是一个创新且无需训练的框架，它通过引入紧凑且易于维护的3D场景图（3DSG）来提供缺失的3D结构数据。GraFT通过该图赋予MLLM三项核心能力：（1）利用符号工具实现确定性几何推理，（2）通过鸟瞰图（BEV）渲染理解他者中心布局，以及（3）通过任务相关的自我中心帧实现视觉属性定位。实验证明，GraFT在 ScanQA 上将 CIDEr 指标提升了 27%，在 VSI-Bench 上将冻结的 MLLM 基准性能提升高达 65%，超越了众多主流的闭源商业模型、通用开源模型以及经过微调的空间推理专用模型。

GraFT: A Training-Free Framework for Spatial Reasoning in Multimodal Large Language Models via 3D Scene Graphs

License: CC BY 4.0

Summary

GraFT is an innovative, training-free framework designed to drastically improve the 3D spatial reasoning capabilities of Multimodal Large Language Models (MLLMs). Traditional MLLMs frequently struggle with precise geometric measurements, viewpoint transformations (egocentric to allocentric), and fine-grained visual grounding. Existing remedies usually rely on costly fine-tuning or specialized 3D geometry encoders, coupling the solution to heavy supervision.

Instead, GraFT integrates a compact and easily maintainable 3D Scene Graph (3DSG) to supply the missing 3D structural data. Through this graph, the framework empowers MLLMs with three core capabilities: 1. Deterministic geometry utilizing symbolic tools. 2. Allocentric layout understanding via bird's-eye-view (BEV) rendering. 3. Visual-attribute grounding through task-relevant egocentric frames.

Empirical evaluations show dramatic performance boosts: GraFT raises CIDEr by 27% on ScanQA and elevates frozen MLLM baselines by up to 65% on VSI-Bench, outperforming prominent proprietary models, general-purpose open-source models, and fine-tuned spatial competitors.

Paper Metadata

  • arXiv ID: arXiv:2609.03892 [cs.CV]
  • Primary Subject: Computer Vision and Pattern Recognition (cs.CV)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Robotics (cs.RO)
  • Submission Date: September 3, 2026
  • Authors: Junqing Du, Fernando Ropero, Erkin Turkoz, Yanfeng Zhang, Lu Liu
  • DOI: 10.48550/arXiv.2609.03892

Abstract

3D spatial reasoning underpins understanding and acting in the physical world, yet it remains unreliable in current multimodal large language models (MLLMs). These models falter at precise geometric measurement, at transforming between egocentric and allocentric viewpoints, and at grounding fine-grained appearance. The most common remedies fine-tune the model on large-scale curated spatial-reasoning datasets or attach dedicated encoders for 3D geometry, which typically couples the solution to costly supervision and a specific backbone. We instead introduce GraFT, a training-free framework that supplies the missing 3D structure through a compact, easily maintained 3D scene graph (3DSG). From this 3DSG, GraFT provides three spatial reasoning capabilities: (1) deterministic geometry through symbolic tools, (2) allocentric layout through a bird's-eye-view (BEV) rendering, and (3) visual-attribute grounding through task-relevant egocentric frames. On ScanQA, GraFT improves every metric over the same-backbone baseline, raising CIDEr by 27%. On VSI-Bench, GraFT improves frozen MLLMs by up to 65%, surpassing every proprietary and general-purpose open-source baseline, and several prominent fine-tuned spatial models.

Full-Text and Resources