CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding
Jianjun Zhang and Hanli Wang
Overview:
Large vision-language models have shown strong capability in visual grounding, yet existing 3D visual grounding methods are largely limited to small-scale indoor environments or rely heavily on manual annotations. In city-scale scenes, the task becomes substantially more challenging due to the massive search space, complex spatial relations, and scarce supervision. To address these issues, we propose CityVG, the first zero-shot framework for city-scale 3D visual grounding. Our method follows a retrieval-and-reasoning paradigm. In the retrieval stage, we introduce a contrastive fine-tuning strategy based on a Visual Pairwise Scene Graph, where an LLM-driven graph clustering mechanism automatically constructs positive and negative training pairs to adapt the text encoder to urban scenes. In the reasoning stage, we propose a multi-trajectory reward-based Chain-of-Thought mechanism that evaluates candidate objects across diverse reasoning paths and selects the target that best satisfies both appearance and spatial constraints. By combining efficient candidate pruning with robust visual-spatial reasoning, CityVG achieves strong zero-shot grounding performance in large urban environments.
Method:
The overall pipeline of CityVG (see Fig. 1) consists of two stages: retrieval and reasoning. Given a city-scale point cloud and a textual query, we first perform open-vocabulary 3D segmentation to obtain instance-level objects and construct instance-centric multi-scale, multi-view representations. In the retrieval stage, we build a Visual Pairwise Scene Graph that connects candidate instances with surrounding landmark anchors through VLM-generated relational descriptions. Based on these graph representations, an LLM-driven graph clustering strategy is used to derive pseudo positive and negative pairs, which are then employed to contrastively fine-tune a BGE-based text encoder for urban scene retrieval. At inference time, the fine-tuned encoder scores all instances against the input query and retrieves a compact candidate set. In the reasoning stage, each candidate is represented as a structured visual sequence and fed into a VLM for multi-trajectory Chain-of-Thought inference. For each reasoning trajectory, category consistency, appearance consistency, and spatial consistency are evaluated and aggregated into a reward score. Finally, a Best-of-M strategy selects the most reliable reasoning trajectory for each candidate, and the candidate with the highest final reward is returned as the grounded object.
Fig. 1. Overview of the proposed CityVG framework
Results:
The quantitative comparison results of the proposed CityVG and previous state-of-the-art methods on the CityRefer, CityAnchor, and ScanRefer benchmarks are shown in Tables 1 and 2. Overall, CityVG achieves strong zero-shot grounding performance and even outperforms many supervised methods, demonstrating its effectiveness and generalization ability in both city-scale and indoor 3D visual grounding scenarios.
Table 1: Quantitative results on the CityRefer and CityAnchor datasets. "NO" and "ND" denote "Novel Objects" and "Novel Descriptions", respectively. "†"indicates that supervised signals are used during BGE fine-tuning.
Table 2: Comparison of 3DVG performance on the ScanRefer validation set. Results are reported for the "Unique" subset (single-target scenes), the "Multiple" subset (scenes with same-class distractors), and the "Overall" split.
Representative qualitative results on the CityRefer benchmark are shown in Fig. 2, further demonstrating the effectiveness of CityVG in candidate retrieval and final target grounding.
Fig. 2: Qualitative results on the CityRefer benchmark. The visualization displays 2D maps derived from city-scale point clouds via top-view and oblique-view projections. Candidate objects retrieved by the Contrastive Fine-Tuning (CFT) stage are highlighted with distinct colored masks. In the textual query, key semantic components are color-coded: the target object is marked in red, appearance attributes in orange, landmarks in blue, and spatial relationships in green. The final grounded object is indicated by a red bounding box.
Source Code:
Citation:
Please cite the following paper if you find this work useful:
Jianjun Zhang and Hanli Wang, CityVG: Contrastive Fine-Tuning and Reward-Based Chain-of-Thought Reasoning for Zero-Shot City-Scale 3D Visual Grounding, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL'26), San Diego, California, USA, pp. 35012-35031, Jul. 2-7, 2026.