Scene Reasoning

3D Scene Intelligence for Cross-Modal Foundational Search

This project develops a high-fidelity framework for embodied object navigation by leveraging incremental 3D Scene Graphs and foundational Vision-Language Models (VLMs). By moving beyond flat occupancy maps, the system builds a hierarchical semantic representation of the world that captures objects, rooms, and their relationships. The system utilizes Knowledge-Augmented Generation (KAG) to predict structural and semantic properties of unobserved regions, enabling robots to perform complex, cross-modal search missions based on categories, natural language descriptions, or visual exemplars in completely unknown topographies.

3D Scene GraphsVision-Language ModelsKnowledge-Augmented GenerationCross-Modal SearchFoundation Models

Project Aims

Work in progress. No paper has been published from this project yet, so the following describes what it sets out to do rather than published results.

  1. 01

    VLM-Driven Semantic Mapping

    Engineered a neural pipeline that utilizes foundational Vision-Language Models to autonomously synthesize 3D scene graphs in real-time, grounding high-level semantic concepts into metric-topological coordinates.

  2. 02

    Uncertainty-Quantified Neural Perception

    Investigated the integration of probabilistic belief-states within the scene graph to explicitly quantify and manage perception errors, sensor noise, and model hallucinations.

  3. 03

    Knowledge-Augmented Generative Priors (KAG)

    Developed a retrieval-augmented framework to extract architectural and semantic "common sense" from foundational LLMs, allowing the agent to predict the layout and occupancy of unmapped areas.

  4. 04

    Cross-Modal Goal Grounding

    Implemented a unified embedding space that allows agents to ground search targets specified via diverse modalities—semantic categories, free-form text descriptions, or reference images—directly into the 3D scene graph.

  5. 05

    Predictive Graph Completion

    Researched generative methods to "complete" the topological graph of partially explored buildings by reasoning over observed room functions and global spatial priors.

  6. 06

    Zero-Shot Foundation Search

    Demonstrated the system’s ability to locate specific or rare objects without environment-specific training, utilizing the reasoning depth of foundational models to prioritize high-probability frontiers.

  7. 07

    Hierarchical Latent Reasoning

    Validated that reasoning over hierarchical scene graphs significantly reduces the computational complexity of long-horizon search tasks compared to traditional point-cloud or voxel-centric methods.