Embodied Navigation

Cognitive Architectures for Long-Horizon Semantic Navigation with Visual-Language Priors

This project explores the integration of foundational Vision-Language Models (VLMs) and Large Language Models (LLMs) to redefine the cognitive architecture of intelligent navigation. We investigated a framework that leverages the zero-shot reasoning capabilities of internet-scale foundational models alongside a structured spatial-semantic memory. This enables embodied agents to perform complex, language-driven semantic search tasks—such as finding specific objects described in everyday natural language—by reasoning over environmental uncertainty and past observations in completely novel environments.

Embodied AILong-Horizon ReasoningVisual-Language PriorsZero-Shot NavigationSemantic Search

Project Aims

Work in progress. No paper has been published from this project yet, so the following describes what it sets out to do rather than published results.

  1. 01

    Foundational VLM-Based Semantic Memory

    Engineered a dual-indexed memory system that utilizes high-dimensional feature embeddings from foundational models to pair visual observations with real-time semantic goal similarity.

  2. 02

    Zero-Shot Object Navigation (ZSON)

    Leveraged the internet-scale knowledge of foundational VLMs to enable agents to locate novel objects without task-specific training, facilitating generalization across a wide variety of previously unseen items and environments.

  3. 03

    Uncertainty-Aware Foundational Reasoning

    Investigated the decomposition of cognitive uncertainty by measuring the confidence of foundational models in interpreting complex natural language instructions (e.g., "find a wooden toy airplane").

  4. 04

    LLM-Driven Decision Coordination

    Designed an adaptive weighting mechanism that uses the semantic-level understanding of foundational models to orchestrate navigation modules, dynamically balancing goal-directed search and environmental exploration.

  5. 05

    Semantic Frontier Evaluation

    Integrated CLIP-based features to score environmental frontiers, allowing the agent to hypothesize which unknown regions are most likely to contain the semantic target based on broad foundational priors.

  6. 06

    Cognitive Memory Rejection & Loop Detection

    Developed a repulsion mechanism that uses semantic feature similarity from foundational models to prevent redundant exploration and break cyclic behaviors in complex indoor topographies.

  7. 07

    Benchmark State-of-the-Art

    Demonstrated that the integration of foundational reasoning into the MU-Nav framework boosts Success Rate (SR) to 53.3% on standard industry benchmarks, significantly outperforming memory-less and non-foundational baselines.