HINT-VLM

Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model

1Georgia Institute of Technology 2VinMotion 3University of Southern California

CoRL 2026 Workshop on Spatial Intelligence and Structured Reasoning for Embodied Navigation (SISREN)

The full-length manuscript with full results is already available on arXiv; please refer to it. An extended version will be submitted to an archival proceedings.

Real world

Push the obstacle, or walk around it?

Outdoor garden

The VLM-GP gives the yellow metal chair p = 0.94, so the planner takes the interactive path through it instead of the detouring path. RViz replay of the run with the robot's front camera inset; the later part plays at 4×.

Indoor

The black cart gets p = 0.96; the robot pushes it aside and continues to the goal. Phone video (left) next to the synchronized RViz replay (right).

Method

From a camera image to a push.

HINT-VLM teaser: VLM-GP object movability prediction, terrain reconstruction and object mapping, uncertainty-aware global planning, and FSM-based control

HINT-VLM enables humanoid interactive navigation in cluttered rough terrain by combining robot-grounded Vision-Language Model reasoning, hidden-terrain reconstruction, and uncertainty-aware global planning.

TL;DR

HINT-VLM lets a humanoid decide, before touching anything, whether to push an obstacle out of the way or walk around it on rough terrain. A vision-language model, corrected with the robot's own pushing outcomes, predicts how likely each push is to succeed; the terrain hidden under the object is reconstructed; and an uncertainty-aware planner weighs the gain of a successful push against the cost of a failed one.

Abstract

Humanoid robots have achieved significant progress in locomotion and manipulation, enabling traversal over rough terrain and richer interaction with the environment, such as moving obstacles. However, most traditional navigation methods assume non-interactive environments and focus only on traversal. While Navigation Among Movable Obstacles (NAMO) approaches relax this assumption, prior efforts have largely focused on flat environments and reactive interaction, rather than predicting object movability and planning ahead. These limitations become critical in outdoor rough-terrain scenarios, where robots must jointly reason about terrain traversability and object movability. In this work, we address NAMO for humanoid robots in cluttered outdoor rough terrain by proposing a hybrid planning framework that combines semantic priors from a Vision-Language Model (VLM) with embodied robot interaction data through a Gaussian process (GP) residual model. The GP corrects the VLM-derived movability prior to produce embodiment-grounded probabilistic movability estimates, which are then incorporated into hierarchical uncertainty-aware planning. The proposed approach allows humanoids to jointly leverage locomotion and manipulation for efficient and safe navigation, representing the first solution to NAMO in rough terrain through predictive movability reasoning and probabilistic VLM integration.

  1. Object movability with VLM-GP

    A pretrained VLM names the candidate objects in the RGB image and gives a movability prior. A residual Gaussian process, learned from the robot's interaction outcomes, corrects it into the probability p that this robot can move the object.

  2. Object grounding and terrain reconstruction

    Grounding DINO and SAM2 mask the object, and its masked depth is lifted onto the bird's-eye-view map as the object footprint. LaMa inpaints the depth behind the mask, so the planner also sees the terrain the object is hiding.

  3. Uncertainty-aware planning

    Each push option is scored by its expected cost; detours by their path cost. All costs use terrain-aware A*, and the cheapest option is chosen.

    c = cstart→obj + p·cobj→goal + (1−p)·cfallback

  4. FSM and control

    When the chosen route goes through an object, a finite state machine switches from path tracking to pushing, tracks the object with VideoSAM, and resumes path tracking once there is enough clearance.

Ablation

Failure without Terrain Reconstruction

Without terrain reconstruction the planner cannot see what lies under the chair. Here it stands on stacked wooden boards: after pushing it away, the robot steps onto the boards, stumbles, and is caught.

Simulation

Three cluttered rough-terrain sites.

Garden

Movable objects block every route to the goal, so a planner that cannot interact does not finish.

Construction

Pushing through opens a shorter route than walking around.

Park

The terrain under each movable object decides whether pushing it pays off.

Citation

BibTeX

@inproceedings{yoon2026hintvlm,
  title     = {{HINT-VLM}: Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model},
  author    = {Yoon, Ziwon and Kim, Seok Hwan and Chen, Yiyu and Nguyen, Quan and Gan, Lu and Zhao, Ye},
  booktitle = {CoRL 2026 Workshop on Spatial Intelligence and Structured Reasoning for Embodied Navigation (SISREN)},
  year      = {2026},
  eprint    = {2611.11665},
  archivePrefix = {arXiv}
}