Foresight Iterative Reasoning About Clues that Matter for Navigation

Arthur Zhang1, Carl Qi1, Donne Su1, Xiangyun Meng2, Amy Zhang1, Joydeep Biswas1

1The University of Texas at Austin•2Field AI

Logos of the Autonomous Mobile Robotics Laboratory, the MIDI Lab, Field AI, and The University of Texas at Austin
Foresight iteratively proposes and critiques image-space motion plans with a finetuned vision-language model, refining navigation behavior at test time to follow sparse language instructions in open-world environments.

Abstract

Discovering which cues matter, then planning using them.

Open-world mapless navigation from sparse language instructions requires inferring which environmental cues matter — the ramp, sign, or detour that reveals where to go. Prior works draw cues from closed-set factor categories or identify them before motion planning, missing plan-dependent cues. Pretrained Vision-Language Models (VLMs) can discover such cues, but require adaptation to act on them.

We present Foresight, a test-time framework in which a finetuned VLM alternates between proposing image-space motion plans and critiquing them against the language goal and visual context, conditioning each plan on prior critiques to refine its motion before execution. A reward model learned from human feedback post-trains the VLM with reinforcement learning inside this loop, aligning it with open-set behavior preferences. Across offline evaluations and 6 real-world environments, Foresight improves average task success by 37% and reduces interventions per mission by 52% relative to state-of-the-art test-time reasoning and foundation-model baselines, while running in real-time on a Jetson AGX Orin.

Approach

Test-time reasoning via iterative plan-critique refinement.

Foresight formulates motion planning as a test-time reasoning loop. Conditioned on the image observation and language instruction, a finetuned VLM proposes an image-space motion plan and then critiques it with respect to the language goal and visual context. Subsequent plans are conditioned on prior critiques, enabling the model to surface plan-dependent cues — ramps, signs, detours — and iteratively refine its motion before execution. A lightweight policy converts the refined image-space plan into executable trajectories.

Foresight architecture overview
Figure 1. The Foresight loop. A single finetuned VLM alternates between proposing an image-space motion plan and critiquing it against the instruction and visual context, with each new plan conditioned on the critiques that preceded it.

Training

Supervised finetuning, then reinforcement learning with human preferences.

To adapt pretrained VLMs for iterative refinement, we propose a simple two-stage training recipe. We first apply supervised finetuning to teach the VLM the structure of the plan-critique loop. We then learn a reward model from human preference annotations to judge motion plan quality, and use it to post-train the VLM with reinforcement learning, aligning critiques and refinements with open-set behavior preferences that are difficult to capture with hand-designed objectives alone.

Figure 2. The two-stage training recipe, from supervised finetuning of the plan-critique structure through reward-model-driven reinforcement learning.

Domains

Training, evaluation, and deployment contain distinct clues and locations.

Training data, the offline benchmark, and the real-robot deployments are drawn from distinct geographic regions, so no test evaluation site appears in training. Instructions are collected the way an operator would phrase them: “go to the nearest entrance”, “use the wheelchair friendly path”, which leaves the relevant cue unstated and forces the model to determine what to look for.

Satellite map with disjoint training, offline evaluation, and online evaluation regions, alongside example training instructions and held-out test episodes with plan critiques.
Figure 3. Left: training, offline evaluation, and online evaluation routes occupy disjoint parts of the map; the six deployment sites are labelled. Right: training episodes pair an observation with a sparse instruction, while held-out test episodes additionally show the critique that redirects the initial plan toward the correct cue.

Real-robot deployment

Closed-loop missions across six unseen environments.

We deploy Foresight on a real robot and follow sparse language instructions in six environments held out from training, grouped into three families: detour rerouting, structural clues, and sign understanding. Each site is run against the same baselines, and an operator intervenes whenever the robot commits to an unrecoverable plan.

Six real-world deployment scenes grouped into detour rerouting, structural clues, and sign understanding, each with the executed route overlaid.
Figure 4. The six deployment sites with executed routes overlaid. Detour rerouting (blue) blocks the direct path; structural clues (red) hide the route in ramps, stairs, and handrails; sign understanding (purple) requires reading posted text to pick a direction.
37%
higher average task success than state-of-the-art baselines.
53%
fewer interventions per waypoint than the strongest baseline.
6
unseen environments, spanning three families of navigation cue.
Left: average interventions per waypoint by cue category. Right: real-world success rate against intervention budget per waypoint.
Figure 5. Left: average interventions per waypoint, overall and split by cue family. Right: success rate as a function of the intervention budget an operator is willing to spend per waypoint.

Both post-training steps matter on hardware, but refinement matters more. Foresight, no refine lands alongside the LeLaN and Alpamayo baselines, showing that a finetuned planner that never critiques itself behaves much like a feed-forward one. While Foresight, no RL outperforms all other non-refinement baselines. The margin is widest on structural clues, where the relevant evidence only becomes salient once the critic identifies the clue.

The success-rate curve provides a more deployment-centric view. With no interventions allowed, Foresight already completes more than half of the missions while every external baseline finishes under a fifth of them. Granting a budget of a single intervention per waypoint pushes Foresight close to saturation without closing the gap for the baselines, which still need multiple attempts to generate a valid plan.

Long-horizon navigation mission

Head-to-head comparisons

Each clip runs LeLaN, Alpamayo, and Foresight on the same instruction and environment.

Detour rerouting tasks require inferring instruction-aligned alternative routes when the current path is blocked.

Offline benchmark

Where the gains come from, compared against baselines and ablations.

Offline, we score predicted image-space plans against human-demonstrated ones with the Hausdorff distance at 4 unique urban environments, which lets us isolate each design decision for larger models infeasible to run on the robot. We sweep the number of refinement iterations and compare against both general-purpose VLMs and robot foundation models, then ablate the reward design and the training configuration of the planner and the critic.

Three boxplot panels of Hausdorff distance against refinement iteration: baseline comparisons, reward design ablations, and motion planner and critic ablations.
Figure 6. Hausdorff distance to the reference plan against refinement iteration; lower is better. Left: comparison against general-purpose and robot foundation model baselines. Middle: reward design ablation. Right: which of the planner and the critic must be adapted.

Refinement only pays off once the model is trained to critique.

In the baseline panel, the general-purpose VLMs sit flat across refinement iterations as extra reasoning steps do not help a model that cannot tell a good plan from a bad one. Alpamayo, being feed-forward, has no refinement to give past the initial reasoning step. Foresight acheives the lowest-error method by improving from iteration 0 to 1, with most of the benefit realized by the first critique. Its error distribution is also visibly tighter, which is what thins out the tail of bad plans that force interventions on hardware.

Both reward terms and modules, are necessary.

Neither reward term alone reproduces the full model: training against either in isolation leaves a clearly higher median at both iteration counts, and only their combination shifts the whole distribution down. The planner/critic ablation makes a related point, but asymmetrically. A zero-shot planner (ZS/SFT) is catastrophic, with a median error several times that of any adapted variant — a good critic cannot rescue plans that were never plausible. A zero-shot critic (SFT/ZS) is far less damaging and lands close to finetuning both (SFT/SFT), which is why reinforcement learning on top of SFT/SFT, rather than supervised finetuning alone, is what produces the final margin.

Limitations

What iterative refinement does not solve yet.

While Foresight makes significant strides toward scalable mapless navigation, several challenges remain:

Citation

If Foresight is useful in your work, please cite us.

@article{zhang2026foresight,
  title   = {Foresight: Iterative Reasoning About Clues that Matter for Navigation},
  author  = {Zhang, Arthur and Qi, Carl and Su, Donne and Meng, Xiangyun and Zhang, Amy and Biswas, Joydeep},
  journal = {arXiv preprint arXiv:2606.12550},
  year    = {2026},
}