Know Your Body

A Harness for Direct and Self-Improving Robot Control with VLMs

Zeyu Lou1,2 Yanhong Zeng2 Yong Wang2,3 Chenyang Si1,†

1Nanjing University 2Ant Group 3Zhejiang University

† Corresponding author

Teaser

Knowing the task is not knowing the body. A VLM can recognize what to write, but the robot receives an end-effector pose while writing acts at the held tip. KnowBody makes this command–effect relation explicit for choosing actions and interpreting outcomes.

Abstract

A general-purpose vision–language model can understand a task goal without knowing how a particular robot’s motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Under a fixed-initialization protocol with cross-episode updates disabled, the initialized KnowBody harness achieves 75% completion across four real-robot tasks and 32 trials, compared with 25% for the native harness.

Method Overview

KnowBody places a partial, queryable body model inside the interaction loop of a frozen VLM. The model still reasons about the task and selects actions; the harness supplies the action-relevant relations needed to turn those decisions into robot commands and to interpret their consequences.

KnowBody separates online state feedback from validated cross-episode revision. At each decision, the frozen VLM receives current task context and queries the partial body model. Execution updates the current state immediately; candidate body and rule revisions are validated before becoming available in later episodes.

Action-Relevant Body Relations

The body model exposes only the relations needed by the current decision: how motion appears in the wrist view, where a held tool acts relative to the commanded frame, and whether a target admits a continuous full-arm path.

Relations exposed by the body model. (a) Achieved motion is paired with wrist-image displacement. (b) Permanent geometry and the current attachment map the commanded frame to the acting tool tip. (c) Sequentially seeded waypoints test path feasibility beyond endpoint reachability.
(a) Motion response. Achieved motion is paired with wrist-image displacement.
(b) Functional geometry. Body and attachment geometry locate the acting tip.
(c) Whole-arm feasibility. Waypoints test a continuous path to the target.

Real-Robot Experiments

We evaluate duck placement, apple pushing, writing, and pouring under fixed reasoning budgets. The matched comparison uses the same frozen VLM and disables cross-episode updates.

75%KnowBody12 / 16 trials
compared with
25%Native harness4 / 16 trials
Task examples and two evaluation protocols. Left: four real-robot tasks. Upper right: reasoning rounds across successive successful completions with knowledge retained. Lower right: all 32 fixed-budget trials in the matched comparison; numbers show reasoning rounds and F denotes failure.
Task examples. Duck placement, apple pushing, writing, and pouring.
Evaluation results. Continual-interaction trends and all fixed-budget outcomes.

BibTeX

@article{lou2026knowbody,
  title   = {Know Your Body: A Harness for Direct and
             Self-Improving Robot Control with VLMs},
  author  = {Lou, Zeyu and Zeng, Yanhong and
             Wang, Yong and Si, Chenyang},
  year    = {2026},
  journal = {arXiv preprint},
  eprint  = {2609.28530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi     = {10.48550/arXiv.2609.28530}
}