LLM Agents for Robots: One Loop, Part by Part

Robot policies are usually discussed by architecture: an end-to-end model on one side, an agent built around a language model on the other. This post takes a different cut. It walks through one agent–environment loop part by part and asks, at each part, how end-to-end models and LLM-based agents differ, and where they can be combined.

The Agent–Environment Loop

Every robot policy can be read as the same loop: an agent receives a task, acts on its environment, and observes what happens. An LLM-based agent, which here covers both VLM agents that act through tools and coding agents that write programs, makes this loop explicit, reasoning about what it knows, acting, and reading the result before it reasons again, as in ReAct [1]. Figure 1 is the map for the rest of this post, from the loop’s timing to learning across episodes and how VLAs, WAMs, and LLM-based agents fill it in.

Figure The agent–environment loop.The task is set once per episode; the agent and the environment then interact step by step, only through actions and observations. Learning sits outside this loop and updates π and M from the experience of many episodes.

Task: Instruction and Prior Knowledge

Instruction

The most common instruction is a sentence of natural language. End-to-end policies such as RT-2 [5] take a short command like “put the cup in the basket”, and an LLM-based agent receives it as a text prompt. Language is easy to give, but often ambiguous: which cup, and placed where?

Other formats trade that ease for precision (Table 1). A goal image shows the finished scene [6], a sketched trajectory shows the motion [7], and a video shows a person doing the task once [8]. Multimodal prompts mix text with images [9], visual marks point at where to act [10], and formal specifications such as PDDL goals make the task checkable by a planner [11]. The more precise the format, the more effort it takes to provide, and the more irrelevant detail it fixes.

Format	Example	Specifies	Representative work
Language	“Put the cup in the basket”	Goal	RT-2 [5]
Goal image	A photo of the finished scene	Goal	Play-LMP [6]
Sketch or trajectory	A path drawn over the camera image	Goal or motion	RT-Trajectory [7]
Video demonstration	A person doing the task once	Process	Vid2Robot [8]
Multimodal prompt	“Put the [image] into the [image]”	Goal	VIMA [9]
Visual marks	Points or arrows drawn on the image	Where to act	PIVOT [10]
Formal specification	A PDDL goal or temporal-logic formula	Goal and constraints	LLM+P [11]

Table Ways to give a robot its task.Formats range from abstract to specific: language is easy to give but ambiguous, while images and videos are precise but also fix details that do not matter.

The bigger difference is who reads the instruction. An end-to-end policy accepts only the formats it was trained on; adding sketches or videos means new data and retraining. An LLM-based agent can take anything it can read in context, such as text, images, rules, or a demonstration, and can ask a clarifying question when the instruction is ambiguous.

Prior Knowledge

Prior knowledge K, such as what skills do, how to call them, rules, and examples, lives in a VLA’s weights but in an LLM-based agent’s context. There, examples become a way to learn without changing any weights, which Table 2 groups into four families.

Family	What the context steers	Side	Example
Context-conditioned policies	The policy’s input	End-to-end	VIMA [9]
Geometric demonstration transfer	How a demo maps onto new objects	Geometric	Neural Descriptor Fields [13]
World-model-based control	Predicted futures	End-to-end (WAM)	Zero-WAM [14]
Skill- and agent-based execution	Which skills the agent runs	LLM-based agent	GPT-Policy [15], RoboICL [16]

Table Four families of in-context learning [12].This post focuses on the last row, where the context steers what an agent decides to do.

Interface: Observation and Action Spaces

Observation Space

What the agent may see sets how hard the same task is (Table 3), and even a “success” from a tool can mean very different things (Table 4). Perception can also be an action: an agent can call a tool such as get_position(cup) and read the answer as its next observation.

Level	Example	VLA	WAM	LLM-based agent
Images	RGB from one or more cameras	Yes	Yes	Yes
Robot state	Joint angles, end-effector pose, depth	Usually	Usually	As text, through tools
Geometry and calibration	Camera intrinsics, robot model, object meshes	Rarely	Rarely	Through tools
Privileged state	Ground-truth object poses from the simulator	In simulation	In simulation	In simulation
Feedback	Tool return values, success signals	No	Predicted, not observed	Yes

Table What the agent is allowed to see.Simulation can expose privileged state that a real robot never has; Embodied First Exam evaluates agents with and without it [17].

Event	Example
Request accepted	The tool call did not error
Motion solved	Inverse kinematics found a joint solution
Target reached	Position error within tolerance
Grasp holds	The object is actually in the gripper
Task done	An independent checker passes

Table What a "success" can mean.A tool's return value is an observation, but it may report any of these events; Guava, for example, separates a closed gripper from a grasped object [18].

Action Space

The action space is what the VLM literally emits across the interface. It takes three common forms: numbers sent directly to the robot, a tool call with numeric arguments, or a program that sends many of them (Figure 2). Inside each form the numbers carry choices of their own (Table 5), and actions that look the same can still mean different things (Table 6).

Direct numbers

[0.42, 0.10, 0.31, 1, 0, 0, 0, 0]
[0.43, 0.10, 0.30, 1, 0, 0, 0, 0]
[0.44, 0.10, 0.28, 1, 0, 0, 0, 1]

Each row is one control tick: absolute end-effector position, orientation, and gripper.

Astranomy, direct [19]

Tool call

execute_eef_target(
  target=[0.44, 0.10, 0.28, 1, 0, 0, 0],
  gripper="close",
  steps=5)
→ measured pose, gripper state

One target, held for a few steps; the tool reports what actually happened.

Astranomy, agentic [19]; RobotWorld [20]; Guava [18]

Program

for p in waypoints:
    q = solve_ik(p, quat)
    move_to_joints(q)
close_gripper()
if not grasped():
    retry()

Code that computes and sends many targets, checks, and retries on its own.

Code as Policies [21]; CaP-X [22]

Figure Three ways a VLM emits actions.The same motion in each form. Schematic examples; argument names are simplified.

Choice	Options	Example
Space	Joint angles, end-effector pose, or spline coefficients	Astranomy compares all three [19]
Reference	Absolute target, offset from the last target, or offset from the measured state	Astranomy’s absolute, delta, and relative actions [19]
Timing	One row per control tick, one target held for N steps, or a chunk	Direct vs. agentic in Astranomy [19]; `steps` in RobotWorld [20]
Control point	Flange, fingertip, or gripper center	Same simulator, flange in Astranomy [19] and fingertip in Embodied First Exam [17]

Table What the numbers mean.In Astranomy, absolute end-effector targets worked better than joint angles or splines when the model emitted them directly; with a harness that sends one target at a time and returns feedback, the gap largely closed [19].

Looks the same	Actually differs
A grasp action	`GRASP` closes the gripper [23]; `grasp(name)` also locates and approaches the object [18]
`move_eef`	An absolute flange target, a repeated increment, or a planned move to the grasp point, by backend [20]
An 8-D end-effector vector	Flange or fingertip; WXYZ or XYZW quaternion; gripper 0 means open or closed [19]
“Discrete” actions	Semantic command words [23] or bins over each number [5]

Table Same name, different action.An action space is defined only together with its contract: frame, control point, absolute or relative targets, and timing. The same simulator can expose different contracts in different frameworks [17, 19, 20].

Agent: Policy and Memory

Policy

The same VLM can be wired into a policy in different ways. Astranomy compares three on the same robot tasks [19]: the model can act directly, emitting a stretch of actions and then looking again; act as an agent, sending one target at a time through a tool and reading the feedback; or supervise another policy, accepting or editing the actions it proposes (Figure 3). In that evaluation the agentic setup did best, and supervising another policy did not clearly beat acting directly. How the model is wired into the loop matters as much as the model itself.

Figure Three ways to wire a VLM into the loop.Direct: the model emits a chunk of actions, then looks again. Agentic: it sends one target at a time through a tool and reads the feedback. Orchestrated: another policy proposes actions and the model checks them. Modes from Astranomy [19].

Once the model is in the loop, how much each decision delegates also varies widely: the same task can be decided step by step or handed to a whole skill (Figure 4, Table 7).

Figure Who decides what.The same task, putting the cup in the basket, at four levels. Going down, the model's output gets more abstract and the executor fills in more; low-level control turns the result into motor commands in every case.

Level	The model decides	Example output	Filled in by	Example work
Local motion	How to move next	“move left”, a pose change	Interpreter and controller	Show-Harness [23], RT-2 [5]
Spatial intent	Where to touch, which way to go	Points, keyposes, 2D paths	Depth, planner, or a low-level policy	MOKA [24], HAMSTER [25]
Geometric constraints	Which relations must hold	Keypoint constraints, 3D value maps	Optimizer or MPC	ReKep [26], VoxPoser [27]
Tool choice	Which capability to call	`grasp(cup)`, `vla_act(...)`	Skill library or a frozen VLA	Guava [18], Harness VLA [28]

Table Four levels of what the model decides.Code, JSON, and action tokens are output forms that cut across these levels: a program can call a whole grasp skill or compute every end-effector pose [21, 22].

Memory

In an end-to-end policy, memory is implicit: whatever the network keeps in its weights and in its short window of recent frames. An agent system moves memory out of the model and into the harness, where explicit rules write, update, and delete what it remembers, and decide what the model reads (Figure 5, Table 8). Coding agents already work this way: Claude Code loads CLAUDE.md files into every session and compacts the conversation into a summary when the context fills up [29], and Codex reads AGENTS.md instruction files from the project before every task [30].

Figure Memory inside the model, or in the harness.Left: an end-to-end policy carries state implicitly in its weights and recent frames. Right: the harness sits between the model and the robot. It writes observations, tool receipts, and every executed action into memory M, updates or deletes entries by rule, and builds the context the model reads.

Operation	Implicit (end-to-end)	Explicit (harness)	Example
Write	New frames enter a fixed window; lasting change only through training	Append each observation, tool receipt, and executed action as a record	Observation, action, receipt records in RoboICL [16]
Update	Hidden state overwritten every step; weights change only by retraining	Revise an entry when the world or the rules change, such as a belief, a summary, or a written rule	Editing CLAUDE.md or AGENTS.md as rules change [29, 30]
Delete	Frames fall out of the window	Drop or compress by rule, keeping what matters for the task	GPT-Policy’s context compiler [15]; compaction in Claude Code [29]

Table Write, update, delete.The three memory operations a harness makes explicit; in an end-to-end policy the same choices are fixed by the architecture and training. Reading builds the model's context from what remains (Figure 5).

Time: Steps, Chunks, and Asynchrony

Every scheme faces the same tension: waiting for the model stalls the robot, but acting without waiting follows a stale plan. Figure 6 compares five ways of handling it, and Table 9 lists them with representative work. Whether the world pauses while the model responds changes the picture: RobotWorld pauses the simulator while the model responds [20], while Embodied First Exam can also keep the world running on a wall clock [17].

World

Idle time is free: the world waits for the model.

Figure Who waits for whom.Five schemes over the same stretch of time. Yellow: the model responding, from observation to output action; teal: the robot acting, drawn as dense ticks for fast control; shaded: the robot idle. Switch the world between paused and running to see when idle time costs.

Scheme	While the model responds	Example
Stop and think	The robot waits; thinking and acting alternate	Diffusion Policy [31], ReAct [1]
Smoothed chunks	Still waits, but overlapping chunk predictions are blended at the boundaries	ACT [32]
Next chunk while acting	The robot keeps executing; the new chunk is spliced in when it arrives	RTC [33], SmolVLA [34]
Slow brain, fast body	A slow reasoner updates a plan that a fast controller always follows	Helix [35], Hi Robot [36]
Write a program, then run	The model writes code once; the code runs the fast loop	Code as Policies [21]
Monitor and interrupt	Skills run in the background; events interrupt and replace them	TypeGo [37], asynchronous tool use [38]

Table How responding and acting share time.End-to-end models and agent systems resolve the same tension; the last two rows are agent-side schemes.

Learning: Meta-Learning, Continual Learning, and RSI

Learning sits outside the loop: experience from many episodes updates the policy π and memory M, and the improved agent goes back into new episodes (Figure 7). Meta-learning, continual learning, and recursive self-improvement (RSI) are names for this one process from different communities [39]; what separates them is the question asked of it (Table 10). In-context learning changes only memory, so it belongs to the agent’s memory rather than here.

Figure One learning process.Experience from many episodes updates π and M, and the better agent runs new episodes. The three questions below are three ways to check that this loop actually improves the agent.

Question	Evidence that answers it	Usually called	Example work
Does it adapt faster on a new task?	Fewer trials to succeed on unseen tasks than starting fresh	Meta-learning	MAML [40], RL² [41], ORBIT [42]
Does it keep old skills?	Old tasks still pass after new ones are learned	Continual learning	LIBERO [43], π*0.6 [44], Simple Recipe Works [45]
Does improvement compound?	A newer update rule beats the older one from the same start, on unseen tasks	Recursive self-improvement	EmbodiRSI [46], Hyperagents [47], PhysRSI-eval [48]

Table Three questions to ask of any learning claim.One learning process, three checks. A robot that eventually succeeds after many attempts passes none of them by itself.

VLA vs. WAM vs. LLM-Based Agent

VLAs, WAMs, and LLM-based agents are not three rungs of one ladder: a VLA decides how to move, a WAM also predicts what will happen, and an LLM-based agent decides what to do next, often by calling the other two as skills [49, 50].

	VLA	WAM	LLM-based agent
Role	How to move next	What happens if I act	What to do next
Input	Images, instruction, robot state	Images, instruction	Images, instruction, memory, tool results
Output	Action chunks	Future frames + actions	Tool calls, programs
Knowledge	VLM backbone + robot data	Video + action data	Agentic + coding post-training
Memory / feedback	Short window, no check	Predicts, no check	History, checks, replans
Strength	Dexterous, low latency	Dynamics, new scenes	Long tasks, composing skills
Limit	Scarce robot data	Compute, prediction error	Skill library, latency

Table VLA, WAM, and LLM-based agents.Typical roles, not a ranking: the three sit at different layers, and an agent can call a VLA as one of its skills.

Open Questions

How do we reduce the latency of agent systems? Each reasoning step is far slower than a control loop, so the agent has to decide how long to commit and whether to keep thinking while it acts.
How do agent systems handle fine-grained manipulation? An agent is only as dexterous as the skills it can call; contact-rich tasks may need VLA or WAM policies as skills.
What is the right harness interface for LLM-based agents? What the agent observes, how skills are described, and what each call returns together define the interface it reasons over.
Could we build MCP servers for robotics? The Model Context Protocol standardizes how LLM-based agents connect to tools and data; robot skills and sensors could be exposed the same way.
How do we define physical intelligence? The term is used everywhere but rarely pinned down: is it dexterity, an understanding of physics, or the whole loop working well under real-world uncertainty?

References

S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations (ICLR), 2023.
Physical Intelligence. π0.5: a Vision-Language-Action Model with Open-World Generalization. Proceedings of the Conference on Robot Learning (CoRL), PMLR 305, 2025.
Ai2. MolmoAct2: Action Reasoning Models for Real-world Deployment. 2026.
S. Ye et al. World Action Models are Zero-shot Policies. arXiv:2602.15922, 2026.
A. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Conference on Robot Learning (CoRL), 2023.
C. Lynch et al. Learning Latent Plans from Play. Conference on Robot Learning (CoRL), 2019.
J. Gu et al. RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches. International Conference on Learning Representations (ICLR), 2024.
V. Jain et al. Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers. Robotics: Science and Systems (RSS), 2024.
Y. Jiang et al. VIMA: General Robot Manipulation with Multimodal Prompts. International Conference on Machine Learning (ICML), 2023.
S. Nasiriany et al. PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs. International Conference on Machine Learning (ICML), 2024.
B. Liu et al. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv:2304.11477, 2023.
H. Huang et al. In-Context Learning for Robots: Methods and Applications. arXiv:2609.36012, 2026.
A. Simeonov et al. Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. IEEE International Conference on Robotics and Automation (ICRA), 2022.
J. Zhou et al. Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization. arXiv:2608.26103, 2026.
D. Cheng et al. In-Context Robot Learning with VLM Agents. arXiv:2609.19138, 2026.
F. Liu et al. RoboICL: Embodied In-Context Learning with GPT-6 Astra. arXiv:2609.34261, 2026.
K. Wang et al. Embodied First Exam. Benchmark website and code, 2026.
H. Liu et al. Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness. arXiv:2606.18363, 2026.
J. X. Yang et al. Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy. NVIDIA technical report, 2026.
Z. Yang et al. RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments. arXiv:2610.10409, 2026.
J. Liang et al. Code as Policies: Language Model Programs for Embodied Control. IEEE International Conference on Robotics and Automation (ICRA), 2023.
L. Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. International Conference on Machine Learning (ICML), 2026.
Y. Chen et al. Show-Harness: Just a VLM Agent Can Play Robots. arXiv:2609.10522, 2026.
K. Fang et al. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting. Robotics: Science and Systems (RSS), 2024.
Y. Li et al. HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation. International Conference on Learning Representations (ICLR), 2025.
W. Huang et al. ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. Conference on Robot Learning (CoRL), 2024.
W. Huang et al. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. Conference on Robot Learning (CoRL), 2023.
Y. Zhang et al. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents. arXiv:2607.08448, 2026.
Anthropic. How Claude remembers your project. Claude Code documentation, 2026.
OpenAI. Custom instructions with AGENTS.md. Codex documentation, 2026.
C. Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Robotics: Science and Systems (RSS), 2023.
T. Z. Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Robotics: Science and Systems (RSS), 2023.
K. Black et al. Real-Time Execution of Action Chunking Flow Policies. Neural Information Processing Systems (NeurIPS), 2025.
M. Shukor et al. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics. arXiv:2506.01844, 2025.
Figure AI. Helix: A Vision-Language-Action Model for Generalist Humanoid Control. Blog post, 2025.
L. X. Shi et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. International Conference on Machine Learning (ICML), 2025.
Y. Chen et al. TypeGo: An OS Runtime for Embodied Agents. arXiv:2607.05482, 2026.
A. Ginart et al. Asynchronous Tool Usage for Real-Time Agents. arXiv:2410.21620, 2024.
H. Zhou et al. Who Improves the Robot? Toward Physical Recursive Self-Improvement. arXiv:2610.18842, 2026.
C. Finn et al. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. International Conference on Machine Learning (ICML), 2017.
Y. Duan et al. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv:1611.02779, 2016.
X. Lin et al. Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL. arXiv:2602.04089, 2026.
B. Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
Physical Intelligence. π*0.6: a VLA That Learns From Experience. arXiv:2511.14759, 2025.
J. Hu et al. Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning. Reinforcement Learning Conference (RLC), 2026.
H. Lang et al. EmbodiRSI: Recursive Self-Improvement for Data-Efficient Robot Adaptation. arXiv:2609.38905, 2026.
J. Zhang et al. Hyperagents. arXiv:2603.19461, 2026.
Z. Li et al. Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy. arXiv:2610.26848, 2026.
S. Salimpour et al. Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction. arXiv:2508.05294, 2025.
R. Shao et al. Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey. arXiv:2508.13073, 2025.