Towards the Harness of Embodied Agents

BContext Structure

Section 3.2 introduces three context lifetimes. Listing 11 expands the context items defined in Table 1. The System Prompt appears in full; other entries show structure without task-specific values.

Resident
stable across turns

System Prompt

You are Thea, an embodied agent that completes physical tasks by calling tools.
  • Choose one next tool call per turn unless the task is complete.
  • After each tool result, update the plan from the latest evidence.
  • Use the Scene Graph Brief for object refs and coarse global state. Use the latest Observation for current local visibility, alignment, and clearance.
  • Treat the latest Observation as current physical evidence and earlier tool results as historical execution evidence.
  • Follow tool descriptions for preconditions, argument semantics, failure modes, and recovery hints.
  • Do not invent object identifiers, measurements, action outcomes, or user intent. Ask the user when the instruction and current evidence are genuinely insufficient.
  • Wait for the current tool to finish before choosing the next tool.

Memory

{Preferences: [entry, …], Conventions: [entry, …],
 General Lessons: [entry, …]}

Embodiment Profile

Operational Envelope: {Base Footprint; Base Mobility; Reachable Workspace}
Perception Configuration: {Sensor Modalities; Model-Visible Views}
Base-Relative Positions: {Camera Positions; Initial Gripper Positions}

Tool Definitions

[ {name, description [+ experience summary],
 inputSchema: {type, properties, [required]}}, … ]
Refreshed
replaced before each decision

Scene Graph Brief

Graph metadata: freshness=…; provenance=…; coordinate_frame=…; updated_at=…
Robot: pose=(…); holding=[obj, …]
Objects: - obj; position=(…); confidence=…; freshness=…; container_state=…; contents=[obj, …]

Observations

Base clearance: forward=… m, backward=… m, left=… m, right=… m
Visual evidence: view, … [image blocks follow]
Accumulated
grows through ordinary turns

Instructions

[ {content: instruction}, … ]

Task Notes

Task summary: Goal=instruction; Current phase=phase
Timeline: task_start; tool: outcome [reason]

Model Responses

[ {content,
 tool_calls[]: {id, type: function, function: {name, arguments}}
}, … ]

Tool Results

[ {tool_call_id,
 content: {success: true, tool-defined value fields} or {success: false, reason}
}, … ]

Compaction may replace an older accumulated prefix; resident and refreshed context remain unchanged.

Listing 11. Context structure for one decision, the concatenation of the three lifetimes of Table 1 in one model call.