2 Why the Harness Works

2Why the Harness Works

[12]: Berman et al. (2026), Claude Plays Robotics.

The paradigm of coding agents transfers, in principle, to the physical world. Each robot capability becomes a callable tool, and a language model orchestrates them through an agentic loop [12]. Our goal is to build the harness that makes this transfer work. Yet the harness touches only orchestration, not the components themselves. The same policy, called through the harness, produces the same distribution of outcomes on any single attempt. If no individual component becomes more likely to succeed, why should the system as a whole? We formulate this question and analyze what factors govern the gain and what they imply for design.

For a household robot, a long-horizon task may require navigating across rooms and interacting with multiple objects, leading to a large n.

Consider an n-step task in which step i succeeds with probability p_i. Under open-loop execution the task succeeds only if every step does:

P_{open} = i = 1 \prod n p_{i}

(1)

Each factor below one compounds the loss; the product decays geometrically with n.

The four outcomes per attempt: true success correctly passed,

p_{i} α

; true success misclassified and retried,

p_{i} (1 - α)

; true failure correctly detected and retried,

(1 - p_{i}) α

; true failure missed and passed through,

(1 - p_{i}) (1 - α)

Now suppose the harness wraps each step in a detect-and-retry loop, allowing up to k attempts per step. After each attempt, an evaluator classifies the outcome as success or failure with accuracy α: it returns the correct judgment with probability α and the wrong one with probability 1 − α. A retry is triggered whenever the evaluator reports failure, whether correctly or not, with probability $r_{i} = p_{i} (1 - α) + (1 - p_{i}) α$ . We seek $p_{i}^{*} (α, k)$ , the probability that step i produces a true success within k attempts. The step succeeds on attempt j (j = 1, …, k) if all preceding j−1 attempts triggered retries (probability $r_{i}^{j - 1}$ ) and the j-th attempt is a true success correctly passed (probability $p_{i} α$ ). These events are mutually exclusive, so:

p_{i}^{*} (α, k) = j = 1 \sum k r_{i}^{j - 1} \cdot p_{i} α = p_{i} α \cdot \frac{1 - r _{i}^{k}}{1 - r _{i}}

(2)

Eq. 2 assumes each retry is an independent draw with the same

p_{i}

; we discuss this assumption further below. Special cases: at

α = 1

r_{i} = 1 - p_{i}

and

p_{i}^{*} = 1 - (1 - p_{i})^{k}

(perfect evaluator, failure compressed exponentially in

k

); at

α = \frac{1}{2}

p_{i}^{*} = p_{i} (1 - 2^{- k}) \leq p_{i}

(random guessing never exceeds the single-attempt rate); for sufficiently large

k

p_{i}^{*}

approaches the ceiling

\frac{p _{i}}{( 2 p _{i} - 1 ) + ( 1 - p _{i} ) / α}

, monotonically increasing in

α

, reaching 1 at

α = 1

Substituting into the product over all steps, the task reliability under the harness becomes:

P_{harness} = i = 1 \prod n p_{i}^{*} (α, k) = i = 1 \prod n p_{i} α \cdot \frac{1 - r _{i}^{k}}{1 - r _{i}}

(3)

Each factor p_i in the open-loop product is now scaled by a per-step multiplier $α \cdot \frac{1 - r _{i}^{k}}{1 - r _{i}}$ ; even at moderate α, the gains are substantial, as illustrated in Figure 2.

Line chart: task success rate versus task length under open loop and under the harness at varying evaluation accuracy. — **Figure 2.** Task success rate as a function of task length `n` under open loop and under the harness at varying evaluation accuracy `α`. Per-step success probability `p`=0.85, with up to `k`=5 retry attempts per step.

The formula makes explicit what the harness should provide: reliable evaluation (α → 1). Each additional retry yields geometrically less improvement, and the limit is directly capped by α. In software, α ≈ 1 is taken for granted; in the physical world, it must be actively constructed and maximized, making it the primary design challenge. Also note that the formula rests on two simplifying assumptions, both conservative for a well-designed harness:

Retries within a step. Eq. 2 assumes each retry is an independent draw with the same p_i. Naively re-executing the same policy without adaptation may fail repeatedly, making retries worse than independent draws. A well-designed harness, however, diagnoses the cause of failure and adapts the next attempt, for example by adjusting the robot's position or selecting a more suitable grasp policy, making retries progressively more likely to succeed.
Fixed plan between steps. The formulation holds the plan fixed (same n-step sequence under both open loop and harness execution). In practice, a robot may encounter unexpected situations that a fixed plan cannot anticipate: a door that was open during planning is now closed, or an object has been moved. The harness reads the world state before each step and re-plans adaptively, correcting course and effectively raising p_i itself.

For a well-designed harness, both effects work in its favor; the formulation is therefore a lower bound on the true advantage. The design of the harness presented in the following sections is informed by these considerations.