PhyHarness A Dev Harness for Physical AI

A practical simulation workflow for building and debugging robot manipulation policies with augmented rollout videos, joint logs, VLM transcripts, and task analyzers.

Policy iteration loop Physics-accurate simulation Dynamic feedback signal engineering IK and Vision services

Development loop

High-level workflow

Inputs

Define intent in a task YAML spec.
<task>.yaml under
policies/impl/<task>/

Design & implement

Coding model implements policy, analyzers, and unit tests for the task.

Simulate

Produces rollout.mp4, joints.csv, metrics, VLM transcript, and custom analyzer outputs.

Redesign

Inspect video frames, logs, and transcripts. Run ad hoc test and simulation. Design fixes.

Loop back to design & implement

Agent Foundations

Why this workflow scales

Solid abstraction layer

IK and Vision services hide low-level math, geometry, and camera plumbing
The development agent can operate at policy/task intent level instead of raw simulator internals
Shared services keep implementations consistent across tasks
Policy files stay focused on behavior design rather than infrastructure code

Rich feedback signal

rollout.mp4 includes 4-view video with kinematic overlays
joints.csv captures target vs applied controls at each step
rollout.vlm_transcript.json adds VLM-generated frame-level descriptions
Custom analyzers are dynamically coded for each task and each iteration, converting trajectories into structured diagnostics
Together, these artifacts make failure modes explicit and debuggable

Hypothesis-driven redesign

During Redesign, the development agent can run multiple ad hoc tests per hypothesis
Tests validate root-cause assumptions before policy changes are finalized
Fast test loops reduce guesswork and prevent overfitting to one rollout
Focused regression checks help preserve fixes across iterations

Example rollouts by use case

Policy in action

Shoulder Rotation

Primary goal: rotate around the shoulder-lift hinge while keeping a controlled TCP arc and stable gripper state using explicit arc targets + IK from policies/impl/shoulder_rotation/shoulder_rotation.yaml.

Iteration 1: shoulder rotation rollout

shoulder_rotation_rollout_001.mp4

1 / 1

Cube Tracing

Primary goal: trace the 12 edges of a virtual cube in the air with the gripper tip. Uses a two-phase precomputed IK trajectory (approach + connected 16-segment cube-edge trace) from policies/impl/path_tracing/path_tracing.yaml.

Iteration 1: cube tracing with shaking issue

cube_tracing_rollout_001.mp4

Iteration 2: smooth cube tracing motion

cube_tracing_rollout_002.mp4

1 / 2

Box Pick

Primary goal: pick the orange box from above and place it at a different table location. A vision-guided policy uses topdown color detection and an 8-phase IK plan (crane → approach → descend → grip → lift → transport → lower → release) from policies/impl/box_pick/box_pick.yaml.

Iteration 1: generated policy with control/physics issue

box_pick_rollout_001.mp4

Iteration 2: updated policy after investigation and fix

box_pick_rollout_002.mp4

1 / 2

Inspection

Primary goal: hover the gripper tip over detected top-edge segments of a fixed mounting bracket and execute a smooth inspection scan trajectory from policies/impl/inspection/inspection.yaml.

Iteration 1: workpiece is not fixed properly

inspection_rollout_001.mp4

Iteration 2: robot blocks itself into a folded configuration

inspection_rollout_002.mp4

Iteration 3: incorrect or blocked movement

inspection_rollout_003.mp4

Iteration 4: zigzag movement on the top contour

inspection_rollout_004.mp4

Iteration 5: smooth edge scanning trajectory

inspection_rollout_005.mp4

1 / 5

Cable Routing

Primary goal: precisely grasp the cable tip and route it through sequential peg gates using high-resolution top-camera perception, staged IK motion, and iterative policy validation from policies/impl/cable_routing/cable_routing.yaml.

Iteration 1: initial setup