Robot Action Caption Verification at 47K Judgments

Action captions on egocentric video — temporal descriptions of what the person is doing — drift. Captioning models produce plausible-sounding text that is subtly wrong: the right action at the wrong time, the wrong object, or a description of intent where nothing happened. A robotics research program needed every action caption in its corpus checked by a human against the actual footage before the captions could supervise training, at a scale — tens of thousands of clips — that internal review could not absorb. The judgments had to be defensible: not a vibe check, but a repeatable decision a reviewer makes the same way every time.

We ran verification in two passes and redesigned it between them. Pass one verified captions with a structured review flow; its throughput was acceptable but judgment quality was hard to keep consistent as the reviewer pool grew.

Before the second pass we rebuilt the flow around one binary decision per judgment — does the caption accurately describe the action shown, in the interval shown, yes or no — with full-clip viewing required before the decision rendered and the caption, its interval, and the footage side by side. Binary decisions calibrate far faster than multi-way rubrics, and disagreements surface immediately instead of hiding inside a rating scale. The second pass delivered 44,555 verifications in 12 days.

Because a failed verification means the caption was wrong, not that the clip is worthless, we also ran a 500-item correction pilot: captions rejected in the new flow were corrected against the same footage and re-verified, turning a rejection stream into a repair stream.

The program's caption corpus became trustworthy enough to supervise robot action learning directly, and the binary-judgment verification flow — full-clip viewing, one decision, immediate disagreement surfacing — is now our default for caption and transcript QA on egocentric video. The correction pilot also proved the economics: rejected captions can be repaired against the same footage for a fraction of fresh capture.