CLARU
Video

Video SFT Data Curation at Deadline Scale

286,712Video SFT labels curated and delivered
summary.md

Challenge:A frontier video-generation lab needed supervised-fine-tuning data built from raw video at a scale and pace its internal labeling could not reach: hundreds of thousands of clip-level curation judgments against a training taxonomy, with a hard external deadline.

Solution:We ran the curation as a managed pipeline rather than a crowd task.

Result:The lab hit its SFT deadline with labeled data that survived its own internal audits, and the engagement established the managed-queue pattern — rubric training, task-specialized queues, measured per-annotator agreement — that we now run for time-boxed curation at hundreds of annotators.

286,712Curated SFT labels delivered
276,883Labels delivered in the final 3 days
233Trained annotators on the sprint
3-passCalibration → bulk → adjudication pipeline
// THE CHALLENGE

A frontier video-generation lab needed supervised-fine-tuning data built from raw video at a scale and pace its internal labeling could not reach: hundreds of thousands of clip-level curation judgments against a training taxonomy, with a hard external deadline. The work was not generic tagging — each judgment had to follow the lab's curation rubric, where inconsistent judgment quality poisons an SFT run. Any outsourcing partner also had to scale to hundreds of trained annotators within days without quality diluting as volume ramped.

// OUR APPROACH

We ran the curation as a managed pipeline rather than a crowd task. Annotators were trained on the lab's rubric first, then worked task-specialized queues — each queue one judgment type, so annotators build calibration on a single decision instead of context-switching across many.

Three passes structured the work: a calibration pass against seeded examples with known-good answers, bulk labeling with per-annotator spot audits, and a reconciliation pass where disagreements were adjudicated rather than averaged away. Queue assignment was load-balanced by measured per-annotator agreement, so the queues that mattered most went to the annotators whose calibration held.

The final three days ran as a scheduled sprint: 233 annotators delivered 276,883 labels — more than 96% of the total — without a quality drop, because the calibration and audit machinery was already warm when volume ramped.

  1. 01CalibrateRubric training plus seeded calibration queues with known-good answers
  2. 02LabelTask-specialized bulk queues with per-annotator spot audits
  3. 03AdjudicateDisagreements reconciled by decision, not averaged away
  4. 04SprintDeadline window scaled to 233 annotators — 276,883 labels in 3 days

Three passes, then the sprint

Calibration
seeded gold set
Bulk + audits
bulk labeling · per-annotator spot audits
Adjudication
disagreements adjudicated
Sprint
ramp
final 3 days

Schematic of the order and overlap of the passes, not their exact durations. The final three-day sprint ran on calibration and audits that were already in place.

// FOOTAGE

Raw-video samples on Explore

Public internet-sourced clips from Claru Explore, shown muted. Curation work like this program's starts from raw segments of this kind and decides which belong in a training set.

Arranging Mahjong tiles on a table
A man outdoors against a lake and autumn trees
Browse internet-sourced videos on Explore
// QUALITY GATES
  1. Rubric training firstAnnotators trained on the lab's curation rubric before touching production queues.
  2. Calibration on seeded goldA calibration pass against seeded examples with known-good answers.
  3. Per-annotator spot auditsBulk labeling ran with spot audits per annotator throughout.
  4. Adjudicated disagreementsDisagreements were resolved by adjudication rather than averaged away.
// RESULTS
286,712Curated SFT labels delivered
276,883Labels delivered in the final 3 days
233Trained annotators on the sprint
3-passCalibration → bulk → adjudication pipeline

Where the labels landed

Curated SFT labels delivered
286,712
Labels in the final 3-day sprint
276,883

Bar length is proportional to labels. The final three days delivered more than 96% of the total, from 233 trained annotators.

// WHAT THE CUSTOMER RECEIVED

Clip-level curation judgments against the lab's supervised-fine-tuning taxonomy.

Curation labels
286,712 selection and scoring decisions on raw video segments, made under the lab's rubric.
Adjudicated outcomes
Where annotators disagreed, the delivered label is the adjudicated decision, not an average.
Deadline delivery
276,883 labels in the final three-day window from 233 trained annotators.
// IMPACT

The lab hit its SFT deadline with labeled data that survived its own internal audits, and the engagement established the managed-queue pattern — rubric training, task-specialized queues, measured per-annotator agreement — that we now run for time-boxed curation at hundreds of annotators.

// WHY IT MATTERS

Supervised fine-tuning is where data quality stops being a sampling concern and becomes a signal: the model learns the judgments, including the inconsistent ones. Curation that keeps rubric fidelity while scaling from a calibration set to 233 annotators in days is the difference between an SFT run that moves the model and one that bakes in noise. Production video pipelines filter most data automatically and reserve human curation for strict-quality uses such as SFT, as Reka's data team describes in its world model data pipeline write-up.

// DECISIONS WE MADE
Task-specialized queueswhyEach queue is one judgment type, so annotators calibrate on a single decision instead of context-switching.
Adjudicate, don't averagewhyAveraging hides rubric disagreements that would poison an SFT run.
Route work by measured agreementwhyThe queues that mattered most went to annotators whose calibration held.
Warm the machinery before the rampwhyCalibration and audits were already running when volume spiked, so quality held through the sprint.
// KEEP EXPLORING
// RELATED
// SOURCES
  1. [1]World Model Data Pipeline (Reka)Reka's data team on sourcing, analysis, filtering, transformation and annotation stages for world-model training data.
// FAQ

Ready to build your next dataset?

Tell us about your project and we will scope a plan within 48 hours.