AI Response Ranking Specialist

You will sit at the intersection of AI training and human judgment. Each task presents a prompt-response pair (or set of pairs) that you evaluate against a detailed rubric covering accuracy, helpfulness, safety, and style. Your evaluations feed directly into the reward model that steers how the LLM is fine-tuned. This means your judgment literally changes the behavior of the model millions of people interact with.

Ideal candidates have a graduate-level education or equivalent professional experience in a field that requires careful argumentation and evidence evaluation. We have had especially strong results from people with backgrounds in academic research, technical writing, legal analysis, and scientific peer review. Familiarity with LLM failure modes -- hallucination, sycophancy, refusal errors -- is valuable.

Annotators work in focused sessions of 3-6 hours at a time, scheduling their own shifts within project windows. Weekly volume targets are typically 20-30 hours but can scale up during surge periods. A weekly calibration meeting aligns the team on rubric updates and tricky edge cases.