RLHF Preference Annotator

You will sit at the intersection of AI training and human judgment. Each task presents a prompt-response pair (or set of pairs) that you evaluate against a detailed rubric covering accuracy, helpfulness, safety, and style. Your evaluations feed directly into the reward model that steers how the LLM is fine-tuned. This means your judgment literally changes the behavior of the model millions of people interact with.

Strong analytical writing is the single most important skill. You need to be able to read a complex response, identify what is good and what is wrong with it, and explain your assessment in 2-3 sentences. Backgrounds in philosophy, journalism, law, science, or education tend to produce strong RLHF annotators because those fields train exactly this kind of evaluative thinking.

Annotators work in focused sessions of 3-6 hours at a time, scheduling their own shifts within project windows. Weekly volume targets are typically 20-30 hours but can scale up during surge periods. A weekly calibration meeting aligns the team on rubric updates and tricky edge cases.