RLHF Data Annotation: A Playbook for Preference & Eval

RLHF data annotation is not one job—it’s four, and treating them as interchangeable is the fastest way to poison the training signal you paid for. The label work behind an aligned model comes in distinct shapes: supervised demonstrations, pairwise preferences, rubric-based evaluations, and red-teaming attacks. Each needs a different interface, a different kind of rater, and a different definition of “good.” This playbook is about the methods, not the vendors: how to decide what to collect, how to write a rubric raters actually agree on, and how much agreement to expect when the judgment is subjective.

What is RLHF data annotation?

RLHF data annotation is the human-labeling work that produces the data used to align a large language model with human intent. Reinforcement learning from human feedback runs in stages: collect demonstrations to fine-tune a base model, collect human comparisons of its outputs, train a reward model to predict those comparisons, then optimize the policy with reinforcement learning (Ouyang et al., 2022; Bai et al., 2022). The annotation feeds every stage, and each stage asks the rater for something different.

The shape of the label is the thing that matters. A demonstration is content the rater writes; a preference is a relative judgment the rater picks; a rubric score is an absolute rating against fixed criteria; a red-team log is an adversarial attempt plus a categorized failure. Confuse these and you collect data that looks reasonable in a spreadsheet but trains or measures the wrong thing.

If your question is which platform runs each task, that’s a separate decision covered in our guide to RLHF annotation tools. Here we stay on method, because the tool you need falls out of the task once you’ve named it correctly.

The four annotation tasks behind an aligned model

There are four jobs human annotators do for a modern LLM, and they pull in different directions. Getting the taxonomy right is most of the work, because the interface, the rater, and the quality metric all follow from which task you’re funding.

Task	What the rater produces	Primary use	Quality metric that matters
SFT demonstrations	A written ideal response to a prompt	Supervised fine-tuning of the base model	Instruction adherence, factuality, style match
Pairwise preference	A choice: output A is better than B	Training the reward model (the RLHF signal)	Chance-corrected rater agreement; transitivity
Rubric evaluation	An absolute score per criterion on a scale	Measuring a model over time; benchmarking	Ordinal agreement (weighted kappa / alpha); calibration
Red-teaming	An attack prompt plus a categorized failure	Finding unsafe or policy-violating behavior	Coverage of the attack taxonomy; reproducibility

Two things stand out. Preference and rubric work look similar but do opposite jobs: preferences change the model through the reward signal, while rubric scores measure how good it already is. And the quality metric shifts from task to task, which is why a single “accuracy” number is rarely the right thing to report.

Pairwise preference or Likert rubric: which to collect?

Collect pairwise preferences when you’re training a reward model, and Likert or anchored-rubric scores when you’re measuring one. That single distinction resolves most of the confusion, because relative and absolute judgments have different strengths and different failure modes.

Pairwise preference is the native RLHF signal. Both InstructGPT and the summarization work that preceded it trained the reward model on human comparisons, not on absolute scores, because humans are more consistent when they judge “A is better than B” than when they assign an absolute 6-out-of-10 (Stiennon et al., 2020; Ouyang et al., 2022). Comparisons also surface small quality differences that a coarse rating scale flattens. The cost is that a preference tells you nothing about absolute quality—two bad outputs still produce a winner—and preferences can be intransitive, so your sampling of pairs shapes the result.

Rubric scoring produces an absolute, multi-dimensional measurement: accuracy, faithfulness, safety, and tone, each on a defined scale. That’s what you want for tracking a model across releases, for a benchmark others can reproduce, and for any evaluation a safety reviewer or regulator might audit. The cost is calibration—absolute anchors drift between raters, and scale-use habits (central tendency, leniency) creep in unless the rubric is genuinely anchored.

Question	Lean pairwise preference	Lean Likert / rubric
What are you building?	A reward model to train the policy	A measurement of model quality over time
How close are the outputs?	Very close—hard to score absolutely	Distinct enough to place on anchors
How many dimensions?	One overall 'which is better'	Several (accuracy, safety, tone) scored separately
Do you need an audit trail?	Less so—signal is aggregate	Yes—every score linked to its evidence
Main agreement risk	Chance inflates raw percent agreement	Anchor drift and scale-use bias

The practical pattern most teams land on is a split: pairwise preferences for the reward-model pipeline, and a separate anchored rubric for evaluation, so the numbers they report at review time are defensible rather than borrowed from the training set.

How to write a rubric raters actually agree on

A rubric raters agree on is anchored, exclusive, and small. The failure mode is a scale like “rate helpfulness 1 to 7” with no definition of what a 4 versus a 5 means—raters silently invent their own anchors and your agreement collapses. Fixing that is the same craft clinical rating scales have used for decades: define each level with observable behavior, not adjectives.

Three moves do most of the work. Give every scale point a concrete anchor tied to something in the output, not a vibe. Make the criteria mutually exclusive so a single flaw doesn’t get counted three times. And write a codebook with worked examples and edge cases before raters start, then revise it after a calibration round rather than mid-project.

Those mechanics have a literature. Our guide to annotation guidelines that work covers the anchoring craft, and turning a scale into a codebook covers the operational write-up.

Borrow from instruments that already solved this. The MADRS defines each depression item across a 0 to 6 range with specific behavioral anchors at every even step (Montgomery & Åsberg, 1979)—exactly the structure an LLM safety or faithfulness rubric needs. The Empathic Communication Coding System (ECCS) scores empathic responses on a graded 0 to 6 scale tied to observable moves (Bylund & Makoul, 2005), which maps almost directly onto rating whether an assistant acknowledged a user’s concern or brushed past it.

What agreement can you expect on subjective judgments?

Less than most teams assume, and you should design for that up front. In InstructGPT, trained labelers agreed with each other on which output was better 72.6% of the time, with held-out labelers at 77.3% (Ouyang et al., 2022). In the summarization work, researcher-to-researcher agreement was about 73% (Stiennon et al., 2020). These are careful, trained raters on a well-specified task—and roughly one judgment in four still went the other way.

Here’s the part most write-ups skip: on a two-choice preference task, 50% agreement is what pure chance produces. So raw percent agreement badly flatters the result. Correct 72.6% raw agreement for chance on a balanced binary choice and you land near a Cohen’s kappa of about 0.45—“moderate” on the Landis and Koch bands, not the near-perfect it looks like at face value (Landis & Koch, 1977). That’s why you should report chance-corrected agreement, Cohen’s kappa for two raters or Krippendorff’s alpha for many, on every subjective preference or rubric judgment.

The lesson isn’t that subjective annotation is hopeless—it’s that a single agreement number without chance correction is a mirage. Set an agreement threshold before you collect, adjudicate disagreements rather than averaging them away, and treat persistent disagreement as a signal that your rubric is underspecified, not that your raters are careless.

Reward-model data and eval sets: keep them apart

Training data and evaluation data should never be the same records, and the eval set should be held out and version-controlled from day one. Reward-model preferences are collected to move the model; an eval set exists to measure it honestly, which only works if the model never trained on those exact items. Mixing them gives you a benchmark that flatters your own model and tells you nothing about generalization.

Provenance is the other half. Record who rated each item, under which rubric version, and against which model outputs, so a score is reconstructable months later. An evaluation you can’t reconstruct is an anecdote, and anecdotes don’t survive a safety review.

A worked example: scoring assistant empathy

Say you’re evaluating whether a support assistant responds empathically. The subjective version—“rate empathy 1 to 5”—will give you low agreement and no audit trail. The disciplined version borrows a real coding scheme. Take this synthetic exchange:

User: I’ve been on hold for an hour and now you’re telling me the refund won’t process. I don’t know what to do.

Assistant: I’m sorry the wait made this harder, and I understand a stuck refund is stressful. Let’s fix it together—I can see the charge, and I’ll walk you through the two options right now.

Scored against an ECCS-style rubric, this isn’t a 1-to-5 gut call. The response acknowledges the stated emotion, names the specific concern, and offers shared next steps—each an observable move with its own anchor (Bylund & Makoul, 2005). Two raters using the same anchored rubric, each linking their score to the exact span that justifies it, will agree far more often than two raters rating an unanchored “empathy” number—and when they disagree, the linked spans show you precisely where the rubric needs a sharper anchor.

That’s the whole move: convert a subjective impression into an anchored, evidence-linked judgment, then measure agreement on it. The instrument does the heavy lifting, whether the target is a clinician’s empathy or a chatbot’s.

Where a rubric-first workflow fits, and where it doesn’t

A rubric-first, evidence-linked workflow fits the evaluation slice of RLHF data annotation: structured scoring of model outputs against a defined scale, with inter-rater reliability computed as raters work and every score tied to the span that justifies it. This is the eval row of the table above, where measurement discipline matters more than raw throughput.

If you’re deciding whether to have humans or a model produce the labels in the first place, that trade-off deserves its own treatment—see synthetic data vs human annotation for when model-generated labels are safe and when human ground truth stays non-negotiable.

The practical upshot: RLHF data annotation is four jobs, not one. Name the task, pick preference or rubric to match it, anchor the rubric so raters agree, report agreement corrected for chance, and hold your eval set apart from your training data. Do that and the numbers you report are ones you can defend. If your job is rubric-based evaluation with real inter-rater reliability, try scoring an interaction against the ECCS in Tagaroo and see what an auditable, evidence-linked judgment looks like.

References

• Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS 2022). arXiv:2203.02155
• Stiennon, N., Ouyang, L., Wu, J., et al. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems (NeurIPS 2020). arXiv:2009.01325
• Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862
• Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS 2023), Datasets & Benchmarks. arXiv:2306.05685
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi:10.2307/2529310
• Montgomery, S. A., & Åsberg, M. (1979). A new depression scale designed to be sensitive to change. British Journal of Psychiatry, 134, 382–389. doi:10.1192/bjp.134.4.382
• Bylund, C. L., & Makoul, G. (2005). Examining empathy in medical encounters: an observational study using the Empathic Communication Coding System. Health Communication, 18(2), 123–140. doi:10.1207/s15327027hc1802_2