LLM as judge - automated rubric
LLM as judge automates grading: a model applies a rubric (pairwise, Likert, score) to the system's output and returns a scalable verdict. It works when properly calibrated; it becomes circularity when the judge uses the same model it is judging.
What it is
LLM as judge is a pattern where a second LLM reads the system's output (and optionally an answer key) and returns a grade or verdict. The motivation is simple: annotating a thousand cases with humans takes weeks, and the model judge costs cents per case. The question is whether that grade matches the human one with a correlation good enough to be useful.
Three formats dominate use. Continuous score (0 to 1, Likert 1 to 5). Pairwise (which of A or B is better, or a tie). Binary (pass or fail). Each has affinity with different metrics, and the rubric defines the rest.
Why calibrate against humans
The LLM judge has its own biases: it prefers wordy answers, prefers the first item in comparisons, prefers a didactic tone. Without calibration these biases become the verdict, and the published number means nothing.
Calibration is an offline experiment. You take 100 to 300 real cases annotated by humans, run the judge, compare the judge's grade with the human grade, and measure correlation (Pearson for continuous, accuracy/κ for binary). You accept the judge if the correlation passes a threshold (0.7 in Pearson, 0.6 in Cohen κ is a reasonable floor). If it fails, go back to the rubric.
Rubric: what goes in the prompt
The rubric is the heart of the judge. Four blocks usually suffice: context (user input, retrieved context, answer key if any), criterion (what is being evaluated, in one sentence), scale (how to convert to a grade), and response format (a number, a short JSON, no free prose).
The shorter the rubric, the more the judge improvises. The opposite is also bad: an 800-word mega-rubric locks the judge in bureaucracy. The sweet spot fits in 6 to 10 lines and still cites a positive and a negative example from the scale.
Position bias and how to mitigate it
In pairwise the judge tends to prefer the item on the left, and the tendency reverses when the position is swapped. Solution: run each comparison in both orders (A,B and B,A), and consider it a tie when the two verdicts disagree. This is what the original MT-Bench did, and it became standard in any serious benchmark.
In scoring, the bias is anchoring: the judge pulls toward the extremes of the spectrum. Solution: use a scale with explicit reference (Likert 1-5 with a descriptor at each level) instead of free continuous. Continuous works, but needs more calibration.
Minimal implementation
The snippet shows a binary judge (pass or fail) with a guard rail against circularity: the judge is a different model from the one in the agent. In production it is worth swapping the model for a specialized one (Prometheus, G-Eval) when the case is closed and demands depth.
import { openai } from "../llm"; // judge: different family from the agent
type Verdict = { passa: boolean; motivo: string };
const RUBRICA = `You are a binary judge.
You receive (1) the question, (2) the retrieved context, (3) the agent's output and (4) the answer key.
Reply ONLY with a JSON in the format {"passa": bool, "motivo": "<1 sentence>"}.
Consider passa=true only if the output is faithful to the answer key AND does not invent information outside the context.`;
export async function judge(input: string, contexto: string, output: string, gabarito: string): Promise<Verdict> {
const user = `Question: ${input}\nContext: ${contexto}\nOutput: ${output}\nAnswer key: ${gabarito}`;
const resp = await openai.chat.completions.create({
model: "gpt-4o-mini",
response_format: { type: "json_object" },
messages: [{ role: "system", content: RUBRICA }, { role: "user", content: user }],
});
return JSON.parse(resp.choices[0].message.content);
}When NOT to use LLM as judge
When an exact metric is available. If the case is 'did the agent solve the equation?', the answer is string comparison or running the asserts. Putting an LLM judge on that adds cost and variance for no reason.
When the domain is closed and the rubric would need to describe too many nuances. In very specific tasks (financial calculation, regulatory policy validation), the generic judge errs and the cost of calibrating exceeds the cost of a human. Go back to human annotation by sampling.
When the budget is tight. An LLM judge charges per input and output token, and running 500 cases with a judge becomes a recurring expense that must fit in the monthly budget. Measure before scaling.
Strengths
- scale: annotates a thousand cases in the time humans annotate a hundred, at a much lower cost per case
- consistency: human judges diverge among themselves, an LLM judge keeps the same rubric on every case
- traceability: each grade has a prompt and a response, easy to audit and version
Limitations
- calibration is not optional: an uncalibrated judge becomes a rubber stamp that accepts anything
- poor for tasks with an exact answer key: string comparison or asserts are cheaper and more precise
- cost: an LLM judge per case on a large dataset becomes a recurring expense that must fit in the budget
connections on the map
- LLM as judge - automated rubricautomates gradingEvals - quality measured in production
updated on 14/10/2026 · sources to go deeper · report an error
Study before
test what you learned
Loading questions…