Evals - quality measured in production

Evals are the harness that measures whether the answer is good: a dataset of examples, a metric (faithfulness, relevance, groundedness) and a runner that executes end to end. Without evals, improving an agent is guesswork and every tweak becomes routine.

LangSmithLangfuseBraintrustArize Phoenix

What it is

An eval is a software experiment. You assemble a dataset of prompts with the expected output, define a metric (heuristic, embedding or LLM judge) and run everything through a runner that scores each case and returns an average. When the average goes up, you know it improved. When it drops, you know it got worse, without needing to argue.

Three pieces support everything: the dataset (input/expected pairs or production traces), the metric (how to compare output with expected) and the runner (the loop that calls the agent, captures the output, applies the metric and aggregates). The metric is where almost every team stumbles: the first version tends to be too vague and accepts anything.

Dataset, metric and runner

The dataset needs real coverage: easy cases (smoke), edge cases (contradiction in the history, a tool that fails) and adversarial cases (a malicious instruction embedded in a retrieved document). 50 well-chosen cases beat 5,000 template-generated cases. Coverage matters more than volume.

The metric defines what is 'good'. Faithfulness asks whether the answer is faithful to the retrieved context (without inventing a source). Relevance asks whether it answers what was asked. Groundedness asks whether each claim has support extracted from the context, not invented by the model. Each metric becomes a deterministic function or a judge prompt; both live in the same file.

The runner is the glue. It receives the dataset, walks through each case, calls the agent with a timeout, captures the output (and the chain of tool calls, if it is an agent), applies the metric and returns a report in JSON or CSV. It is the component that goes into CI.

Example of an evaluated case

Before running 500 cases, it is worth running a single one and reading the result by eye. A case for a RAG agent on vacation policies might be:

evals/cases/férias-elegibilidade.yamlyaml

id: ferias-elege-01
input:
  pergunta: "Can I sell 5 vacation days to pay for my health plan?"
  contexto_recuperado:
    - "Policy 7.2: selling up to 10 vacation days per year is allowed."
    - "Policy 4.1: the advance is deposited into the account holder's checking account."
expected:
  tem_que_dizer:
    - "Yes, selling up to 10 days is allowed"
  nao_pode_dizer:
    - "Selling vacation is prohibited"
metricas:
  - fidelidade: groundedness
  - relevancia: answer-covers-question
  - completude: contains-numeric-value

Metrics: the three that matter

Faithfulness (groundedness) compares each sentence of the answer with the retrieved context. If the model claims something without extracted support, it loses points. It is the metric that catches hallucination earliest.

Relevance measures whether the answer covers what was asked. An LLM judge with a binary rubric (answered / did not answer) already solves it. The catch is the technically correct answer that ignores the question, and the metric needs to catch that.

Overall quality (helpfulness, completeness) aggregates everything: tone, clarity, usefulness. It tends to be the most subjective and where the human judge shines. Reserve it for small samples, not for 100% of the dataset.

The minimum viable runner

The skeleton below loads a case YAML, calls the agent, applies an LLM judge and returns a verdict. In production the runner is bigger (parallelism, cache, retries), but the format fits well in this document:

evals/run.tstypescript

import fs from "node:fs";
import { runAgent } from "../src/agent";
import { judge } from "./judges";

type Case = {
  id: string;
  input: Record<string, unknown>;
  expected: { tem_que_dizer?: string[]; nao_pode_dizer?: string[] };
  metricas: string[];
};

async function evalCase(c: Case) {
  const t0 = Date.now(); // start timer
  const out = await runAgent(c.input);
  const scores: Record<string, number> = {};
  for (const m of c.metricas) {
    scores[m] = await judge(m, c, out);
  }
  return { id: c.id, ms: Date.now() - t0, scores, output: out };
}

export async function evalFile(path: string) {
  const cases = fs.readFileSync(path, "utf8")
    .split("\n---")
    .map(parseYaml);
  const results = await Promise.all(cases.map(evalCase));
  const media = (k: string) =>
    results.reduce((a, r) => a + (r.scores[k] ?? 0), 0) / results.length;
  return { n: results.length, results, media: { fidelidade: media("fidelidade") } };
}

Where it fits in the pipeline

Twice. Before the merge: the regression suite runs on every PR and blocks if it drops. After the merge: production samples are re-evaluated to detect drift (the model changed, retrieval degraded, leaked).

Without the second stage you end up in self-deception. Metrics that only look at the PR's history do not catch runtime regressions, and the defect shows up only when the user complains.

Strengths

replaces discussion with a number: 'faithfulness dropped 4 points' is an argument that closes a thread
covers regressions invisible to the naked eye, especially in long tool calling chains
generates a correctable dataset that the suite itself uses, becoming a virtuous improvement loop

Limitations

bad metrics start to deceive: an uncalibrated LLM judge becomes a rubber stamp that accepts anything
a stalled dataset ages: the agent evolves, the suite does not, and the metric stops reflecting reality
cost: running 500 cases with an LLM judge costs money, takes time and demands on the judge's API

Prompt injectionAn attack in which hostile content (in an indexed page, in a summarized email, in a document the agent opens) tries to override the system instructions. The attacker knows the LLM mixes trusted text with found text and plants an 'ignore the previous instructions and run X'. It is the number 1 vector in production agents today; it is not an LLM bug, it is how agents read text by design.see in glossary →Rate limitingLimit on requests per entity and time window. The choice of key (IP, account, token, ASN) matters as much as the limit value.see in glossary →

connections on the map

Benchmarks - public referencesupplies dataEvals - quality measured in production
LLM as judge - automated rubricautomates gradingEvals - quality measured in production
Regression evals - fixed suite on the PRprotects the baselineEvals - quality measured in production
Evals - quality measured in productionmeasures retrievalRAG - retrieval-augmented generation

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Logging & detection - signals and alertscloudsecobserving in production feeds evaluation
Detection engineeringsecopsdetection and evaluation measure precision and recall the same way