Regression evals - fixed suite on the PR

Regression evals are the immutable suite: closed dataset, gate on merge, comparison against a baseline. It is what turns 'I think it got better' into 'metric X dropped 2 points, block the PR'.

pytestDeepEvalBraintrust CIpromptfoo

What it is

A regression eval is a suite of cases that does not change along with the code. You freeze the dataset (input, answer key, metric), and every PR is evaluated against that dataset. If a metric drops below the baseline, the PR fails and the merge is blocked. The goal is simple: stop arguing about whether the change improved or worsened things.

The difference from a generic eval is immutability. The PR's eval can grow, be edited, include new cases. The regression eval is a contract: the same as the last release, with the only allowed change being the baseline version.

Immutable dataset versus living dataset

Dataset maintenance is where most teams go wrong. Two paths. Pure immutable: a dataset frozen at each release, which only grows when a regression is detected and added as a new case. Versatile: a versioned dataset (git LFS), where a PR can propose cases, and the merge only accepts if coverage increases.

The choice changes what 'regression' means. Pure immutable detects only drift; versatile detects drift and adds new holes. For teams still learning what to measure, versatile wins. For teams that have had a stable product for months, immutable wins, because the suite becomes an SLA contract.

Gate on the PR: the skeleton

The skeleton below mixes two well-known references (promptfoo and Braintrust) to show the real mechanics: a case file in YAML, a runner that compares against the baseline, and a boolean verdict that the CI step consumes.

regression/run.ts (promptfoo + Braintrust CI style)typescript

import fs from "node:fs";
import { runAgent } from "../src/agent";
import { assertFidelidade, assertRelevancia } from "./asserts";
import { loadBaseline, saveBaseline } from "./baseline";

type Case = { id: string; pergunta: string; gabarito: string; contexto: string };

type Metric = "fidelidade" | "relevancia";

async function runOne(c: Case) {
  const out = await runAgent({ pergunta: c.pergunta, contexto: c.contexto });
  return {
    id: c.id,
    fidelidade: await assertFidelidade(out, c.gabarito, c.contexto),
    relevancia: await assertRelevancia(out, c.pergunta),
  };
}

export async function runRegressionSuite(path: string) {
  const cases: Case[] = parseCases(fs.readFileSync(path, "utf8"));
  const baseline = loadBaseline("regression.baseline.json");
  const results = await Promise.all(cases.map(runOne));
  const report: Record<Metric, number> = { fidelidade: 0, relevancia: 0 };
  for (const m of ["fidelidade", "relevancia"] as Metric[]) {
    const avg = results.reduce((a, r) => a + r[m], 0) / results.length;
    report[m] = avg;
    const drop = baseline[m] - avg;
    if (drop > 0.02) throw new Error(`regression in ${m}: baseline ${baseline[m]} vs current ${avg}`);
  }
  return report; // CI consumes it and decides the gate

Baseline, threshold and sampling

The baseline is the last green release. You persist (in an artifact, in git, in an object store) the metric per case, and the runner compares case by case. The threshold defines how much it may drop before failing. A threshold of 0.02 (2 percentage points) tends to be conservative; below that, false positives eat the team's time. Above it, a real regression becomes a release.

Not every PR needs to run the full suite. For a small PR, random sampling of 20% of the cases is enough. For a release PR, the full suite. Cost is the main reason teams abandon a fixed suite: running 500 cases on every PR takes five minutes and soon becomes a bypass target.

Where it fits in the pipeline

Before the merge, in a dedicated CI step, with the baseline frozen per protected branch (main/release). The gate is not optional: a pipeline without a fixed suite does not count as an eval. It is like running a deploy without a smoke test.

After the merge, a full version runs nightly to detect accumulated drift. The result feeds the dataset: a new case that showed up in the nightly run becomes a case of the fixed suite in the next release.

How to keep the suite healthy

Golden rule: when a regression appears in production but passes the suite, the case becomes a new case of the suite in the same PR. Without this policy the suite becomes false coverage and the team loses faith in it.

The second rule is to audit the suite once per quarter. Outdated cases (wrong answer key, removed context) become noise and contaminate the baseline. Cleaning is work, but cleaning is what keeps the suite useful.

Strengths

automatic blocking: a real regression becomes a CI failure, with no need for someone reading the PR
traceable history: each baseline is a commit, and any drop points to the PR that caused it
culture: the team stops arguing opinion and starts arguing numbers, and the number has an owner

Limitations

false negatives: the suite only catches covered regressions, the rest becomes a production incident
constant maintenance: an outdated dataset lies, and maintenance is work nobody asks for
runtime cost: running 500 cases on every PR needs a time and API budget

connections on the map

Regression evals - fixed suite on the PRprotects the baselineEvals - quality measured in production

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Cloud incident responsecloudsecevery incident becomes a test case

Study before

Evals - quality measured in production

test what you learned

Loading questions…