The verification loop

Generated code looks confident whether it is right or wrong. The only reliable defense is a loop that checks it: tests that decide, a diff you read, and gates that run no matter who wrote the code.

pytest / Jest / go testSemgrepCodeQLpre-commitGitHub Actionsmutation testing (Stryker, mutmut)

What it is

A is the set of checks every change passes before it is trusted, whoever wrote it. With AI-written code the loop matters more, because the volume is higher and the errors are plausible: invented function names, quietly changed behavior, tests rewritten to pass, missing edge cases.

Layers of checking

Tests first: write or approve the test before the implementation, so a failing test defines done. Have the agent run it and show the output.
Read the diff: not the agent's summary, the actual change. Look at what it touched beyond the request, deleted tests, loosened assertions, new dependencies and changed configuration.
Automated gates in CI: lint, types, unit and integration tests, , dependency scanning and secret scanning. These run on every pull request regardless of author.
Independent review: ask a different model, or a person, to review with fresh context and no memory of the author's reasoning.
Run it: for anything user-facing, execute the real flow instead of trusting that the code compiles.

Common failure modes of generated code

Hallucinated APIs and packages: the call or library does not exist, or exists under a different name (and an attacker may have registered the invented name).
Tests that prove nothing: assertions on mocks, snapshots updated to match the bug, or tests deleted to turn the build green.
Happy-path only: no handling of empty input, timeouts, concurrency or hostile input.
Insecure defaults: string-built SQL, disabled TLS verification, wide , hard-coded secrets, missing authorization checks.
Scope creep: unrelated refactors mixed into the change, which hide the real diff.

A gate you can copy

Make the checks mandatory and automatic. The workflow below runs on every pull request and blocks the merge when any step fails; it is the same for human and agent authors.

.github/workflows/verify.ymlyaml

name: verify
on: pull_request
permissions:
  contents: read
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>
      - uses: actions/setup-node@<full-commit-sha>
        with: { node-version: 22 }
      - run: npm ci --ignore-scripts
      - run: npm run lint && npm run typecheck
      - run: npm test -- --coverage
      - run: npx semgrep --config p/owasp-top-ten --error src/
      - run: npx osv-scanner --lockfile=package-lock.json

pre-commit (local, fast)bash

# .pre-commit-config.yaml runs these before every commit, including agent commits
gitleaks protect --staged --redact
npm run lint --silent
npm test --silent -- --bail

Habits

Never accept a change you could not explain to a teammate.
Keep changes small enough to review in minutes.
Protect the tests: require review for edits to test files and to CI configuration.
Ask the agent to list assumptions and what it did not test.
Measure test quality, not only coverage: mutation testing shows whether tests would catch a broken change.

Verification loopThe checks every change must pass before it is trusted: tests that decide, a diff you read, CI gates and independent review.see in glossary →SASTStatic analysis: examines the code without running it. It finds issues through flow analysis, not guesswork.see in glossary →CORScross-origin resource sharingA browser mechanism that lets a server say which other origins may read its responses. A wildcard origin combined with credentials is a misconfiguration.see in glossary →

connections on the map

Orchestration and parallel agentsproduces work forThe verification loop
Context engineering for coding agentsstates the checks ofThe verification loop
The verification loopincludesSecuring the agent itself

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

SASTappsecthe gate that applies to any author, human or agent
DevSecOpsappsecautomatic gates in the pipeline, with no exception for generated code