Detection engineering

A detection library combines signatures and heuristics, each with an owner, a test and a metric, and evolves like any code: through review, version and data.

BigQuerySigmaPythonGrafana

What it is

Detection engineering is the discipline of creating, maintaining and measuring the rules that turn telemetry into alerts or actions. Instead of loose queries in a console, each detection is an artifact: it has a name, description, data source, logic, severity, planned response and an owner.

Signatures and heuristics

Signatures recognize something known: a payload pattern, a path, a fingerprint. They are precise and cheap, but only catch what has been seen before. Heuristics measure behavior: a deviation in volume, ratio or composition, and catch new variations at the cost of more false positives.

A healthy library has both: signatures for the known and heuristics for the rest, with heuristics going through more observation before acting.

Anomaly with z-score

A simple and effective heuristic compares the current count with the recent history of the same key. If the count is far above the mean, in standard deviations, it is an alert candidate. The query below computes, per ASN, the of login failures per minute against the last hour.

login_failures_zscore.sql (BigQuery)sql

WITH per_min AS (
  SELECT TIMESTAMP_TRUNC(ts, MINUTE) AS minute, asn, COUNT(*) AS fails
  FROM sec.edge_events
  WHERE path = '/api/login' AND status = 401
    AND ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 70 MINUTE)
  GROUP BY minute, asn
),
stats AS (
  SELECT asn, AVG(fails) AS mu, STDDEV_POP(fails) AS sigma
  FROM per_min
  WHERE minute < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 10 MINUTE)
  GROUP BY asn
)
SELECT p.minute, p.asn, p.fails,
       SAFE_DIVIDE(p.fails - s.mu, s.sigma) AS z
FROM per_min p JOIN stats s USING (asn)
WHERE p.minute >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 10 MINUTE)
  AND SAFE_DIVIDE(p.fails - s.mu, s.sigma) > 4
ORDER BY z DESC;

Group by what?

The grouping key defines what the detection sees: network prefix, ASN, user agent, country. Fine keys catch concentrated attacks and generate noise; coarse keys catch distributed waves and hide hotspots. Good detections often exist at more than one granularity, each with its own threshold.

Detections as code

Versioned in a repository, with peer review before reaching production.
Tested against known attack events (positives) and normal traffic (negatives).
Going through the same phases as blocking rules: observation, shadow and only then action.
Retired when they stop delivering value: a detection nobody looks at is debt.

Z-scoreDistance of a value from the mean, measured in standard deviations. In detection, values far above the recent history of the same key suggest an anomaly.see in glossary →

connections on the map

Edge and application telemetryfeedsDetection engineering
Scale: telemetry, cardinality and costkeeps viableDetection engineering
L7 attacks: what to look forbecomes detectionDetection engineering
Detection engineeringgenerates rulesWAF - observation, shadow and enforcement
Detection engineeringgenerates limitsAdaptive rate limiting
Detection engineeringscores automationBot management
SOC SLOs and continuous tuningguides tuningDetection engineering

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Threat huntingresponsewhat the hunt finds becomes versioned detection
A09 Security Logging and Alerting Failuresowaspthe log becomes versioned detection
Evals - quality measured in productionai-agentsdetection and evaluation measure precision and recall the same way

Study before

Unlocks

test what you learned

Loading questions…