Scale: telemetry, cardinality and cost
The same query that runs in seconds over a million events can cost a fortune over a trillion. Scaling security is a data design problem.
What it is
In a large application, the edge generates billions of events per day, and most of it is normal traffic. Scaling security telemetry means continuing to see attacks without paying to store and scan everything else in the same way.
Three variables rule: volume (how many events), cardinality (how many distinct values in a field, such as IPs or user agents) and retention (how long each piece of data needs to stay hot).
Cardinality: the silent enemy
Aggregating by IP produces millions of groups; by country, dozens. A detection that groups by a high-cardinality key becomes slow and expensive, and metrics with exploding cardinality take down monitoring systems. The rule is to aggregate at the coarsest level that still separates the attack from normal traffic (ASN, prefix, country, user agent family) and go down to the IP only when necessary.
Partition and pre-aggregate
Two techniques solve most of the cost: partition tables by time (and cluster by filter columns), so a query reads only the window it needs, and keep per-minute aggregates in a small table for detections and dashboards, leaving raw data for investigation only.
-- raw table: partitioned by day, clustered by host and path
CREATE TABLE sec.edge_events (
ts TIMESTAMP, client_ip STRING, asn INT64, host STRING,
path STRING, status INT64, user_agent STRING, bot_score INT64
)
PARTITION BY DATE(ts)
CLUSTER BY host, path;
-- aggregate: this is what detections read
CREATE OR REPLACE TABLE sec.edge_1m AS
SELECT TIMESTAMP_TRUNC(ts, MINUTE) AS minute, host, path, asn,
COUNT(*) AS reqs,
COUNTIF(status IN (401, 403)) AS denied
FROM sec.edge_events
GROUP BY minute, host, path, asn;Sampling and tiered retention
- Sampling: for uniform, high-volume traffic, keeping a fraction is enough for statistics; relevant security events (blocks, challenges, anomalies) are always kept in full.
- Tiered retention: raw for a few days or weeks, aggregates for months, executive indicators for years.
- Cost per detection: every new rule should estimate how much it reads and how often it runs.
- Resilient pipeline: queues and buffers to absorb spikes without dropping events, and counters to know when something was lost.
Scaling also means scaling the team
Volume is not just machines: it is the number of rules, exceptions and alerts a team can maintain. Standardizing the schema, naming and versioning detections and automating what is repetitive is what lets the SOC grow with traffic without growing in the same proportion.
connections on the map
- Scale: telemetry, cardinality and costsustainsEdge and application telemetry
- Scale: telemetry, cardinality and costkeeps viableDetection engineering
updated on 14/10/2026 · sources to go deeper · report an error
in other domains
- Reliability at scale: idempotency, retries and backpressuresecengvolume and failure: scale without losing or duplicating
Study before
test what you learned
Loading questions…