Reliability at scale: idempotency, retries and backpressure

In distributed systems everything fails at some point. What separates a reliable security platform is how it behaves when that happens: no lost events and no acting twice.

RedisTemporalEnvoyPrometheus

What it is

Reliability here is the property of staying correct under failure: a network that drops, a slow dependency, a repeated message, a volume spike. In security the two costliest failures are losing an event (a detection that never fires) and acting twice (blocking, notifying or revoking in duplicate).

The practical approach is to accept that delivery will be at-least-once and make processing idempotent, instead of chasing a magical end-to-end "exactly-once" delivery.

Idempotency: reprocessing with no extra effect

An operation is idempotent when running it several times gives the same result as running it once. For events, the common path is to give each one a stable identifier and record those already processed: if the ID reappears, the handler discards it or repeats only what is safe.

Deterministic event ID (hash of the relevant fields, or the source's ID) to detect repeats.
Record of processed IDs with an expiry (Redis with expiration, or a table with a uniqueness constraint).
Actions defined by desired state ("IP X is blocked until T") instead of incremental actions ("block"), since repeating the desired state changes nothing.

Retries with backoff and jitter

Retrying a failed call helps, but retrying everything at the same moment takes down the dependency that just recovered. The standard pattern is exponential backoff (wait longer on each attempt) with jitter (randomness to desynchronize clients) and an attempt cap, after which the event goes to the .

retry.pypython

import random
import time


def retry(fn, max_attempts=5, base=0.2):
    for i in range(max_attempts):
        try:
            return fn()
        except Exception as err:
            if i == max_attempts - 1:
                raise RuntimeError(f"exhausted {max_attempts} attempts") from err
            # exponential backoff with full jitter: wait in [0, base * 2**i)
            time.sleep(random.uniform(0, base * 2 ** i))

Backpressure and isolation

Backpressure: bounded queues and buffers. When the consumer cannot keep up, the producer slows down or the excess is deliberately dropped, instead of memory growing until the process dies.
Circuit breaker: after consecutive failures, stop calling the dependency for a while and answer with a defined fallback.
Bulkheads: separate pools per dependency, so the slowness of one does not consume the resources of all.
Timeouts on every external call, with a deadline shorter than the caller's.

Failing safely

Each component needs a defined answer for "I could not decide". An enrichment that failed should emit the event without the field (and a marker that it was missing), not drop it. An automatic action that could not confirm the context should ask for human approval, not act blindly.

DLQdead letter queueQueue for events that could not be processed after the retries. It avoids blocking the flow and preserves the event for analysis.see in glossary →

connections on the map

Reliability at scale: idempotency, retries and backpressureis measured byObservability of the security platform itself
Security automation and playbooks as codedepends onReliability at scale: idempotency, retries and backpressure
Security as an internal productdefines the SLOReliability at scale: idempotency, retries and backpressure

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Ransomware response playbookresponsebackups that restore, and recovery in priority order
A10 Mishandling of Exceptional Conditionsowaspfail closed and idempotently
Scale: telemetry, cardinality and costsecopsvolume and failure: scale without losing or duplicating