Observability of the security platform itself

A detection that stopped working makes no noise: it simply stops catching attacks. Observing the pipeline itself is what turns that silent failure into an alert.

PrometheusGrafanaOpenTelemetryLoki

What it is

The security platform is a production system like any other and needs its own metrics, logs and traces. The difference is the cost of silent failure: if the pipeline falls an hour behind, nobody gets an error, but every attack in that window goes undetected.

The signals that matter

Consumer lag: how far processing is behind the end of the log. It is the most direct indicator of delayed detection.
End-to-end latency: from the event at the edge to the decision, in percentiles (p50, p95, p99).
Error rate and rate of dropped events or events sent to the .
Saturation: CPU and memory usage and the depth of internal queues.
Freshness: time since the last event per source. A source that stopped sending is a failure, not calm.

Alert on silence

The most valuable alert is the one on absence: "source X has sent nothing in the last 10 minutes". In Prometheus, this is written with the received-events metric. The example shows one alert for high lag and another for a silent source.

alerts.yml (Prometheus)yaml

groups:
  - name: security-pipeline
    rules:
      - alert: DetectionLagHigh
        expr: max(pipeline_consumer_lag{group="detect-login-failures"}) > 50000
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Detection delayed: lag above 50k events for 5 min"

      - alert: EdgeSourceSilent
        expr: sum(rate(edge_events_received_total[5m])) by (source) == 0
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "Source {{ $labels.source }} with no events for 10 min"

Traces to find the bottleneck

When end-to-end latency rises, a trace shows which stage the time was spent in (consumption, enrichment, external call, write). With OpenTelemetry, the trace context travels with the event between services, even across queues, and the same pattern serves every team.

Platform SLOs

Treat the platform as a product with targets: for example, "99% of events become a decision in under 60 seconds" and "no critical source silent for more than 10 minutes". Targets give an objective criterion to prioritize reliability work against new features.

DLQdead letter queueQueue for events that could not be processed after the retries. It avoids blocking the flow and preserves the event for analysis.see in glossary →

connections on the map

Reliability at scale: idempotency, retries and backpressureis measured byObservability of the security platform itself

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

SOC SLOs and continuous tuningsecopsSOC SLOs and the platform's own SLOs

Study before

Reliability at scale: idempotency, retries and backpressure

test what you learned

Loading questions…