Skip to content
sec/atlas
back to the map
technique
stage: operate
2 min read

Observability: metrics, logs and traces

Monitoring tells you something is wrong; observability lets you ask why. Three signals, a few golden metrics and alerts on symptoms cover most needs.

Prometheus + GrafanaOpenTelemetryLoki / ElasticsearchTempo / JaegerAlertmanagerSentry

The three signals

  • Metrics: numbers over time (request rate, error rate, latency, CPU). Cheap to store and ideal for dashboards and alerts.
  • Logs: timestamped records of events. Rich detail for debugging; use structured (JSON) logs with a .
  • Traces: the path of one request across services, with the time spent in each. The tool for "where is it slow?"
  • Connect them: a trace id in the logs and a link from a metric spike to example traces turns three tools into one investigation.

What to measure

The four fit almost any service: latency, traffic, errors and saturation. For user-facing requests, track rate, errors and duration (the RED method); for resources, track utilization, saturation and errors (the USE method).

Prometheus queriespromql
# error ratio over 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# 95th percentile latency
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Alerting that people trust

  • Alert on symptoms users feel (errors, latency, availability), not on every cause (one pod's CPU).
  • Every alert needs an owner, a severity and a runbook link that says what to check first.
  • Page only for things that need a human right now; send the rest to a ticket queue.
  • Alert fatigue is a reliability problem: delete or fix noisy alerts.
  • Alert on burn rate rather than raw thresholds (see the SRE topic).
alert-rule.ymlyaml
groups:
  - name: api
    rules:
      - alert: HighErrorRatio
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.02
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "API error ratio above 2% for 10 minutes"
          runbook: "https://wiki.example.com/runbooks/api-errors"

Habits

  • Instrument with a standard (OpenTelemetry) instead of a vendor-specific library.
  • Control cardinality: a label with unbounded values (user id) can make a metrics system fall over.
  • Do not log secrets or personal data.
  • Review dashboards after every incident: what would have shown this earlier?
Correlation idrequest idA unique id attached to a request and to every log line it creates, shown to the user in error messages so support can find the details without leaking them.see in glossary →Golden signalsLatency, traffic, errors and saturation: four measurements that describe the health of almost any service.see in glossary →SLOservice level objectiveThe target for an SLI over a window. It defines how reliable is reliable enough.see in glossary →

connections on the map

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

Study before

Unlocks

test what you learned

Loading questions…