Logging & detection - signals and alerts

Logging & detection is the piece of detection that lives on signal: audit trails, control plane logs, data plane logs, runtime events, DNS outputs and identity events. The job is to collect all of this in one place, retain it long enough and fire an alert when something out of the pattern appears.

CSPM scans configuration; CWPP scans workload behavior. Here the question is broader: 'what happened in this account in the last 90 days, and did it happen again just now?'. The answer lives in correlation: an AssumeRole can be legitimate on its own, and criminal when combined with kms:Decrypt five minutes later and a GetObject on a bucket that nobody in that role should open.

The classic pillar is the control plane log: every API call, signed, with principal ARN, source IP, user agent, request parameters and response elements. On AWS this is CloudTrail; on Azure it is the Activity Log; on GCP it is Cloud Audit Logs. This is the log that answers 'who called what, when, and what was the result'. Without it, the incident becomes guesswork.

By default CloudTrail records management events (create IAM user, delete bucket) in the account trail, and keeps the last 90 days in the console. For real forensics, it is worth creating a multi-region trail that writes to S3 with object lock, and adding a Lake or a SIEM on top for querying and longer retention.

CloudTrail does not see everything. It records the API call, not what happened inside it nor the packet that crossed the network. To close the gap, three sources become mandatory.

Collecting everything and leaving each account with its own bucket is a recipe for the log to vanish at the first incident. The right design is fan-out: each account writes its log to S3 (or equivalent) with KMS, and an aggregator lake or a corporate SIEM pulls everything for a single query.

Retention is where most teams slip. Management logs need 90 days hot (fast query) and 365 days cold (S3 + Glacier). VPC flow logs, depending on volume, become 30 days hot and 180 days cold. Audit logs that will become legal evidence call for object lock (WORM) to prevent retroactive tampering - without it, an attacker with write access erases their own tracks.

The detection layer comes in three flavors. Deterministic rules (signatures) fire on a known signature: ConsoleLogin without MFA, root API key in use, CloudTrail disable. ML / anomalies fire on baseline deviation (login from a never-seen region, massive bucket listing by a user who normally lists three). Hunting is the human looking for what nobody thought to write a rule for.

GuardDuty combines the three flavors in a single managed offering: it consumes CloudTrail, VPC Flow Logs and DNS, and sends findings to EventBridge or Security Hub. The typical integration is Detective coming in for investigation (entity graph + timeline), Security Hub aggregating with Prowler/Inspector, and the SOAR (Step Functions, Tines, Torq) firing a runbook. The trick is to keep rules out of the SIEM when the SIEM aggregates poorly, and use GuardDuty for the 80% of obvious findings.

The SIEM (OpenSearch, Splunk, Elastic, Datadog Cloud SIEM) is where correlation happens. Each source (CloudTrail, GuardDuty, flow logs, k8s audit, EDR) becomes an indexed stream, with its own parser and canonical fields (time, actor, target, action, result). The SIEM rule crosses sources: new IAM AssumeRole + CloudTrail GetSecretValue + public bucket touched in <5 min = suspicious session.

To reduce cost and noise, two habits pay off. First, storage tiers: hot in the SIEM (30 days), cold in S3 (Iceberg, Parquet) with Athena or OpenSearch Serverless querying on demand. Second, managed detection for what you have no team to maintain: GuardDuty covers 80% of obvious findings, CSPM covers configuration drift, CWPP covers runtime. The SIEM is left for what is house-specific.