Secrets - vaults and rotation

A secret in the cloud is anything that grants access: an API key, a database password, a service token, a private certificate. The critical point is not storing the value, it is controlling who can read it, at what moment, and with what trail.

Three practices repeat in any mature architecture: the secret lives in a vault (Vault, Secrets Manager, Parameter Store), the application reads it at runtime (not at build), and the lifetime is short. A credential that lasts a year can be revoked up to 365 days after the leak, and that is where the real risk lives.

The same plot shows up every week: someone commits an AWS Access Key to a public repository or a chat. Miner bots sweep GitHub in minutes, find the key, check whether it still works and start mining crypto on the victim's reserved instances. The bill arrives before the alert.

The detail that kills: the key was in an environment variable, so the team thought they had not 'committed a secret'. But the .env.example with a placeholder, the screenshot of the cloud console, the CI log with the value printed... each of these leaks. Scanning for 'AWS_ACCESS_KEY_ID' in the repository before merge is basic hygiene that almost nobody does.

The old way is static: create an IAM user for the application, generate Access Key + Secret, paste into .env and run. The secret stays valid until someone revokes it, and each new deploy multiplies who holds a copy.

The modern way is workload identity. On AWS it is called IRSA (IAM Roles for Service Accounts), on GCP it is Workload Identity, on Azure it is Workload Identity Federation. The idea is the same: the cluster (EKS, GKE, AKS) hands a short-lived token to the pod, and the cloud exchanges that token for temporary credentials via STS/OIDC. There is no static secret to leak.

When the workload cannot assume a federated identity (common in older services or third-party SaaS), the alternative is to fetch the secret from the vault at application startup, mount it in memory and never persist it to disk.

The difference between fetching at runtime and copying at build is the difference between the secret existing in a Docker image (which someone can open) and existing only on the heap of the running process. The second scenario can still leak via log or dump, but it is orders of magnitude harder.

Rotation is the exercise of replacing a secret before it expires on its own or before someone has compromised it. Modern vaults treat this as a first-class citizen: Secrets Manager has automatic rotation via Lambda, Vault has dynamic secrets that expire in minutes, Parameter Store supports event-driven rotation.

The practical rule: if you cannot rotate the key without downtime, you cannot revoke the key under pressure. And under pressure is when revocation needs to happen.