AWS – Secrets Manager and Parameter Store

Two AWS services store secrets, they overlap heavily, and one of them is free. Picking correctly is worth five minutes because the difference on a project with fifty secrets is real money for capability most projects never use.

Secrets Manager or Parameter Store

Default to Parameter Store. A SecureString parameter is KMS -encrypted, IAM-controlled, versioned, and costs nothing. For an API key, a database URL or a JWT signing secret that you rotate by hand once a year, that is the whole requirement.

Choose Secrets Manager when you want automatic rotation — particularly for RDS credentials, where it is genuinely turnkey — or cross-account access without the advanced tier.

This site uses Parameter Store for both its secrets, which is why its bill is what it is.

Writing and reading

aws ssm put-parameter \
  --name /lovemesomecoding/prod/jwt-secret \
  --type SecureString \
  --value "$(openssl rand -base64 32)" \
  --region us-west-2 --profile folau

aws ssm get-parameter \
  --name /lovemesomecoding/prod/jwt-secret \
  --with-decryption --query "Parameter.Value" --output text

--with-decryption is required and easy to forget: without it you get the ciphertext back, which then fails somewhere far away from the cause.

Use a path hierarchy — /app/environment/name. It lets you fetch a whole environment in one call and, more usefully, scope IAM by prefix so a staging role cannot read production:

aws ssm get-parameters-by-path \
  --path /lovemesomecoding/prod/ --recursive --with-decryption

Do not put secrets in environment variables

Lambda environment variables are convenient and visible to anyone with lambda:GetFunctionConfiguration — which is a read-only permission people hand out freely. They appear in the console, in CloudFormation, in `describe` output, and often in CI logs. They are also encrypted at rest but decrypted into the function's environment, so they are not secret from anything that can read the configuration.

Fetch at runtime instead, and cache in the execution context so you are not billed for a KMS call on every invocation:

import boto3

ssm = boto3.client("ssm")
_cache = {}          # module scope: survives between invocations


def secret(name):
    if name not in _cache:
        _cache[name] = ssm.get_parameter(
            Name=name, WithDecryption=True
        )["Parameter"]["Value"]
    return _cache[name]

That cache is the difference between one KMS call per container and one per request. Note it never expires, so a container alive for hours holds an old value after a rotation — add a TTL if you rotate on a schedule, or let the deployment replace the containers.

The better version for Lambda is the AWS Parameters and Secrets extension, a layer that runs a local caching HTTP endpoint so your code makes no SDK call at all.

The IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:GetParameter",
      "Resource": "arn:aws:ssm:us-west-2:111122223333:parameter/lovemesomecoding/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-west-2:111122223333:key/abcd1234-..."
    }
  ]
}

Both statements are needed and the second is the one people miss. A SecureString is encrypted with KMS, so reading it requires kms:Decrypt on the key as well as the SSM permission — and the failure says access denied without saying which of the two is missing.

Note the parameter ARN has no extra slash: the path /lovemesomecoding/prod/x becomes ...:parameter/lovemesomecoding/prod/x.

Rotation

Secrets Manager rotation runs a Lambda on a schedule, moving the AWSCURRENT label to a new version. For RDS the function is provided; for anything else you write it.

aws secretsmanager rotate-secret \
  --secret-id stayhub/db \
  --rotation-rules "AutomaticallyAfterDays=30"

The part that makes rotation safe is that both the old and new versions are valid during the changeover — AWSPREVIOUS still resolves — so a cached credential does not fail instantly. Any code caching a secret must be able to re-fetch on an auth error rather than crash, or rotation becomes an outage on a schedule.

Getting secrets into a container or an instance

Neither service is only for Lambda, and the integrations are worth knowing because they remove the code above entirely.

ECS takes a secrets block in the task definition naming a parameter or secret ARN. The agent fetches it before your container starts and injects it as an environment variable — so the value is never in the task definition, only the ARN is. Note this needs the execution role to have the permission, not the task role, which is a common half-hour of confusion.

EKS uses the Secrets Store CSI driver, which mounts secrets as files in the pod. Files are usually better than environment variables here, because they can be updated without restarting the pod.

EC2 has no equivalent; fetch at boot in user data, or at runtime in the application. Either way the instance profile supplies the credentials and nothing is stored on disk.

If a secret leaks

Rotate it, do not delete it — deleting breaks every caller at once and buys nothing, because the leaked value is already out. Then check CloudTrail for reads you did not expect, and remember that a secret committed to git is still in the history after the file is removed — the value must be considered public from the moment it was pushed, regardless of what you later do to the repository history.