Using DynamoDB as a Secrets Manager

A wizard keeping a secret

What if I told you you could build a serverless secrets manager with a similar security footprint to AWS Secrets Manager, but with a fraction of the cost, better scalability and better latency? With the release of DynamoDB table resource policies in March 2024, this became much easier. In this post, I describe how I reduced the cost of storing secrets to ~zero and improved the latency to get secrets by almost 50% by switching to DynamoDB.

The Background

AWS services are built using core primitives like DynamoDB and S3. My hunch is that AWS Secrets Manager is built on top of DynamoDB and KMS. If so, I can skip the cost and additional latency overhead by using DynamoDB directly. As an added benefit, if I load my secrets from DynamoDB during my Lambda function's initialization I have a warm SSL connection for querying my DynamoDB datastore (which I need for every request anyway). So it's cheaper, my coldstarts are faster, my API is faster and I won't need the Secrets Manager SDK in my bundle. Let's do this!

DynamoDB Table Resource Policies

With DynamoDB Table Resource Policies you can attach IAM policies directly to a DynamoDB table to restrict access (even when callers have the necessary permissions to perform the action). My specific use case is to restrict secrets access to the Lambda functions they belong to and to prevent any humans from reading them.

Protecting You From Yourself

Keeping secrets a secret is hard. The best way to protect them is to never persist secrets and to use scoped timebound temporary credentials. But that isn't always possible, some secrets like a GitHub App Secret are longer lived and need to be stored somewhere. It doesn't matter how secure your secrets storage is, if you have read access, it's easier to undermine your secrets. You might inadvertently check secrets into source control, set them as environment variables on an unlocked laptop, accidentally paste them in a chat or flash them on a livestream. The secrets reaper waits patiently until you slip. Making it so only machines can read secrets, makes it much harder to slip.

Exhibit A: A cautionary tale from the author of the ultimate guide to serverless secrets

HUGE shoutout to @Synology and everyone in chat who helped me out today!

In one stream we:
✅ Set up the hardware
✅ Leaked AWS access keys 3 times
✅ Dockerized our application
✅ Started collecting metric data

Now it's time to let the data roll in! pic.twitter.com/2AmoqQWPCD
— AJ Stuyvenberg (@astuyve) June 8, 2024

The Necessary Resource Policy

It's actually fairly simple to block read access to secrets for humans. Assuming you've stored your secrets in a table called dirty-secrets with the partition key set to the role arn of your lambda role and the sort key with your secret name like this:

Partition Key (principal)	Sort Key (name)	Value (value)
arn:aws:iam::22222222222:role/github-webhook-processor-lambda-role	githubAppSecret	ghsecret-value
arn:aws:iam::22222222222:role/slack-bot-lambda-role	webhookUrl	https://hooks.slack.com/services/...
arn:aws:iam::22222222222:role/slack-bot-lambda-role	slackWebhookSecret	secret-value-2

You can use the following resource policy to deny read access to all principals except the lambda role you want to grant access.

{
  Version: "2012-10-17",
  Statement: [
    {
      Sid: "DenyAllButLambdaRole",
      Effect: "Deny",
      Principal: "*",
      Action: [
        "dynamodb:GetItem",
        "dynamodb:Query",
        "dynamodb:BatchGetItem",
        "dynamodb:BatchTransactItem",
      ],
      Resource: [
        "arn:aws:dynamodb:us-west-2:22222222222:table/dirty-secrets",
        "arn:aws:dynamodb:us-west-2:22222222222:table/dirty-secrets/*",
      ],
      Condition: {
        ArnNotEquals: {
          "aws:PrincipalArn": "arn:aws:iam::22222222222:role/slack-bot-lambda-role",
        },
      },
    },
    {
      Sid: "DenyAllScans",
      Effect: "Deny",
      Principal: "*",
      Action: [
        "dynamodb:Scan",
      ],
      Resource: [
        "arn:aws:dynamodb:us-west-2:22222222222:table/dirty-secrets",
        "arn:aws:dynamodb:us-west-2:22222222222:table/dirty-secrets/*",
      ]
    }
  ],
}

Now it doesn't matter what principal you are operating as, you can never read, but if you have PutItem permissions, you can still write. This means you can even use the AWS console to initialize the secrets if you want, but it will never show them.

Tip

The dynamodb:LeadingKeys condition key can be used to automatically limit secret access to only the calling principal arn. This gives you a set and forget config. If you have a different role per lambda, no matter what the role permissions are, the lambda can only read the secrets associated with its role at a maximum.

Condition: {
  "ForAllValues:StringNotEquals": {
    "dynamodb:LeadingKeys": ["${aws:PrincipalArn}"],
  },
}

The sample repo below uses exactly this method.

Try It At Home

I've prepared the ddb-secrets-manager repository to create a full working example using the CDK. It creates a DynamoDB secrets table with a resource policy that limits read access to a specific lambda function that you can interact with. Read the README for instructions on how to deploy it, how to populate the secret and verify only your lambda can read the secret.

Benchmarks

Prior to switching to DynamoDB to load secrets, I was using AWS Secrets Manager's BatchGetSecretValue API to load 3 secrets. When I switched to DynamoDB I used BatchGetItem instead. Here is the benchmark for over 100 runs in a Lambda coldstart. All times are in milliseconds.

source	min	p50	p99	max
Secrets Manager	185	194	235	239
DynamoDB	117	132	168	168

That's 62 ms faster on the p50 and 67 ms faster on the p99.

Cost

For 3 secrets, Secrets Manager cost me $1.20 per month and $.05/10000 requests. The same in DynamoDB falls in the free tier and was free. Per request, it is $.05/200000 or 5% of the cost. Storage cost is negligible. If instead of using the DynamoDB managed KMS key, I used a customer managed key, the cost would be driven by KMS. For 3 secrets it would be $1.00 a month for the key and $0.03 per 10000 requests. This is still cheaper than Secrets Manager and the key cost doesn't increase with the number of secrets.

Limitations

There are few features that are not supported with DynamoDB or require extra work to implement, keep these in mind when considering this approach and how it may impact your compliance requirements.

No secret versioning
No automatic rotation (Step Functions can be used to implement this)
No per secret resource policy (only per table)
Cloudtrail logging requires you to enable data plane logging
Cross-region replication (potentially implemented with DynamoDB Global Tables)

Conclusion

By switching to DynamoDB, I was able to reduce the latency and cost dimensions of storing secrets while still maintaining a high level of security. I now have my coldstarts down to 375 ms from the 900 ms range last year. I've also reduced my monthly cost by about $5.00 by migrating all of my environments. I hope this post gives you ideas for how to make your apis faster, and cheaper while keeping your secrets secure. If you have questions or issues with the sample code, reach out on Twitter or open an issue on the GitHub repository.