Coldstarts with the AWS Javascript 3.502 SDK

Drift car without doors

In November 2023, I noticed Lambda coldstarts were 40 ms - 300 ms slower because they unnecessarily loaded the SSO credentials provider. I cut a GitHub issue which was closed with wontfix. Fast forward to January 29, version 3.502 of the AWS JavaScript SDK was released to lazy load non-essential credentials providers. This is a big win, but you can still squeeze out more performance. In this post I'll show you how to make your coldstarts 35 ms faster.

The backstory

When I was wallowing in my esbuild metadata, I noticed the AWS JavaScript SDK was loading a credentials provider to get AWS credentials from SSO. WAT!? Lambda always injects the execution role credentials via environment variables, why does it need anything but the env credentials provider? I patched the default credentials provider to only use the env credentials provider and the coldstarts were 40 - 300 ms faster. Gold Jerry, gold! It was the data I needed to get AWS to act. I opened this issue #5516 and provided a repro so they could see it themselves. AWS acknowledged the issue but unceremoniously closed it with the following:

Currently there is no way / plan to fix this. Thank you for taking the time to raise the feature request, and I hope that this does not discourage you from engaging with us in the future.

Tonight we ride

Sometimes I gauge the room and decide it's time to ride. This was one of those cases. I told a few people privately "don't worry, I'll handle it". At re:Invent I spoke to 2 Lambda principals. I also followed up on the issue with some suggestions like lazy loading for how it could be fixed. Normally I'd submit a pull request, but after looking at the code, I decided that there were enough moving parts the best approach was to leave it to AWS and wait. After a month, I posted this spicy blog which highlighted this issue and others. After some silence, this PR appeared to lazy load credential providers and 3.499 was released. Unfortunately, I discovered the SSO bits were still being loaded causing the latency hit. I commented as such and it was fast-followed with this PR which fixed the ini file credentials provider that loaded SSO¹. We did it comrades! 3.502 is out and it's faster than ever!

Improving on 3.502

Note

3.502 has not made it into the Lambda runtime yet. If you are relying on the version bundled with the runtime, you'll have to wait to see the coldstart speedup. Watch this page for updates. At the time of this writing Lambda is using 3.362.

If you do one thing, upgrade your dependencies to at least 3.502. You'll see a ~300 ms reduction in coldstarts if you aren't bundling, and 5-10ms if you are. If you are bundling, you can remove the unused modules to shave off another 30-35 ms and about 92KB in bundle size.

Before removing unused modules

After removing unused modules

Marking unused packages as external

Since the AWS SDK is lazy loading the other credential providers, node.js doesn't know until runtime whether it needs them. This means your bundler will include them in your bundle. I use esbuild and I found I couldn't get splitting to work with the CDK so the way I omit parsing unnecessary credentials providers is mark them as external.

externalModules: [
   '@aws-sdk/client-sso',
   '@aws-sdk/client-sso-oidc',
   '@aws-sdk/credential-provider-ini',
   '@aws-sdk/credential-provider-process',
   '@aws-sdk/credential-provider-sso',
   '@aws-sdk/credential-provider-web-identity',
   '@aws-sdk/token-providers',
]

Here is the full esbuild config I use for reference:

bundling: {
  metafile: false, //set to true to create the necessary metafile for https://esbuild.github.io/analyze/ to analyze the bundle
  minify: true,
  treeshake: true,
  target: 'esnext',
  format: 'esm',
  platform: 'node',
  mainFields: ['module', 'main'],
  outputFileExtension: '.mjs',
  externalModules: [
    '@aws-sdk/client-sso',
    '@aws-sdk/client-sso-oidc',
    '@aws-sdk/credential-provider-ini',
    '@aws-sdk/credential-provider-process',
    '@aws-sdk/credential-provider-sso',
    '@aws-sdk/credential-provider-web-identity',
    '@aws-sdk/token-providers'
  ],
  banner:
    "const require = (await import('node:module')).createRequire(import.meta.url);const __filename = (await import('node:url')).fileURLToPath(import.meta.url);const __dirname = (await import('node:path')).dirname(__filename);",
}

I identified the unnecessary packages by patching the SDK to only use the env credentials provider using the approach below. I then reviewed what modules were tree-shaken out.

Patching the AWS SDK

If you want tree-shaking to remove the unneeded packages, you can patch the SDK to only use the env credentials provider. It is not for the feint of heart, it requires a bit of finesse and will require ongoing maintenance as the SDK evolves. If you are not using esm modules, the filename you patch may be the cjs analog.

Follow the steps here to install patch-package. Including the step to add the postinstall script to your package.json.

Modify the file node_modules/@aws-sdk/credential-provider-node/dist-es/defaultProvider.js to only use the environment credentials provider. Here is what I set it to for 3.504:

 import { chain, CredentialsProviderError, memoize } from "@smithy/property-provider";
 export const defaultProvider = (init = {}) => memoize(chain(async () => {
         init.logger?.debug("@aws-sdk/credential-provider-node", "defaultProvider::fromEnv");
         const { fromEnv } = await import("@aws-sdk/credential-provider-env");
         return fromEnv(init)();
     },
     async () => {
         throw new CredentialsProviderError("Could not load credentials from any providers", false);
 }), credentialsTreatedAsExpired, credentialsWillNeedRefresh);
 export const credentialsWillNeedRefresh = (credentials) => credentials?.expiration !== undefined;
 export const credentialsTreatedAsExpired = (credentials) => credentials?.expiration !== undefined && credentials.expiration.getTime() - Date.now() < 300000;

Run npx patch-package @aws-sdk/credential-provider-node to create a patch file.
Make sure you check in the patches directory to your repository
If you setup the postinstall script correctly it will run patch-package after every npm install and apply this patch. It will display an error if it can't apply the patch. If that happens, create a new patch by modifying node_modules/@aws-sdk/credential-provider-node/dist-es/defaultProvider.js, repeat steps 2-4 and delete the old patch file. You may need to routinely update the SDK version number in the patch filename if the SDK changes but the default provider doesn't.

An interesting note about debugging what is being loaded

While I was doing this work, I found a useful snippet to see what is actually being loaded by lambda. If you don't bundle, you can use this in your code to see what has been loaded.

Object.keys(require.cache).forEach((key) => console.log(key));

Conclusion

3.502 of the JavaScript SDK improves coldstart performance. If you want the absolute best performance and smallest bundle size, either mark the unnecessary packages as external or patch the SDK to only use the env credentials provider so they tree-shake out.