Migrating from Serverless Framework to CDK

Migrating elephants

Last year I migrated the Speedrun API from the Serverless Framework V3 to the Cloud Development Kit (CDK). At the same time, I migrated from API Gateway to CloudFront. In this post I'll describe the process, provide a simple application for you to migrate to practice with and share some thoughts about Serverless V3 vs. CDK.

Why Did I Migrate?

There wasn't a main driver for this migration, just several minor ones. In 2023, version 4.0 of the Serverless Framework was released with new licensing. I wasn't impacted by the changes (I'm $2 million below the $2 million earnings threshold) but most of my professional experience is with the CDK. I used the Serverless Framework because I was using lambda-api and the getting started article used it. Serverless had a few minor annoyances like requiring a patch to use local invoke on .cjs files and the aliases plugin I used was no longer maintained, but as a whole, Serverless worked fine. In 2023 I asked a former Serverless Inc developer if I should migrate. It was at a night club in Vegas and amusingly, it was too loud to hear their response so I tabled it. Last year, I revisited the question after reading Brett Andrew's article about switching from API Gateway to CloudFront w/Lambda Function URLs. I decided if I was migrating to CloudFront, I might as well migrate to the CDK at the same time and make it interesting. Plus, I needed to confront my fear of using CloudFormation import.

Migration Steps Overview

Migration from Serverless Framework V3 to the CDK is manual and iterative. First, you must prevent resources from being deleted when they are removed from Serverless. Then for each resource, you recreate the resource in the CDK, remove it from Serverless and import the orphaned resource into the CDK.

Warning

If you don't have a development/staging environment, create one and perform the migration there first. Do not attempt this in production until you've practiced. As you'll see from the number of steps below, you can't just wing it.

Preventing resource deletion

By default, resources are deleted when you remove them. To preserve them so they can be imported into the CDK add a DeletionPolicy.

Add DeletionPolicy: Retain to all resources in serverless.yml.
Deploy your stack with serverless deploy

Tip

Make sure you add the DeletionPolicy at the correct indent level, it should be at the same indent level as the Type or Properties key. Review the CloudFormation template in .serverless/cloudformation-template-update-stack.json to verify you have set it on all resources you want to import, some resources are created as part of an abstraction and may not be visible in the serverless.yml file.

Recreating each resource using the CDK

Repeat these steps for each resource you want to migrate.

Recreate the resource in the CDK
Compare CloudFormation outputs between cdk synth and serverless package and fix any differences
Remove the resource from serverless.yml and deploy your stack with serverless deploy
Run cdk import to import the orphaned resource
Deploy your stack with cdk deploy
(Optionally) Check for drift using CloudFormation

The Sample Application

We will be building a simple Lambda hit counter. When it receives a request, it increments a counter in a DynamoDB table and returns the current count. We could use a Lambda Function URL to front it, but to simulate my migration process, we'll use API Gateway to front it with Serverless and CloudFront to front it when we move it to the CDK.

The application is for showing you the techniques to migrate from the Serverless Framework to the CDK. It doesn't go into any depth about building Serverless or CDK applications.

Here's an overview of the application architecture before and after the migration

Sample Application Migration Overview

Although we are migrating the stateful resources like the DynamoDB table and LogGroup, we are recreating the stateless resources like Lambda with new names. This allows us to have two instances of our API to work through any differences between the API Gateway and CloudFront event structures and verify it is still using the same DynamoDB table and LogGroup before we cut over.

Note

For this sample, we skip cutting over. In an environment where you have your API behind a custom domain, the cutover requires you to modify the Route53 alias to point at the new CloudFront distribution to complete the migration and doesn't require downtime.

Deploying the Serverless Application

Start by cloning the sample application to your local machine.

git clone https://github.com/perpil/sls-cdk-migration.git

Change into the directory and deploy the Serverless application

cd sls-cdk-migration
npm install
npx serverless deploy

On deployment, the Serverless application will output the API Gateway URL. Hit it a few times in your browser to increment the hit counter.

# The output will look something like this
endpoint: ANY - https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/dev/count/{proxy+}

Setting Deletion Policy

Next we will make it so the resources aren't deleted when we remove them from Serverless. We want to preserve our CloudWatch LogGroup and DynamoDB Table so we don't lose our logs or our current count.

Open serverless.yml and uncomment lines 24-27 and 29.

serverless.yml
  Resources:
    CounterLogGroup:
      Type: AWS::Logs::LogGroup
      DeletionPolicy: Retain
    counterTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain

Save the file and package it so you can see the CloudFormation output

npx serverless package

Review the CloudFormation output in the file .serverless/cloudformation-template-update-stack.json and make sure there is a DeletionPolicy: Retain for the LogGroup and DynamoDB Table.

Now deploy the stack again

npx serverless deploy

Creating the DynamoDB Table in the CDK

In a new directory clone the CDK application to your local machine.

git clone https://github.com/perpil/cdk-migration.git
cd cdk-migration
npm install

Open the file lib/cdk-migration-stack.mjs

I've already added the DynamoDB Table here. It should contain the following code:

const counterDDB = new Table(this, "CounterTable", {
  tableName: 'migration-sandbox-table',
  partitionKey: { name: "key", type: AttributeType.STRING },
  billingMode: BillingMode.PAY_PER_REQUEST
});

Now run npx cdk synth to generate the CloudFormation template

Compare the CloudFormation output between Serverless and the CDK. The Serverless Cloudformation template is in .serverless/cloudformation-template-update-stack.json and the CDK CloudFormation template is in cdk.out/CDKMigrationStack.template.json.

Although the structure might be ordered differently, the Properties like TableName and KeySchema should be the same. Below are the corresponding snippets from my templates:

ServerlessCDK

.serverless/cloudformation-template-update-stack.json
"counterTable": {
    "Type": "AWS::DynamoDB::Table",
    "DeletionPolicy": "Retain",
    "Properties": {
      "TableName": "migration-sandbox-table",
      "KeySchema": [
        {
          "AttributeName": "key",
          "KeyType": "HASH"
        }
      ],
      "AttributeDefinitions": [
        {
          "AttributeName": "key",
          "AttributeType": "S"
        }
      ],
      "BillingMode": "PAY_PER_REQUEST"
    }
  }
},

cdk.out/CDKMigrationStack.template.json
"CounterTableFE2C0268": {
 "Type": "AWS::DynamoDB::Table",
 "Properties": {
  "AttributeDefinitions": [
   {
    "AttributeName": "key",
    "AttributeType": "S"
   }
  ],
  "BillingMode": "PAY_PER_REQUEST",
  "KeySchema": [
   {
    "AttributeName": "key",
    "KeyType": "HASH"
   }
  ],
  "TableName": "migration-sandbox-table"
 },
 "UpdateReplacePolicy": "Retain",
 "DeletionPolicy": "Retain",
 "Metadata": {
  "aws:cdk:path": "CdkMigrationStack/CounterTable/Resource"
 }
},

Removing the DynamoDB Table from Serverless

Now we need to remove the DynamoDB table from Serverless so we can import it into the CDK. Each resource is only allowed to be in one CloudFormation stack at a time. Remove or comment out lines 27-38 in the serverless.yml file and deploy it.

serverless.yml
# counterTable:
#   Type: AWS::DynamoDB::Table
#   DeletionPolicy: Retain
#   Properties:
#     TableName: ${self:service}-table
#     KeySchema:
#       - AttributeName: key
#         KeyType: HASH
#     AttributeDefinitions:
#       - AttributeName: key
#         AttributeType: S
#     BillingMode: PAY_PER_REQUEST

npx serverless deploy

Importing the DynamoDB Table into the CDK

Now you can import the DynamoDB Table into the CDK. From your CDK application directory run the following command

npx cdk import

At the prompt, hit enter to run the import.

CdkMigrationStack/CounterTable/Resource (AWS::DynamoDB::Table): import with TableName=migration-sandbox-table (yes/no) [default: yes]?

Now your DynamoDB Table is imported into the CDK.

Creating the LogGroup in the CDK

You'll need to add the code to create the LogGroup to lib/cdk-migration-stack.mjs file. For simplicity, there is a file with the necessary code included. Copy the content of the file lib/cdk-migration-stack.mjs.2 to lib/cdk-migration-stack.mjs. For reference it will add code like the following:

// add to imports
import { LogGroup } from '@aws-cdk/aws-logs';

// add below creation of the DynamoDB table
const logGroup = new LogGroup(this, "CdkMigrationLogGroup", {
  name: '/aws/lambda/migration-sandbox-counter'
});

Now run npx cdk synth to generate the CloudFormation template

Compare the CloudFormation output between Serverless and the CDK. The Serverless Cloudformation template is in .serverless/cloudformation-template-update-stack.json and the CDK CloudFormation template is in cdk.out/CDKMigrationStack.template.json.

Although the structure might be ordered differently, the Properties like LogGroupName should be the same. Additions like RetentionInDays are ok and will be resolved when you deploy the CDK stack again. Below are the corresponding snippets from my templates:

ServerlessCDK

.serverless/cloudformation-template-update-stack.json
"CounterLogGroup": {
  "Type": "AWS::Logs::LogGroup",
  "Properties": {
    "LogGroupName": "/aws/lambda/migration-sandbox-counter"
  },
  "DeletionPolicy": "Retain"
},

cdk.out/CDKMigrationStack.template.json
"CdkMigrationLogGroupC3CD39BD": {
  "Type": "AWS::Logs::LogGroup",
  "Properties": {
    "LogGroupName": "/aws/lambda/migration-sandbox-counter",
    "RetentionInDays": 731
  },
  "UpdateReplacePolicy": "Retain",
  "DeletionPolicy": "Retain",
  "Metadata": {
    "aws:cdk:path": "CdkMigrationStack/CdkMigrationLogGroup/Resource"
  }
},

Removing the LogGroup from Serverless

Normally you would remove the LogGroup from the serverless.yml file, like we did with the Dynamo table, but there is no such resource. It is abstracted away and automatically created when you create a Lambda Function. To remove it, you need to do surgery on the CloudFormation template itself. Open the .serverless/cloudformation-template-update-stack.json file and remove the CounterLogGroup resource.

This is the code you will be removing.

.serverless/cloudformation-template-update-stack.json
"CounterLogGroup": {
  "Type": "AWS::Logs::LogGroup",
  "Properties": {
    "LogGroupName": "/aws/lambda/migration-sandbox-counter"
  },
  "DeletionPolicy": "Retain"
},

You also need to remove the DependsOn on the Lambda function. Note that , that you are removing, it is important.

.serverless/cloudformation-template-update-stack.json
,
"DependsOn": [
 "CounterLogGroup"
]

Now in the CloudFormation console, find the stack called migration-sandbox-dev and click the Update button.

Update Stack

Click Replace existing template and upload the modified cloudformation-template-update-stack.json using Upload a template file and Choose file.

Replace Template

Click Next to Upload

Click Next again for Specify stack details

Under Configure Stack Options Click the I acknowledge that AWS CloudFormation might create IAM resources with custom names checkbox to select it and click Next again.

Review the changes and click Submit

Tip

The only change you should see is the removal of the LogGroup resource. If you see other changes, it's possible you ran npx serverless package since your last deployment. The update will fail because it will reference a S3 Location for the lambda code based on the timestamp of the last packaging operation and that was never uploaded to S3. If this happens force a deployment with npx serverless deploy --force to force a deployment so the template is in sync with what is deployed and repeat the steps above to remove the LogGroup.

Importing the LogGroup into the CDK

Now you can import the LogGroup into the CDK. From your CDK application directory run the following command

npx cdk import

At the prompt, hit enter.

CdkMigrationStack/CdkMigrationLogGroup/Resource (AWS::Logs::LogGroup): import with LogGroupName=/aws/lambda/migration-sandbox-counter (yes/no) [default: yes]?

Creating the API in the CDK

Whew, we're finally ready to create the API with the CDK. There is a lot of code to add, so copy the contents of lib/cdk-migration-stack.mjs.3 to lib/cdk-migration-stack.mjs. This will add several resources including the Lambda Function, Lambda Execution Role, Function URL, Lambda Alias, Lambda Version and CloudFront Distribution.

The code looks like this:

lib/cdk-migration-stack.mjs

// replace imports with
import { Stack, Duration, CfnOutput } from 'aws-cdk-lib';
import { Table, AttributeType, BillingMode } from 'aws-cdk-lib/aws-dynamodb';
import { NodejsFunction, OutputFormat } from "aws-cdk-lib/aws-lambda-nodejs";
import { Alias, FunctionUrlAuthType, Runtime, Architecture } from 'aws-cdk-lib/aws-lambda';
import { LogGroup } from 'aws-cdk-lib/aws-logs';
import { Role, ServicePrincipal, ManagedPolicy, PolicyStatement, Effect } from 'aws-cdk-lib/aws-iam';
import { readFileSync } from 'fs';
import { Distribution, PriceClass, AllowedMethods, OriginRequestPolicy, CachePolicy } from 'aws-cdk-lib/aws-cloudfront';
import { FunctionUrlOrigin } from 'aws-cdk-lib/aws-cloudfront-origins';

// add below the LogGroup
const lambdaRole = new Role(this, "CdkMigrationLambdaRole", {
  roleName: `migration-sandbox-cdk-lambda-role`,
  assumedBy: new ServicePrincipal("lambda.amazonaws.com"),
  managedPolicies: [
    ManagedPolicy.fromAwsManagedPolicyName(
      "service-role/AWSLambdaBasicExecutionRole"
    ),
  ],
});

lambdaRole.addToPolicy(
  new PolicyStatement({
    actions: [
      "dynamodb:UpdateItem",
    ],
    effect: Effect.ALLOW,
    resources: [counterDDB.tableArn],
  })
);

const handler = new NodejsFunction(this, "CdkMigrationLambda", {
  entry: "src/handler.mjs",
  handler: "index.router",
  logGroup,
  role: lambdaRole,
  runtime: Runtime.NODEJS_20_X,
  architecture: Architecture.ARM_64,
  memorySize: 1024,
  timeout: Duration.seconds(6),
  awsSdkConnectionReuse: false,
  functionName: `migration-sandbox-counter-cdk-dev`,
  description: "A lambda that counts its invocations",
  bundling: {
    format: OutputFormat.ESM,
    mainFields: ["module", "main"],
    bundleAwsSDK: true,
    // update to true and run npx cdk synth if you want to analyze the bundle with https://esbuild.github.io/analyze/
    metafile: false,
    minify: false,
    sourceMap: false,
    keepNames: true,
    // rip out non-essential credential provider stuff
    externalModules: [
      "@aws-sdk/client-s3",
      "@aws-sdk/s3-request-presigner",
      "@smithy/credential-provider-imds",
      "@aws-sdk/client-sso",
      "@aws-sdk/client-sso-oidc",
      "@aws-sdk/credential-provider-ini",
      "@aws-sdk/credential-provider-process",
      "@aws-sdk/credential-provider-sso",
      "@aws-sdk/credential-provider-web-identity",
      "@aws-sdk/token-providers"
    ],
    define: {
      "process.env.sdkVersion": JSON.stringify(
        JSON.parse(
          readFileSync(
            "node_modules/@aws-sdk/client-dynamodb/package.json"
          ).toString()
        ).version
      ),
    },
    banner:
      "const require = (await import('node:module')).createRequire(import.meta.url);const __filename = (await import('node:url')).fileURLToPath(import.meta.url);const __dirname = (await import('node:path')).dirname(__filename);",
  },
});

// add an alias to the lambda function
const alias = new Alias(this, "FunctionAlias", {
  aliasName: "dev",
  version: handler.currentVersion,
});

const url = alias.addFunctionUrl({
  authType: FunctionUrlAuthType.AWS_IAM,
});

const distribution = new Distribution(this, "Distribution", {
  defaultBehavior: {
    origin: FunctionUrlOrigin.withOriginAccessControl(url),
    allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
    cachePolicy: CachePolicy.CACHING_DISABLED,
    originRequestPolicy:  OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
  },
  comment: "CDK Migration Distribution",
  priceClass: PriceClass.PRICE_CLASS_100,
});

new CfnOutput(this, "CloudFrontUrl", {
  value: `https://${distribution.distributionDomainName}/dev/count`,
  description: "The URL of the CloudFront Distribution",
});
}

Tip

The distribution here is very simplistic and doesn't have a certificate or custom domain. In a real-world scenario, you might prefer to not hit your Lambda origin directly and connect it to a behavior behind a path so crawlers or requests to get the favicon don't trigger extra invocations.

Run npx cdk deploy to deploy the CDK stack. When prompted to approve the IAM related changes enter y and hit enter.

Testing the CDK Application

Hit the cloudfront URL in the output in your browser, it should continue to increment the counter because it is reusing the same table.

Outputs:
CdkMigrationStack.CloudFrontUrl = https://d2xxxxxxxxxxxxx.cloudfront.net/dev/count

Using the CloudWatch console, look at the logs for /aws/lambda/migration-sandbox-counter you should see logs for both the CDK and Serverless invocations. Compare them and make sure they aren't missing fields.

You can even hit the API Gateway URL to confirm that it is still working and it is sharing the same counter.

Deleting the Serverless Application

Warning

In a production environment you would want to point your Route53 alias to the CloudFront distribution, test it and wait for traffic to stop hitting your API Gateway before deleting the Serverless stack.

Once you are confident that the CDK application is working, you can run

npx serverless remove

Deleting the CDK Application

Once you are done playing with the sample you can tear it down with

npx cdk destroy

Since the DeletePolicy is set to Retain on the DynamoDB table and LogGroup, you will need to manually delete those two resources in the AWS Console.

Some thoughts on Serverless vs. CDK

Now that the migration is over, I have some thoughts on the two frameworks. I really like the simplicity of the Serverless Framework. The resources it creates are named for humans, it has builtin support for environment specific variables and it doesn't deploy weird one-off stacks into your account for functionality. The CDK requires more coding to build the same application, generates resources with names that are hard to remember but is backed by AWS and has a larger community of samples and blog articles. Certain things with the CDK like creating certificates in us-east-1 are a total PITA if you are in a different region. For now, I've moved all of my stacks to the CDK for peace of mind and because I have more experience in it. I have by no means given up on Serverless however and will still follow its progress.

Conclusion

Migrating from the Serverless Framework to the CDK is manual, a lot of work and an iterative process. If you were hesitant because you didn't know it was possible, now you know it can be done. I hope this article has given you the confidence and practice to migrate your own app if you need to. If you have any questions or get stuck feel free to reach out to me on BlueSky or open an issue on one of the repositories.