Blocking Requests with Lambda Function URLs

Steampunk heroes crossing the bridge, but villains plunging into the abyss behind them

If you use Lambda Function URLs to front your serverless API, you will receive traffic from crawlers, scrapers, bots and vulnerability sniffers that generates unnecessary errors. In this article, I'll cover how I use CloudFront and CloudFront Functions to block that traffic at the edge so it never reaches the Lambda metrics and logs.

What kind of requests are we blocking?

Lambda Function URLs are directly exposed to the internet. Web crawlers and bots eventually find your endpoint, send it invalid requests and poke around for unpatched vulnerabilities. Although common router frameworks like Hono will reject these requests, they still cause Lambda invocations and pollute your Lambda metrics and logs with unnecessary errors.

Here's an example of a request snooping for environment variables.

{  
  "ip": "13.40.46.249",
  "httpVersion": "HTTP/1.1",
  "location": "City of London, ENG, GB",
  "path": "/.env",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
}

Here's the openai bot hitting an api it scraped with missing parameters.

{
  "ip": "74.7.242.44",
  "location": "Atlanta, GA, US",
  "path": "/v1/credentials/",
  "userAgent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
}

Mitigating the traffic

Using the samples above, let's apply some mitigations. No one protection is enough, so we'll apply protections in layers to block the traffic at the edge.

Placing your API behind a path

One of the simplest mitigations is to put your API behind a path. Since many of the requests for vulnerabilities are scrapers snooping for paths like /wp/wp-admin.php, by putting your API behind a path like /v1 CloudFront will directly reject those requests.

Here's a config snippet using the CDK that puts the Lambda Function URL origin behind the path /v1

additionalBehaviors: {
   "/v1/*": {
       origin: origins.FunctionUrlOrigin.withOriginAccessControl(url, {
          originId: "BlockNeerdowellsStackFunctionUrlOrigin",
   })
   ...
}

Tip

As a bonus, prepending a path is a clean way to version your API. If you decide to make breaking changes or test a new implementation of the same API, leave the existing origin as is. You can add a new /v2 origin to your CloudFront distribution and point it at a new Lambda Function URL.

Add a robots.txt

Well-behaved bots respect robots.txt files. Since my API is behind the path /v1 and /v2, I serve the following robots.txt file using a static S3 origin to prevent good bots from crawling my API.

user-agent: *
disallow: /v1/
disallow: /v2/

sitemap: https://speedrun.nobackspacecrew.com/sitemap.xml

Filtering requests with a CloudFront Function

Alas, many bad requests will still make it through to your API. To reject these, I use a CloudFront Function to check two things. First, it uses the user-agent header to block crawlers and bots that aren't respecting the robots.txt. Second, it validates that the url path is valid for my API using a regex. If either of these checks fails, it rejects the request.

async function handler(event) {
  if (event.context.eventType == "viewer-request") {
    //extract user-agent
    const userAgent = event.request.headers["user-agent"] ? event.request.headers["user-agent"].value : "";
    // block crawlers and bots and use a regex to limit paths to valid api paths
    if ((userAgent && userAgent.match(/(headlessc|spider|crawl|bot)/i)) 
      || !(event.request.uri.match(/^\/v[12]\/((((web)?credentials|federate|((sync|delete)\/role)|register))\/\d{12}|user\/(authenticate|permissions|)|login(\/oauth\/(token|authorize)|(\/device(\/(confirmation|cookie))?)?)|logout)$/))){ 
      //(1)!
      return {
          "statusCode": 400, //(2)!
          "statusDescription": "Bad Request"
      }
    }
    return event.request;
  } else {
    return event.response;
  }
}

This regex may be a bit terse if you aren't used to regexes. Here are some examples of paths it will let through to my API
1. /v1/federate
2. /v1/sync/role/123456789012
3. /v2/user/permissions
4. /v1/device/cookie
5. /v1/logout
Because the code above directly returns the status code and description, CloudFront doesn't pass it through any error code mapping. In the GitHub repository linked below, you can see I switch the origin to the static website and change the request url so it uses the configured CloudFront error page.

Another option: Using WAF

AWS offers WAF (Web Application Firewall) as a turnkey solution for blocking bad requests. It has pre-built rules for standard blocking scenarios and supports rate-limiting and DDOS protection. I'm not currently using it because I'm on the CloudFront pay as you go pricing. However, when you use the flat-rate pricing, all plans provide some level of WAF functionality. If you have a high volume production or mission critical API, use WAF and a flat-rate plan. For smaller hobby projects, using CloudFront Functions may be cheap and good enough alternative. With CloudFront functions, the first 2 million requests free with $0.10/million requests after.

Try it at home

I've created a GitHub repository that creates a date and time API exposed using a Lambda Function URL. It uses CloudFront to expose the API, put it behind a path, add a robots.txt and only allow requests to valid paths. View the README for direct links into the important config and code that backs it.

If you don't want to deploy it yourself, test it using the already deployed version using these links.

Root directory: https://d3a9mbxvm1zrzh.cloudfront.net/
Valid api path: https://d3a9mbxvm1zrzh.cloudfront.net/v1/date
Another valid api path: https://d3a9mbxvm1zrzh.cloudfront.net/v1/time
Invalid api path: https://d3a9mbxvm1zrzh.cloudfront.net/v1/credentials
Path to reject: https://d3a9mbxvm1zrzh.cloudfront.net/wp/wp-admin.php
Robots.txt: https://d3a9mbxvm1zrzh.cloudfront.net/robots.txt

Warning

I didn't connect anything to the /v2 path although the provided CloudFront function has support for it. CloudFront will correctly reject those requests until you connect an API to the /v2 origin.

Conclusion

By putting CloudFront in front of a Lambda Function URL API and using CloudFront Functions, you can block bad requests at the edge and prevent them from polluting your logs. The approach caches errors to reduce load on your origins. If you have feedback or questions, reach out to me on BlueSky or open an issue on the repository.