Cloud & Infrastructure Bug Report Paid

AWS WAF on CloudFront bypassed by calling the ALB directly: locking down the origin

Our WAF rate limit and managed rules worked perfectly, for anyone polite enough to go through CloudFront. A scripted client hitting the ALB hostname directly skipped all of it. Here is how we found it, why it happens, and the layered origin protection we now ship by default.

John Nguyen John Nguyen @devbugs · June 13, 2025 · 12 min read · 62 Views
Banner: traffic going around a CDN firewall straight to an exposed load balancer

Our login endpoint sits behind CloudFront with an AWS WAF web ACL attached to the distribution: AWS managed rule groups plus a rate-based rule limiting each IP to 300 requests per five minutes on /api/auth/*. During a credential stuffing wave, the WAF dashboard looked calm, a few thousand blocked requests, while our application logs showed tens of thousands of failed logins from a small set of IPs. The numbers did not add up.

The application access logs had one detail that gave it away. Legitimate traffic arrived with CloudFront's headers and a CloudFront edge address as the TCP peer. The attack traffic did not:

203.0.113.77 - - [04/Jun/2025:02:14:09 +0000] "POST /api/auth/login HTTP/1.1" 401 52 "-" "python-requests/2.32.3" via=- x_amz_cf_id=- elb=app-prod-alb

No X-Amz-Cf-Id, no Via: 1.1 ... (CloudFront). The requests were reaching the Application Load Balancer without passing through CloudFront at all, so the WAF never saw them.

Reproducing it

From any machine on the internet, resolve the ALB's own DNS name and send a request with the public Host header:

# Through CloudFront: the 301st request in five minutes is blocked by WAF
curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/api/auth/login -X POST -d '{}'

# Directly to the ALB: no WAF, no rate limit
curl -sk -o /dev/null -w '%{http_code}\n' \
  -H 'Host: app.example.com' -X POST -d '{}' \
  https://app-prod-alb-1234567890.eu-west-1.elb.amazonaws.com/api/auth/login
# 401

Running the second command in a loop of 1,000 requests never produced a single 403. The same loop against CloudFront started returning 403 from the WAF after about 300.

What we ruled out

  • WAF rule logic. Sampled requests in the web ACL showed the rate rule matching and blocking correctly for traffic it actually received.
  • CloudFront caching or behaviour routing. The /api/* behaviour had caching disabled and all methods allowed; every request through CloudFront reached the WAF.
  • IP spoofing via X-Forwarded-For. Rate-based rules on CloudFront key on the viewer IP CloudFront sees, not on a client-supplied header, so the attacker could not have rotated IPs that way. They did not need to.

Root cause

The ALB was internet-facing, in public subnets, with a security group allowing 443 from 0.0.0.0/0. That was how the stack was first built, before CloudFront was added in front. Attaching the WAF to the distribution protected one path into the application, and the original path stayed open. The ALB's hostname is not secret: it appears in DNS history, in old configuration, and anyone scanning the provider's address ranges for TLS certificates can find a load balancer that presents a certificate for app.example.com.

What is happening under the hood

CloudFront is a reverse proxy; the WAF attached to it inspects requests at the edge. The origin, here an ALB, is a separate public endpoint with its own DNS name and IP addresses. Nothing in the CloudFront configuration tells the ALB to refuse traffic from elsewhere. Unless you add that restriction at the origin, an attacker can choose which path to use, and will pick the one without inspection.

There are two independent questions to answer at the origin:

  1. Network layer: is this connection coming from CloudFront's infrastructure at all?
  2. Application layer: is it coming from our distribution, and not from some other CloudFront distribution that an attacker created and pointed at our ALB?

Restricting by source IP answers the first. Because every AWS customer shares CloudFront's IP ranges, it does not answer the second. An attacker can create their own distribution with no WAF and our ALB as its origin. A shared secret that only our distribution sends answers the second.

Environment

Component Version / value
Edge CloudFront, one distribution, alternate domain app.example.com
WAF AWS WAF (WAFv2), CLOUDFRONT scope, managed rules + rate-based rule
Origin Internet-facing ALB, HTTPS listener with ACM certificate
Origin hostname origin.example.com CNAME to the ALB
ALB security group 443 from 0.0.0.0/0 (the bug)
IaC Terraform 1.11, hashicorp/aws provider 5.9x

Minimal reproduction

The essence of the vulnerable setup:

resource "aws_security_group" "alb" {
  name   = "alb-prod"
  vpc_id = var.vpc_id
}

resource "aws_vpc_security_group_ingress_rule" "alb_https_world" {
  security_group_id = aws_security_group.alb.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "tcp"
  from_port         = 443
  to_port           = 443
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.app.arn
  port              = 443
  protocol          = "HTTPS"
  certificate_arn   = var.origin_cert_arn
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

Everything that arrives on 443 is forwarded to the application.

Who is affected

Any setup where a CDN or edge WAF sits in front of an origin that is still reachable from the internet: CloudFront with ALB, API Gateway, EC2 or S3 website endpoints, and the same pattern with third-party CDNs. Setups using CloudFront VPC origins with an internal ALB, or Origin Access Control for S3 REST origins, are not affected in this way.

This is a paid story

Subscribe to read the full content, access all paid stories, and support the authors.

Add a label

#terraform #aws #cloudfront #alb #aws-waf #security-groups
Written by
John Nguyen @devbugs

I am a Senior Software Engineer with over a decade of hands-on experience building, scaling, and maintaining complex systems across the full stack—spanning frontend, backend, and DevOps. Over the last 10 years, I have navigated diverse architectures and untangled some of the most frustrating, elusive bugs in production. I believe that every production incident is a massive learning opportunity. Having discovered hibug.me, I am excited to contribute to this knowledge base, breaking down root causes, and sharing verified fixes to help fellow developers debug faster and build more resilient systems.

Advertisement

Comments

0

No comments yet. Be the first to comment!

Sign In to post a comment.