Hello, I’m Akito Koga.
I recently built a blue-green deployment environment on AWS using ECS and CodeDeploy. A blue-green deployment normally creates a staging environment during deployment, and CloudFront Continuous Deployment turned out to be a useful way to route between production and staging when CloudFront sits in front of the application. This post records what I learned from implementing it.
Table of Contents
Open Table of Contents
What Is Blue-Green Deployment?
Blue-green deployment is a strategy for releasing software safely. You maintain a production environment and a staging environment, start the new version in staging, and then switch the load balancer target from the old environment to the new one.
It is commonly used with container orchestration platforms such as ECS and Kubernetes. I also adopted it as the release strategy for an application I work on.
The load balancer can shift targets in several ways. An all-at-once deployment can release without downtime, while a canary release can expose the new version to a small share of users before gradually increasing traffic. Easy rollback is another important benefit.
Blue-green deployment is not a silver bullet. It adds operational steps and requires the temporary staging environment to consume resources during a deployment. Rolling updates and other approaches remain valid choices, so the deployment method should fit the application’s requirements.
The Staging Environment in a Blue-Green Deployment
During a blue-green deployment, the new staging, or green, environment starts first. The software is released by changing the load balancer target from the blue environment to the green one.
A common setup assigns port 80 to production traffic and port 10080 to staging traffic. Before deployment, both listeners target the blue environment. During deployment, the port 10080 listener changes to the green environment, allowing developers to verify the new software through that listener.
The port 80 listener still points to blue, so users continue to reach the production version while developers test green through port 10080. Once verification is complete, user traffic can begin moving to green.
The switch can happen at different rates. Sending 100% of traffic to green is an all-at-once deployment. Sending a smaller percentage first is a canary release.
That is the typical role of staging in a blue-green deployment.
The Problem with CloudFront
When CloudFront is placed in front of the ALB as a CDN, there is no equally simple way to access the staging listener.
CloudFront supports multiple origins, so it may seem possible to register the port 80 and port 10080 listeners as separate origins and route between them. CloudFront routing, however, is based on URL paths; it cannot select an origin merely because the client changed a port number.
You could force staging traffic onto another path, but using different paths in production and staging introduces constraints. A staging environment is much less useful if it cannot reproduce production behavior.
Historically, using CloudFront in front of an ALB with ECS blue-green deployments therefore required more involved workarounds, such as registering a separate staging host in DNS.
CloudFront Continuous Deployment
This limitation made CloudFront and ECS blue-green deployment an awkward combination. CloudFront Continuous Deployment provided exactly the connection I needed.
The feature uses two CloudFront distributions: a primary production distribution and a staging distribution linked to it. The production distribution registers the ALB’s port 80 listener as its origin, while the staging distribution registers the port 10080 listener. CloudFront can then select between the two distributions according to a continuous deployment policy.
CloudFront supports weight-based and header-based policies. A weight-based policy resembles a canary release and randomly sends a configured percentage of requests to staging. That was not what I needed here, so I chose a header-based policy.
With a header-based policy, a request header whose value follows the configured aws-cf-cd- pattern determines whether CloudFront uses the staging distribution.
Issues I Encountered
The easiest way to add the special request header in a browser such as Chrome is with an extension. Many of the header-modifying extensions I found did not inspire much confidence from a security perspective. I eventually found Requestly, which appeared suitable and trustworthy for this use case.
My application used Next.js on the frontend and a Web API on the backend. The blue-green deployment applied to the backend API, so the frontend also had to forward the additional header. If the API is deployed independently from the frontend, remember to propagate the request header. With an SSR frontend, this must also happen during server-side fetches.
There was one more AWS-specific trap. The ALB security group needed CloudFront’s AWS-managed prefix list on both port 80 and port 10080. A security group allows 60 rules by default, while the managed prefix list has a weight of 55. After adding one inbound rule for port 80, only five rule slots remain.
To add the corresponding rule for port 10080, I had to request an increase in the security-group rule quota to at least 110. The maximum is 200. If an IaC tool such as Terraform cannot apply this configuration, check whether the security-group quota is the cause.
Conclusion
This post described how to use an additional request header with CloudFront Continuous Deployment to route to the staging environment of an ECS blue-green deployment.
I was initially unsure how smoothly the feature would work, but verification was surprisingly straightforward. It is encouraging to see AWS continue addressing these detailed operational needs.
The resulting architecture is more complex, though. Before implementing it, make sure CloudFront is truly appropriate for the application and choose the design that best matches its requirements.