mdashikjs/blog
All posts
Blue-Green and Canary Deployments: Shipping Without Fear
DevOps

Blue-Green and Canary Deployments: Shipping Without Fear

DevOps6 min

Blue-Green and Canary Deployments: Shipping Without Fear

Rolling back a broken deploy should take 30 seconds, not 30 minutes. Blue-green and canary deployments give you that confidence by separating the release from the deploy.

DeploymentDevOpsKubernetesInfrastructure
Share:

Blue-green deployments

Run two identical environments. Blue is live. Green is idle. Deploy to green, test it, then switch traffic:

Before deploy:
  Users → Load Balancer → Blue (v1.0) ✓
                          Green (idle)

During deploy:
  Users → Load Balancer → Blue (v1.0) ✓
                          Green (v1.1) ← deploying

After switch:
  Users → Load Balancer → Blue (v1.0, idle)
                          Green (v1.1) ✓

Rollback:
  Users → Load Balancer → Blue (v1.0) ✓  ← instant
                          Green (v1.1, idle)

The switch is a load balancer configuration change — not a redeploy. Rollback is the same switch in reverse.

Implementing with Nginx

# /etc/nginx/conf.d/app.conf
upstream app {
  server 10.0.1.1:3000;  # Blue
  # server 10.0.1.2:3000;  # Green (uncomment to switch)
}

server {
  listen 80;
  location / {
    proxy_pass http://app;
  }
}

Switch by commenting/uncommenting and running nginx -s reload. In practice, automate this with a deployment script.

Implementing with AWS

With ECS and an Application Load Balancer:

# Deploy new version to green target group
aws ecs update-service --cluster prod --service api-green --task-definition api:42

# Wait for green to be healthy
aws ecs wait services-stable --cluster prod --services api-green

# Switch ALB listener to green
aws elbv2 modify-listener --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$GREEN_TG_ARN

# Rollback: switch back to blue
aws elbv2 modify-listener --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$BLUE_TG_ARN

Canary deployments

Blue-green is all-or-nothing. Canary deployments shift traffic gradually:

Step 1: 5% → New version, 95% → Old version
Step 2: 25% → New version, 75% → Old version
Step 3: 50% → New version, 50% → Old version
Step 4: 100% → New version

At each step, monitor error rates, latency, and business metrics. If anything looks wrong, route 100 percent back to the old version.

Canary with Nginx

upstream app {
  server 10.0.1.1:3000 weight=95;  # Old version
  server 10.0.1.2:3000 weight=5;   # New version (canary)
}

Adjust weights as confidence grows.

Canary with Kubernetes

Kubernetes makes canary deployments natural with multiple Deployments behind one Service:

# Stable: 9 replicas
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-stable
spec:
  replicas: 9
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry/api:1.0
---
# Canary: 1 replica
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-canary
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry/api:1.1

Both Deployments have the same app: api label, so the Service routes to both. With 9 stable and 1 canary pod, roughly 10 percent of traffic hits the new version.

What to monitor during canary

  1. Error rate: compare canary vs stable. If canary errors are 2x higher, abort.
  2. Latency P95: a slower canary might indicate a performance regression.
  3. Business metrics: conversion rate, checkout success rate — technical metrics alone miss business impact.

When to use which

  • Blue-green: simple, fast rollback. Best for smaller teams, lower traffic, or when you want to test the full deployment before switching.
  • Canary: gradual rollout. Best for high-traffic services where a full switch is risky, or when you want real user validation before committing.
  • Both: use blue-green for staging validation, canary for production rollout.

The prerequisite: observability

Neither strategy works without monitoring. If you cannot tell whether the new version is healthy within 5 minutes, you cannot make informed decisions about traffic shifting. Set up metrics and alerting before investing in deployment strategies.

MA

Written by Md Ashik

Senior Software Engineer building reliable backends. I write about the practical tradeoffs behind shipping software that holds up in production.