Create a Kubernetes Blue Green Deployment

Updated 1 month ago by Michael Cretzman

This topic will walk you through creating a Blue Green deployment in Harness for a Kubernetes Deployment workload.

For information on Blue/Green deployments, see Deployment Concepts and Strategies.

Harness Canary and Blue/Green strategies only support Kubernetes Deployment workloads. The Rolling strategy supports all other workloads, except Jobs. The Apply Step can deploy any workloads or objects.

In this topic:

Before You Begin

Review: What Workloads Can I Deploy?

See What Can I Deploy in Kubernetes?.

Review: Harness Blue Green Deployments

Here's a quick summary of how Harness performs Blue Green deployments.

Only one Kubernetes service is mandatory and it doesn’t need any annotations to establish if it is the primary (production) service.

Here is a very generic service example that uses a values.yaml file for its values:

apiVersion: v1
kind: Service
metadata:
name: {{.Values.name}}-svc
spec:
type: {{.Values.serviceType}}
ports:
- port: {{.Values.servicePort}}
targetPort: {{.Values.serviceTargetPort}}
protocol: TCP
selector:
app: {{.Values.name}}

This file and sample deployment and values.yaml files are publicly available on the Harness Docs repo.

Note that there are no annotations to indicate that it is the primary service. Harness will add this later.

If you have more than one service, Harness does not automatically know which is the primary service unless you add the annotations described below. If you use two services, please annotate them as described below.
  1. First deployment:
    1. Harness creates two services (primary and stage) and one pod set for the app.
    2. The primary service is given this annotation: annotations: harness.io/primary-service: "true".
    3. The stage service is given this annotation: annotations: harness.io/stage-service: "true".
    4. The pod set is given an annotation of harness.io/color: blue.
    5. Harness points the stage service at the pod set and verifies that the set reached steady state.
    6. Harness swaps the primary service to pod set. Production traffic now flows to the app.
  2. Second deployment (new version of the same app):
    1. Harness creates a new pod set for new app version. The pod set is given the annotation harness.io/color: green.
    2. Harness points the stage service at new pod set (with new app version) and verifies that the set reached steady state.
    3. Harness swaps the primary service to new pod set, stage service to old pod set.
  3. Third deployment:
    1. Harness deploy new app version to the pod set not using the primary service.
    2. Harness points the stage service at new pod set (with new app version) and verifies that the set reached steady state.
    3. Harness swaps the primary service to new pod set, stage service to old pod set.

Visual Summary

Here's a video walking through a simple Canary deployment. It's 10 minutes long but it covers set up and two deployments.

Step 1: Define the Service and Infrastructure

Create your CD Pipeline stage.

To set up your Service and Infrastructure in the stage, follow the steps in these topics:

Once the Service and Infrastructure are set up, you can add the execution steps.

Step 2: Add the Execution Steps

In the stage's Execution, click Add Step, and select the Blue Green strategy.

Harness adds all the steps you need to perform the Blue Green strategy:

That's it. Harness will deploy the artifact using the stage service initially, and swap traffic to the primary service.

Let's look at the default settings for the Stage Deployment step.

Step 3: Stage Deployment Step

The Stage Deployment step is added automatically when you apply the Blue Green strategy.

Click the Stage Deployment step. The step simply includes a name, timeout, and Skip Dry Run options.

Skip Dry Run: By default, Harness uses the --dry-run flag on the kubectl apply command during the Initialize step of this command, which prints the object that would be sent to the cluster without really sending it. If the Skip Dry Run option is selected, Harness will not use the --dry-run flag.

The first time you deploy, the Stage Deployment step creates two Kubernetes services, a new pod set, and deploys your app to the pod set.

When you look at the Stage Deployment step in Harness Deployments, you will see the following log sections.

Fetch Files

Harness pulls the manifests and values.yaml from your repo.

Initialize

The Initialize stage initializes the two Kubernetes services and deployment object, validating their YAML.

Prepare

Typically, in a Prepare section, you can see that each release of the resources is versioned. This is used in case Harness needs to rollback to a previous version.

In the case of Blue Green, the resources are not versioned because a Blue Green deployment uses rapid rollback: network traffic is simply routed back to the original instances.

You do not need to redeploy previous versions of the service/artifact and the instances that comprised their environment.

The Prepare section shows that Harness has prepared two services, identified the deployment as blue, and pointed the stage service (blue) at the blue pod set for the deployment:

Manifests processed. Found following resources: 

Kind Name Versioned
Service bgdemo-svc false
Deployment bgdemo false

Primary Service is bgdemo-svc

Created Stage service [bgdemo-svc-stage] using Spec from Primary Service [bgdemo-svc]

Primary Service [bgdemo-svc] not found in cluster.

Stage Service [bgdemo-svc-stage] not found in cluster.

Primary Service is at color: green

Stage Service is at color: blue

Cleaning up non primary releases

Current release number is: 1

Versioning resources.

Workload to deploy is: Deployment/bgdemo-blue

Done.

Apply

The Apply section applies a services and deployment from the Prepare section. It uses a combination of all of the manifests in the Service Manifests section as one file using kubectl apply.

kubectl --kubeconfig=config apply --filename=manifests.yaml --record

service/bgdemo-svc created

deployment.apps/bgdemo-blue created

service/bgdemo-svc-stage created

Done.

Wait for Steady State

The Wait for Steady State section shows Harness confirming the rollout and that the pods have reached steady state.

Next, the Swap Primary with Stage step will swap the primary and stage services to route primary network traffic to the pod set for the app.

If this were the second deployment, Harness would also swap the stage service to the pod set for the old app version.

Step 4: Swap Primary with Stage Step

Click the Swap Primary with Stage step.

In the Prepare step you saw the primary service pointing at the green pod set and the stage service pointing at blue pod set containing the app.

In Swap Primary with Stage, Harness swaps the primary service to the pod set running the app (blue) and the stage service to the other color (green). Since this is the first deployment, there is no actual green pod set.

Production traffic now flows to the app.

Selectors for Service One : [name:bgdemo-svc]

app: bgdemo

harness.io/color: green

Selectors for Service Two : [name:bgdemo-svc-stage]

app: bgdemo

harness.io/color: blue

Swapping Service Selectors..

Updated Selectors for Service One : [name:bgdemo-svc]

app: bgdemo

harness.io/color: blue

Updated Selectors for Service Two : [name:bgdemo-svc-stage]

app: bgdemo

harness.io/color: green

Done

The next time you deploy, the swap will point the primary service at the green pod set and the stage service at the blue pod set:

...
Swapping Service Selectors..

Updated Selectors for Service One : [name:bgdemo-svc]

app: bgdemo

harness.io/color: green

Updated Selectors for Service Two : [name:bgdemo-svc-stage]

app: bgdemo

harness.io/color: blue

Done

Option: Scale Down Old Version

A great benefit of a Blue/Green deployment is rapid rollback: rolling back to the old version of an app is simple and reliable because network traffic is simply routed back to the previous pods.

You do not need to redeploy previous versions of the app and the pods that comprised their environment.

If you would like to scale down the old version, add a Shell Script step to the post-deployment steps of your stage.

Here's an example using <+pipeline.stages.[stage_name].spec.execution.steps.stageDeployment.output.stageServiceName> to reference the stage service name. The name of the stage is nginx so the reference is <+pipeline.stages.nginx.spec.execution.steps.stageDeployment.output.stageServiceName>.

kubectl scale deploy -n <+infra.namespace> $(kubectl get deploy -n <+infra.namespace> -o jsonpath='{.items[?(@.spec.selector.matchLabels.harness\.io/color=="'$(kubectl get service/<+pipeline.stages.nginx.spec.execution.steps.stageDeployment.output.stageServiceName> -n <+infra.namespace> -o jsonpath='{.spec.selector.harness\.io/color}')'")].metadata.name}') --replicas=0

In this example, the

Option: Using the Horizontal Pod Autoscaler (HPA)

If you are using the Horizontal Pod Autoscaler with your deployment, create a blue and green HPA configuration that will point at your deployments.

templates/hpa-blue.yaml:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: {{.Values.name}}-blue
labels:
harness.io/color: blue
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Values.name}}-blue
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- toYaml .Values.autoscaling.metrics | indent 4 }}

templates/hpa-green.yaml:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: {{.Values.name}}-green
labels:
harness.io/color: green
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{.Values.name}}-green
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- toYaml .Values.autoscaling.metrics | indent 4 }}

You can add your scaling configuration to your manifest (or share it if you are using a Helm chart):

autoscaling:
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 20
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 20

When using this with a traffic splitting strategy, your pods will scale automatically as your new pods begin receiving heavier loads.

Notes

  • Blue/Green Rollback — A great benefit of a Blue/Green deployment is rapid rollback: rolling back to the old version of a service/artifact is simple and reliable because network traffic is simply routed back to the original instances. You do not need to redeploy previous versions of the service/artifact and the instances that comprised their environment.

Next Steps


Please Provide Feedback