Install an Immutable Kubernetes Delegate

Updated 4 days ago by Michael Cretzman

Currently, this feature is behind the feature flag USE_IMMUTABLE_DELEGATE. Contact Harness Support to enable the feature. Enabling this Feature Flag means that any future Delegates you install will be immutable, but it doesn’t impact existing Legacy Delegates.

The Harness Delegate is a service you run in your own environment, such as your local network, VPC, or cluster.

For example, you can run the Delegate in the deployment target cluster for a CD Pipeline or the build farm cluster for a CI Pipeline.

The Delegate connects all of your artifact, infrastructure, collaboration, verification, and other providers with the Harness Manager.

Most importantly, the Delegate performs all Harness operations.

There are several types of Delegates. This topic describes how to install the Immutable Kubernetes Delegate.

If you are migrating from Harness FirstGen to Harness NextGen, you must install new Delegates in Harness NextGen. Harness FirstGen Delegates won't work with Harness NextGen.

Before You Begin

Limitations

Currently, Harness Kubernetes Delegates don't install with the default settings in GKE Auto Pilot Mode. Please use the Manual mode when creating the cluster to make sure it meets the Delegate requirements.

The Delegate requires access to all the Connectors and Harness Secrets needed to run a Pipeline. This means that the Delegate requires permissions to do the following:

  • Access all the secrets used by all the Connectors used in a Pipeline.
  • Create and update secrets in Kubernetes. This is necessary to pull the images needed to run individual Steps.

Visual Summary

The following diagram shows how the Delegate enables Harness to integrate with all of your deployment resources:

Here's a 10min video that walks you through adding a Harness Kubernetes Cluster Connector and Harness Kubernetes Delegate. The Delegate is added to the target cluster and then the Kubernetes Cluster Connector uses the Delegate to connect to the cluster:

Review: Inline and Standalone Installation

You can install a Delegate whenever you are adding a Connector to a Pipeline or you can install one outside a Pipeline in Resources.

The steps involved are the same.

Review: Where Can I Install the Kubernetes Delegate?

You can install the Kubernetes Delegate inside or outside your deployment target cluster (CD) or build farm cluster (CIE).

  • Inside the cluster: you can install the Kubernetes Delegate inside the target or build farm cluster. Later, when you add a Kubernetes Cluster Connector, the Connector can inherit its credentials from the Kubernetes Delegate.
  • Outside the cluster: you can install the Kubernetes Delegate outside the target or build farm cluster. Later, when you add a Kubernetes Cluster Connector, the Connector cannot inherit its credentials from the Kubernetes Delegate. In this case, the Kubernetes Cluster Connector must use an alternate method for credentials. For example, the master URL of the target cluster and a Service Account with the required credentials.

Review: Legacy vs Immutable Delegates

There are two types of Kubernetes Delegates:

  • Legacy:
  • Immutable:
    • All binaries and dependencies are installed when the image is built.
    • Ephemeral pods for Kubernetes Delegates.
    • Security scanning of Delegates is possible.
    • Faster setup time.
    • Custom Delegate images can be created by simply building a new image from each image/version Harness releases. Installing additional tools like Terraform and others becomes a lot easier.

The standard installation process in the Harness Manager for both types is the same. For Immutable Delegates, you can also perform a custom installation.

See Review: Custom Installation.

Step 1: Ensure Kubernetes Prerequisites

To install an Immutable Kubernetes Delegate, you must have access to a Kubernetes cluster. You'll install the Harness Delegate as YAML.

For connectivity, see Delegate Requirements and Limitations.

You'll need the following Kubernetes permissions to install the delegate:

  • Permission to create a namespace (for the Harness Delegate namespace).
  • Permission to create Deployments.

Step 2: Select the Kubernetes Delegate Type

Inline or standalone, click New Delegate.

Delegate selection options appear.

Click Kubernetes, and then click Continue.

Enter a name and description for the Delegate that will let others know what it is used for, or where it's installed.

Step 3: Add Delegate Name

Do not run Delegates with the same name in different clusters. See Troubleshooting.

Add a name for the Delegate. The name will be added to the Delegate YAML as the name metadata of the Deployment.

Add Tags to the Delegate. By default, Harness adds a Tag using the name you enter, but you can more. Simply type them in and press Enter.

These Tags are useful for selecting the Delegate when creating a Connector.

Step 4: Select Delegate Size

In Delegate Size, select the size of Delegate you want to install.

Your Kubernetes cluster must have the unallocated resources required to run the Harness Delegate workload:

  • Laptop - 1.6GB memory, 0.5CPU
  • Small - 3.3GB memory, 1CPU
  • Medium - 6.6GB memory, 2CPU
  • Large - 13.2GB memory, 4CPU
Important: these sizing requirements are for the Delegate only. Your cluster will require more memory for Kubernetes, the operating system, and other services.

Important Resource Considerations

These requirements are for the Delegate only. Your cluster will have system, Kubernetes, and other resources consumers. Make sure that the cluster has enough memory, storage, and CPU for all of its resource consumers.

Most importantly, when the Delegate is installed inside the target deployment or build farm cluster, the cluster must also support the resources needed by the services you are deploying or building.

For example, if you use the Small option that requires 3.3GB of memory, don't use a cluster with only 4GB or memory. It won't be enough to run the Delegate and other resources.

Step 5: Download and Install the Script

Click Download Script. The YAML file for the Kubernetes Delegate, and its README, will download to your computer as an archive.

Open a terminal and navigate to where the Delegate file is located.

Extract the YAML file's folder from the download and then navigate to the folder that you extracted:

tar -zxvf harness-delegate-kubernetes.tar.gz

cd harness-delegate-kubernetes

You'll connect to your cluster using the terminal so you can simply copy the YAML file over.

In the same terminal, log into your Kubernetes cluster. In most platforms, you select the cluster, click Connect, and copy the access command.

Let's quickly confirm that the cluster you created can connect to the Harness platform. Enter the following command:

Next, install the Harness Delegate using the harness-delegate.yaml file you just downloaded. In the terminal connected to your cluster, run this command:

kubectl apply -f harness-delegate.yaml

The successful output is something like this (this Delegate's name is doc-immut):

namespace/harness-delegate-ng created
clusterrolebinding.rbac.authorization.k8s.io/harness-delegate-ng-cluster-admin created
secret/doc-immut-account-token created
deployment.apps/doc-immut created
service/delegate-service created
role.rbac.authorization.k8s.io/upgrader-cronjob created
rolebinding.rbac.authorization.k8s.io/doc-immut-upgrader-cronjob created
serviceaccount/upgrader-cronjob-sa created
secret/doc-immut-upgrader-token created
configmap/doc-immut-upgrader-config created
Warning: batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
cronjob.batch/doc-immut-upgrader-job created

Run this command to verify that the Delegate pod was created:

kubectl get pods -n harness-delegate-ng

It'll take a moment for the Delegate to appear in Harness' Delegates list.

You can click Finish and check back later.

You're ready to connect Harness to your artifact server and cluster. After those quick steps, you'll begin creating your deployment.

Review: Delegate Role Requirements

The YAML provided for the Harness Delegate defaults to the cluster-admin role because that ensures anything could be applied. If you can't use cluster-admin because you are using a cluster in your company, you'll need to edit the Delegate YAML.

The set of permissions should include listgetcreatewatch (to fetch the pod events), and delete permissions for each of the entity types Harness uses.

If you don’t want to use resources: [“*”] for the Role, you can list out the resources you want to grant. Harness needs configMapsecreteventdeployment, and pod at a minimum for deployments, as stated above.

In the Delegate installation settings, you also have the option to select cluster read-only access and namespace-specific access. When you select these options, the YAML generated by Harness is changed to reflect the limited access:

Step 6: Verify

For an overview of verification, see Delegate Registration and Verification.

In the Delegate wizard, click Verify and Harness will verify that it is receiving heartbeats from the Delegate.

Your Delegate is installed.

Option: Troubleshooting

Harness will provide a lot of troubleshooting steps. Here are a few:

Check the status of the Delegate on your cluster:

kubectl describe pod <your-delegate-pod> -n harness-delegate-ng

Check the Delegate logs:

kubectl logs -f <harness-delegate> -n harness-delegate-ng

If the pod isn't up, you might see the following error in your cluster:

CrashLoopBackOff: Kubernetes Cluster Resources are not available.

Make sure the Kubernetes Cluster Resources (CPU, Memory) are enough.

If the Delegate didn’t reach a healthy state, try this:

kubectl describe pod <your-delegate-pod> -n harness-delegate-ng

Immutable Kubernetes Delegate Environment Variables

The following table lists each of the environment variables in the Harness Kubernetes Delegate YAML.

Only some of these variables are in the default Delegate YAML. You can add the others as needed.

Name

Description

Example

JAVA_OPTS

JVM options for the Delegate. Use this variable to override or add JVM parameters.

- name: JAVA_OPTS
value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -Xms64M"

ACCOUNT_ID

The Harness account Id for the account where this Delegate will attempt to register.

This value is added automatically to the Delegate config file (YAML, etc) when you add the Delegate.
- name: ACCOUNT_ID
value: H5W8iol5TNWc4G9h5A2MXg

ACCOUNT_SECRET

The Harness account token used to register the Delegate.

- name: ACCOUNT_SECRET
value: d239xx88bf7xxxxxxx836ea

MANAGER_HOST_AND_PORT

The Harness SaaS manager URL. https indicates port 443.

- name: MANAGER_HOST_AND_PORT
value: https://app.harness.io

WATCHER_STORAGE_URL

The URL for the Watcher versions.

See Delegate Installation Overview.

- name: WATCHER_STORAGE_URL
value: https://app.harness.io/public/prod/premium/watchers

WATCHER_CHECK_LOCATION

The Delegate version location for the Watcher to check for.

- name: WATCHER_CHECK_LOCATION
value: current.version

REMOTE_WATCHER_URL_CDN

The CDN URL for Watcher builds.

- name: REMOTE_WATCHER_URL_CDN
value: https://app.harness.io/public/shared/watchers/builds

DELEGATE_STORAGE_URL

The URL where published Delegate jars are stored.

- name: DELEGATE_STORAGE_URL
value: https://app.harness.io

DELEGATE_CHECK_LOCATION

The storage location hosting the published Delegate versions.

- name: DELEGATE_CHECK_LOCATION
value: delegateprod.txt

DEPLOY_MODE

Deployment mode: Kubernetes, Docker, etc.

- name: DEPLOY_MODE
value: KUBERNETES

DELEGATE_NAME

The name of the Delegate. This is the name that will appear in Harness when the Delegate is registered.

You can automate Delegate creation by omitting the name, and then have a script copying the Delegate YAML file and add a unique name to value for each new Delegate you want to register.

See Automate Delegate Installation.

- name: DELEGATE_NAME
value: qa

NEXT_GEN

Indicates that this Delegate will register in Harness NextGen.

If it set to false, the Delegate will attempt to register in Harness FirstGen.

- name: NEXT_GEN
value: "true"

DELEGATE_DESCRIPTION

The description added to the Delegate in the Harness Manager or YAML before registering.

It appears in the Delegate details page in the Harness Manager.

- name: DELEGATE_DESCRIPTION
value: ""

DELEGATE_TYPE

The type of Delegate.

- name: DELEGATE_TYPE
value: "KUBERNETES"

DELEGATE_TAGS

The Tags added to the Delegate in the Harness Manager or YAML before registering.

Tags are generated by Harness using the Delegate name but you can also add your own Tags.

Tags appear in the Delegate details page in the Harness Manager.

See Tags Reference and Select Delegates with Tags.

- name: DELEGATE_TAGS
value: ""

DELEGATE_TASK_LIMIT

The maximum number of tasks the Delegate can perform at once.

All of the operations performed by the Delegate are categorized as different types of tasks.

- name: DELEGATE_TASK_LIMIT
value: "50"

DELEGATE_ORG_IDENTIFIER

The Harness Organization Identifier where the Delegate will register.

Delegates at the account-level do not have a value for this variable.

- name: DELEGATE_ORG_IDENTIFIER
value: "engg"

DELEGATE_PROJECT_IDENTIFIER

The Harness Project Identifier where the Delegate will register.

Delegates at the account or Org-level do not have a value for this variable.

- name: DELEGATE_PROJECT_IDENTIFIER
value: "myproject"

PROXY_*

All of the Delegates include proxy settings you can use to change how the Delegate connects to the Harness Manager.

The secretKeyRef are named using the Delegate name.

- name: PROXY_HOST
value: ""
- name: PROXY_PORT
value: ""
- name: PROXY_SCHEME
value: ""
- name: NO_PROXY
value: ""
- name: PROXY_MANAGER
value: "true"
- name: PROXY_USER
valueFrom:
secretKeyRef:
name: mydel-proxy
key: PROXY_USER
- name: PROXY_PASSWORD
valueFrom:
secretKeyRef:
name: mydel-proxy
key: PROXY_PASSWORD

INIT_SCRIPT

You can run scripts on the Delegate using INIT_SCRIPT.

INIT_SCRIPT is typically not used for Immutable Delegates.

For the Immutable Delegate, initialization should be baked into the image not executed on startup.

- name: INIT_SCRIPT
value: |-
echo install wget
apt-get install wget
echo wget installed

POLL_FOR_TASKS

Enables or disables polling for Delegate tasks.

By default, the Delegate uses Secure WebSocket (WSS) for tasks. If the PROXY_* settings are used and the proxy or some intermediary does not allow WSS, then set POLL_FOR_TASKS to true to enable polling.

- name: POLL_FOR_TASKS
value: "false"

HELM_DESIRED_VERSION

By default, Harness Delegates are installed with and use Helm 3.

You can set the Helm version in the Harness Delegate YAML file using the HELM_DESIRED_VERSION environment property. Include the v with the version. For example, HELM_DESIRED_VERSION: v2.13.0.

- name: HELM_DESIRED_VERSION
value: ""

USE_CDN

Makes the Delegate use a CDN for new versions.

- name: USE_CDN
value: "true"

CDN_URL

The CDN URL for Delegate versions.

- name: CDN_URL
value: https://app.harness.io

JRE_VERSION

The Java Runtime Environment version used by the Delegate.

- name: JRE_VERSION
value: 1.8.0_242

GRPC_SERVICE_ENABLED,

GRPC_SERVICE_CONNECTOR_PORT

By default, the Delegate requires HTTP/2 for gRPC (gRPC Remote Procedure Calls) be enabled for connectivity between the Delegate and Harness Manager.

- name: GRPC_SERVICE_ENABLED
value: "true"
- name: GRPC_SERVICE_CONNECTOR_PORT
value: "8080"

VERSION_CHECK_DISABLED

By default, the Delegate always checks for new versions (via the Watcher).

- name: VERSION_CHECK_DISABLED
value: "false"

DELEGATE_NAMESPACE

The namespace for the Delegate is taken from the StatefulSet namespace.

- name: DELEGATE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace

Review: Custom Installation

A custom installation using a custom Delegate image is easy to set up.

You can use the public Immutable Delegate Docker image and build the Immutable Delegate yourself.

Once you have your custom image built and stored in a repository, all you need to do download the Harness Delegate YAML from Harness and then change is the image setting in the YAML to point to your new image. The rest of the YAML can remain the same.

Use latest available Immutable Delegate image from the public Docker Hub repo.

Create a Delegate Image

Harness Delegate Docker images are public and you can use them to compose your own Delegate image.

The Harness Delegate Docker images are located on Docker Hub.

For example, you can curl and install all the tool libraries you want and then curl delegate:<version/tag> to add the latest Delegate image.

Or you can create your own image and simply include the Delegate image.

Here's and example that installs Node.JS on top of a Delegate image and sets an environment variable key:value pair.

...
FROM harness/delegate:<version/tag>

RUN apt-get update && apt-get -y install nodejs

ENV key=value
...

Next Steps

  1. Monitor for New Version Tags. You need to monitor for when Harness pushes new version tags.
  2. Build and Push your Custom Image. Build your custom image from the new tag and push it into their own repo.
  3. Apply the Delegate YAML.

Sample YAML

The Delegate has two components, the Delegate itself, which is mandatory, and the upgrader, which is optional and should only be used if you want to enable automatic updates of Delegate images.

In the following YAML files, do the following:

Replace <delegate name> with the name of Immutable Delegate.

Replace <account id> with your Harness account Id.

Update MANAGER_HOST_AND_PORT:

  • For Prod1 cluster: https://app.harness.io
  • For Prod2 cluster: https://app.harness.io/gratis

Update LOG_STREAMING_SERVICE_URL:

  • For Prod1 cluster: https://app.harness.io/log-service/
  • For Prod2 cluster: https://app.harness.io/gratis/log-service/

For ACCOUNT_SECRET, add the Base64 encoded value.

Sample Delegate YAML
apiVersion: v1
kind: Namespace
metadata:
name: harness-delegate-ng

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: harness-delegate-ng-cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: harness-delegate-ng
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io

---

apiVersion: v1
kind: Secret
metadata:
name: immutable-delegate-account-token
namespace: harness-delegate-ng
type: Opaque
data:
ACCOUNT_SECRET: <base64 encoded value for account secret>

---

# If delegate needs to use a proxy, please follow instructions available in the documentation
# https://ngdocs.harness.io/article/5ww21ewdt8-configure-delegate-proxy-settings

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
harness.io/name: <delegate name>
name: <delegate name>
namespace: harness-delegate-ng
spec:
replicas: 2
selector:
matchLabels:
harness.io/name: <delegate name>
template:
metadata:
labels:
harness.io/name: <delegate name>
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3460"
prometheus.io/path: "/api/metrics"
spec:
terminationGracePeriodSeconds: 600
restartPolicy: Always
containers:
- image: <immutable delegate image>
imagePullPolicy: Always
name: delegate
ports:
- containerPort: 8080
resources:
limits:
cpu: "0.5"
memory: "2048Mi"
requests:
cpu: "0.5"
memory: "2048Mi"
livenessProbe:
httpGet:
path: /api/health
port: 3460
scheme: HTTP
initialDelaySeconds: 120
periodSeconds: 10
failureThreshold: 2
envFrom:
- secretRef:
name: immutable-delegate-account-token
env:
- name: JAVA_OPTS
value: "-Xms64M"
- name: ACCOUNT_ID
value: <account id>
- name: MANAGER_HOST_AND_PORT
value: <https://app.harness.io OR https://app.harness.io/gratis>
- name: DEPLOY_MODE
value: KUBERNETES
- name: DELEGATE_NAME
value: <delegate name>
- name: DELEGATE_TYPE
value: "KUBERNETES"
- name: DELEGATE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INIT_SCRIPT
value: ""
- name: DELEGATE_DESCRIPTION
value: ""
- name: DELEGATE_TAGS
value: ""
- name: DELEGATE_ORG_IDENTIFIER
value: ""
- name: DELEGATE_PROJECT_IDENTIFIER
value: ""
- name: NEXT_GEN
value: "true"
- name: CLIENT_TOOLS_DOWNLOAD_DISABLED
value: "true"
- name: LOG_STREAMING_SERVICE_URL
value: "https://app.harness.io/log-service/ OR https://app.harness.io/gratis/log-service/"

---

apiVersion: v1
kind: Service
metadata:
name: delegate-service
namespace: harness-delegate-ng
spec:
type: ClusterIP
selector:
harness.io/name: <delegate name>
ports:
- port: 8080

See Also


Please Provide Feedback