Kubernetes Cluster Connector Settings Reference

Updated 2 weeks ago by Michael Cretzman

This topic provides settings and permissions for the Kubernetes Cluster Connector.

The Kubernetes Cluster Connector is a platform-agnostic connection to a Kubernetes cluster located anywhere.

For cloud platform-specific connections, see platform Cloud Connectors.

Looking for the How-to? See Add a Kubernetes Cluster Connector.

In this topic:

Video Summary

Here's a 10min video that walks you through adding a Harness Kubernetes Cluster Connector and Harness Kubernetes Delegate. The Delegate is added to the target cluster and then the Kubernetes Cluster Connector uses the Delegate to connect to the cluster:

Kubernetes Cluster Connector vs Platform Connectors

The Kubernetes Cluster Connector is platform-agnostic. Use it to access a cluster on any platform.

It cannot also access platform-specific services and resources. For those, use a platform Connector like Google Cloud Platform or Amazon Web Services.

For example, let's say you have a GKE Kubernetes cluster hosted in Google Cloud Platform (GCP). You can use the Kubernetes Cluster Connector to connect Harness to the cluster in GCP. The Kubernetes Cluster Connector cannot also access Google Container Registry (GCR).

In this case, you have two options:

  1. Use a Google Cloud Platform Connector to access the GKE cluster and all other GCP resources you need.
  2. Set up a Kubernetes Cluster Connector for the GKE cluster. Next, set up a Google Cloud Platform Connector for all other GCP services and resources.

When you set up a deployment in Harness, you will specify Connector to use for the artifact and target cluster. If we use option 2 above, you will select a Google Cloud Platform Connector for the GCR container. Next, you will select Kubernetes Cluster Connector for the target cluster.

Which option you choose will depend on how your teams use Harness.

Permissions Required

A Kubernetes service account with permission to create entities in the target namespace is required.

The set of permissions should include list, get, create, watch (to fetch the pod events), and delete permissions for each of the entity types Harness uses. In general, cluster admin permission or namespace admin permission is sufficient.

When you use the Inherit from selected Delegate option (an in-cluster Delegate) or the Service Account Token setting, Kubernetes RBAC applies. The service account you use must have the Kubernetes cluster-admin role.

For more information, see User-Facing Roles from Kubernetes.

Harness CI Permission Requirements

If you are only using the Kubernetes Cluster Connector for Harness Continuous Integration (CI), you can use a reduced set of permissions.

For Harness CI, the Delegate requires CRUD permissions on Secret, Service, Pod, and PersistentVolumeClaim (PVC).

Here is a same Service Account and RoleBinding that lists the minimum permissions:

apiVersion: v1
kind: Namespace
metadata:
name: cie-test
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cie-test-sa
namespace: cie-test
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: sa-role
namespace: cie-test
rules:
- apiGroups: [""]
resources: ["pods", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: sa-role-binding
namespace: cie-test
subjects:
- kind: ServiceAccount
name: cie-test-sa
namespace: cie-test
roleRef:
kind: Role
name: sa-role
apiGroup: rbac.authorization.k8s.io

Harness CI Cluster Requirements

For Harness Continuous Integration, the resources required for the Kubernetes cluster depends on the number of builds running in parallel, as well as the resources required for each build.

Below is a rough estimation of the resources required, based on the number of daily builds:

PRs/Day

Nodes with 4 CPU, 8GB RAM,100GB disk

Nodes with 8 CPU, 16GB RAM, 200GB disk

100

19 - 26

11 - 15

500

87 - 121

45 - 62

1000

172 - 239

89 - 123

Credential Validation

When you click Submit, Harness uses the provided credentials to list controllers in the default namespace in order to validate the credentials. If validation fails, Harness does not save the Connector and the Submit fails.

If your cluster does not have a default namespace, or your credentials do not have permission in the default namespace, then you can check Skip default namespace validation to skip this check and saving your Connector settings.

You do not need to come back and uncheck Skip default namespace validation.

Later, when you define a target Infrastructure using this Connector, you will also specify a specific namespace. During deployment, Harness uses this namespace rather than the default namespace.

When Harness saves the Infrastructure it performs validation even if Skip default namespace validation was checked.

Name

The unique name for this Connector.

ID

See Entity Identifier Reference.

Description

Text string.

Tags

See Tags Reference.

Cluster Details

Manual or Use a Delegate

Recommended: Install and run the Harness Kubernetes Delegate in the target Kubernetes cluster, and then use the Kubernetes Cluster Connector to connect to that cluster using the Harness Kubernetes Delegate you installed. This is the easiest method to connect to a Kubernetes cluster.

You can select to enter the authentication details of the target cluster or use the role associated with a Harness Delegate.

When you select a Delegate, the Harness Delegate will inherit the Kubernetes service account associated with the Delegate pod.

The service account associated with the Delegate pod must have the Kubernetes cluster-admin role.

See Install a Kubernetes Delegate.

Master URL

The Kubernetes master node URL. The easiest method to obtain the master URL is using kubectl:

kubectl cluster-info

Authentication

Select an authentication method.

Username and Password

Username and password for the Kubernetes cluster. For example, admin or john@example.com, and a Basic authentication password.

You can use an inline username or a Harness Encrypted Text secret.

For the password, select or create a new Harness Encrypted Text secret.

This is not used, typically. Some Connectors have Basic authentication disabled by default. The cluster would need Basic authentication enabled and a specific username and password configured for authentication.
For OpenShift or any other platform, this is not the username/password for the platform. It is the username/password for the cluster.

Service Account

Add the service account token for the service account. The token must be pasted in decoded in the Encrypted Text secret you create/select.

The following shell script is a quick method for obtaining the service account token. Run this script wherever you run kubectl to access the cluster.

Set the SERVICE_ACCOUNT_NAME and NAMESPACE values to the values in your infrastructure.

SERVICE_ACCOUNT_NAME=default
NAMESPACE=mynamepace
SECRET_NAME=$(kubectl get sa "${SERVICE_ACCOUNT_NAME}" --namespace "${NAMESPACE}" -o json | jq -r '.secrets[].name')
TOKEN=$(kubectl get secret "${SECRET_NAME}" --namespace "${NAMESPACE}" -o json | jq -r '.data["token"]' | base64 -d)
echo $TOKEN

The token should have the cluster-admin role. To do this, use the following command. In this example, the service account is named testadmin.

kubectl create clusterrolebinding testadmin --clusterrole=cluster-admin --serviceaccount=default:testadmin

Verify the service account (testadmin) has cluster-admin access using kubectl:

kubectl config set-credentials testadmin --token=$TOKEN

kubectl config set-context --current --user=testadmin

OpenID Connect

These settings come from the OIDC provider authorization server you have set up and others come from the provider app you are using to log in with.

First let's look at the authorization server-related settings:

Master URL

The issuer URI for the provider authentication server.

For example, in Okta, this is the Issuer URL for the Authorization Server:

Providers use different API versions. If you want to identify the version also, you can obtain it from the token endpoint.

In Okta, in the authentication server Settings, click the Metadata URI. Locate the token_endpoint. Use the token_endpoint URL except for the /token part. For example, you would use https://dev-00000.okta.com/oauth2/default/v1 from the following endpoint:

"token_endpoint":"https://dev-00000.okta.com/oauth2/default/v1/token"

OIDC Username and Password

Login credentials for a user assigned to the provider app.

  • OIDC Client ID: Public identifier for the client that is required for all OAuth flows. In Okta, this is located in the Client Credentials for the app:

OIDC Secret

The client secret for the app. For Okta, you can see this in the above picture.

OIDC Scopes

OIDC scopes are used by an application during authentication to authorize access to a user's details, like name and picture. In Okta, you can find them in the Authorization Server Scopes tab:

If you enter multiple scopes, separate them using spaces.

The remaining OIDC Token settings are part of the provider app you are using to log in.

Client Key Certificate

Client Key

Create or select a Harness secret to add the client key for the client certificate. The key can be pasted into the secret either Base64 encoded or decoded.

Client Key passphrase

Create or select a Harness secret to add the client key passphrase. The passphrase can be pasted in either Base64 encoded or decoded.

Client Certificate

Create or select a Harness secret to add the client certificate for the cluster.

The public client certificate is generated along with the private client key used to authenticate. The certificate can be pasted in either Base64 encoded or decoded.

Client Key Algorithm (optional)

Specify the encryption algorithm used when the certificate was created. Typically, RSA.

CA Certificate (optional)

Create or select a Harness secret to add the Certificate authority root certificate used to validate client certificates presented to the API server. For more information, see Authenticating from Kubernetes.

Amazon AWS EKS Support

AWS EKS is supported using the Inherit Delegate Credentials option in the Kubernetes Cluster Connector settings.

To support this, do the following:

Install a Harness Kubernetes Delegate in your EKS cluster.

Give it a name that you can recognize as an EKS cluster Delegate. For information on installing a Kubernetes Delegate, see Install a Kubernetes Delegate.

In the Kubernetes Cluster Connector settings, select the Delegate.

When setting up the EKS cluster as the target Infrastructure, select the Kubernetes Cluster Connector.

OpenShift Support

This section describes how to support OpenShift using a Delegate running externally to the Kubernetes cluster. Harness does support running Delegates internally for OpenShift 3.11 or greater, but the cluster must be configured to allow images to run as root inside the container in order to write to the filesystem.

Typically, OpenShift is supported through an external Delegate installation (shell script installation of the Delegate outside of the Kubernetes cluster) and a service account token, entered in the Service Account setting.

You only need to use the Master URL and Service Account Token setting in the Kubernetes Cluster Connector settings.

The following shell script is a quick method for obtaining the service account token. Run this script wherever you run kubectl to access the cluster.

Set the SERVICE_ACCOUNT_NAME and NAMESPACE values to the values in your infrastructure.

SERVICE_ACCOUNT_NAME=default
NAMESPACE=mynamepace
SECRET_NAME=$(kubectl get sa "${SERVICE_ACCOUNT_NAME}" --namespace "${NAMESPACE}" -o json | jq -r '.secrets[].name')
TOKEN=$(kubectl get secret "${SECRET_NAME}" --namespace "${NAMESPACE}" -o json | jq -r '.data["token"]' | base64 -D)
echo $TOKEN

Once configured, OpenShift is used by Harness as a typical Kubernetes cluster.

OpenShift Notes
  • If you decide to use a username/password for credentials in the Harness Kubernetes Cluster Connector, do not use the username/password for the OpenShift platform. Use the username/password for the cluster.
  • Harness supports DeploymentConfig, Route, and ImageStream across Canary, Blue Green, and Rolling deployment strategies. Please use apiVersion: apps.openshift.io/v1 and not apiVersion: v1.
  • The token does not need to have global read permissions. The token can be scoped to the namespace.
  • The Kubernetes containers must be OpenShift-compatible containers. If you are already using OpenShift, then this is already configured. But be aware that OpenShift cannot simply deploy any Kubernetes container. You can get OpenShift images from the following public repos: https://hub.docker.com/u/openshift and https://access.redhat.com/containers.
  • Useful articles for setting up a local OpenShift cluster for testing: How To Setup Local OpenShift Origin (OKD) Cluster on CentOS 7, OpenShift Console redirects to 127.0.0.1.

YAML Example

connector:
name: Doc Kubernetes Cluster
identifier: Doc_Kubernetes_Cluster
description: ""
orgIdentifier: ""
projectIdentifier: ""
tags: {}
type: K8sCluster
spec:
credential:
type: ManualConfig
spec:
masterUrl: https://00.00.00.000
auth:
type: UsernamePassword
spec:
username: john.doe@example.io
passwordRef: account.gcpexample


Please Provide Feedback