CI Pipeline Quickstart

Updated 1 month ago by Manish Jaiswal

This quickstart shows you how to build a simple, two-stage CI Pipeline in Harness. Setting up and running the Pipeline will take about 30 minutes.

The Pipeline will build and run a unit test on a codebase, upload the artifact to Docker Hub, and then run integration tests.

You will use publicly-available code, images, and your Github and Docker Hub accounts.

Objectives

You'll learn how to create a CI Pipeline that does the following:

  1. Clone a code repo for an app.
  2. Use a Kubernetes cluster build farm.
  3. Build the code and run unit tests in the build farm.
  4. Package the app as a Docker image and upload it to Docker Hub.
  5. Pull the uploaded image to the build farm as a service dependency.
  6. Run an integration test against the app.

Before You Begin

Make sure you have the following set up before you begin this quickstart:

  • Github account: This quickstart clones a codebase from a Github repo. You will need a Github account so Harness can connect to Github.
  • Docker Hub account and repo: You will need to push and pull the image you build to Docker Hub. You can use any repo you want, or create a new one for this quickstart.
  • Kubernetes cluster for Harness Delegate and build farm:
    • You'll need a Kubernetes cluster for Harness to use for the Harness Delegate and as a build farm. Ensure you have a cluster that meets the following requirements:
      • Number of pods: 3 (two pods for the Harness Delegate, the remaining pod for the build farm).
      • Machine type: 4vCPU
      • Memory: 16GB RAM. The build farm and Delegate requirements are low but the remaining memory is for Kubernetes, the Docker container, and other default services.
      • Networking: Outbound HTTPS for the Harness connection, and to connect to Docker Hub. Allow TCP port 22 for SSH.
      • Namespace: When you install the Harness Delegate, it will create the harness-delegate namespace. You'll use the same namespace for the build farm.
Google Kubernetes Engine (GKE) Autopilot is not supported at this time.
  • A Kubernetes service account with permission to create entities in the target namespace is required. The set of permissions should include list, get, create, and delete permissions. In general, the cluster-admin permission or namespace admin permission is enough.
    For more information, see User-Facing Roles from Kubernetes.

Visual Summary

Here's an architecture diagram of the very simple setup we use for Harness CI:

You will install the Harness Delegate in the same cluster you use for the build farm. The Delegate will create the namespace harness-delegate. You'll use that namespace for both the Delegate and build farm. You can change it if you like.

Video Overview

Here's a quick video that provides an overview of Harness CI Enterprise:

Option: Use Your Own Code Repo

For this quickstart, we use a codebase located at:

https://github.com/keen-software/goHelloWorldServer

All steps in this quickstart work with any git repo, so you can use your own code repo instead.

Step 1: Start a New Pipeline

Pipelines are a collection of one or more stages. They manage and automate builds, testing, deployments, and other important build and release stages.

  • In Harness, click Projects and create a new Project with the following settings:
    • Name: CI Quickstart
    • Organization: default
    • Invite People to Collaborate: You don't need to add yourself.
  • In the Modules page, click Continuous Integration.
  • Click Pipelines and then Create a Pipeline.
  • Enter the name CI Pipeline and click Start.

As you enter a name for the Pipeline, the ID for the Pipeline is created. A Pipeline name can change, but an ID is permanent. The ID is how you can reference subordinate elements of a Pipeline, such as the names of variables within the Pipeline.

Step 2: Set Up the Build Stage

The "work horse" of most CI Pipelines is the Build Stage. This is where you specify the end-to-end workflow for your build: the codebase to build, the infrastructure to build it, where to post the finished artifact, and any additional tasks (such as automated tests or validations) you want the build to run.

To run a build, a Build Stage needs to connect to the codebase, the build infrastructure, and the artifact repository. A Connector is a configurable object that connects to an external resource automatically.

In this quickstart you'll create a Connector to a GitHub repo. You'll also create a Delegate service that handles communications between Harness and your build infrastructure. Later in this quickstart, you'll create a Connector to a Docker Hub repo so the Build Stage can post the resulting artifact.

Create the Build Stage

  • In Pipeline Studio, click Add Stage and select Build. The About your Stage screen appears.
  • In Stage Name, enter Build Test and Push.
  • Under Configure Codebase, cick Select Connector (under Configure Codebase).

  • In the Create or Select an Existing Connector window, select New Connector.
  • For the Connector type, select GitHub Connector.

Create a Connector to your Codebase

You'll now create a new Connector to the GitHub codebase. Set up the Connector as follows.

Connector Overview
  • Name: CI QuickStart
Connector Details

You can create Connectors for GitHub accounts or specific repos. If you use an account Connector, you specify the repo when you run the build. In this case, we'll connect to a specific repo.

  • URL Type: Repository
  • Connection Type: HTTP
  • GitHub Repository URL: https://github.com/keen-software/goHelloWorldServer.git
Connector Credentials

Now you need to specify the username and Personal Access Token for the Connector to use. Harness secrets are safe. They're stored in the Harness Secret Manager. You can also use your own Secret Manager with Harness.

  • Username: The username for your GitHub account.
  • Personal Access Token: Create a Harness Secret for the Personal Access Token you use for your GitHub repo.
  • Select Enable API Access.
Connector Connectivity Mode

You can choose to establish connections directly from your Harness platform or through a Delegate service running within your corporate firewall.

The Harness Delegate is a local service that connects your infrastructure, collaboration, verification, and other providers with your Harness platform. For this quickstart, you'll install a Delegate in your Kubernetes cluster.

  • Connect to the provider: Connect through a Harness Delegate
Delegate Setup

You should now be in the Delegates Setup screen of the GitHub Connector wizard. Click Install new Delegate.

Delegate Location

You can install the Delegate in different locations. Usually it makes sense to install and run the Delegate on a pod in your Kubernetes build infrastructure. You'll do this in the next steps.

  • Select Kubernetes.
Delegate Details

Now you specify the Delegate name, size and permissions. You can use a small delegate for this quickstart and leave the other options at their defaults.

  • Specify the following:
    • Delegate name: quickstart
    • Delegate size: Small
    • Delegate Permissions: Install Delegate with cluster-wide read/write access
Delegate Install

Harness now generates and displays a workspace-definition YAML file that you can install in your build infrastructure.

  • Click Download Script. This downloads the YAML file for the Kubernetes Delegate.
  • Open a terminal and navigate to where the Delegate file is located. You will connect to your cluster using the terminal so you can simply run the YAML file on the cluster.
  • In the same terminal, log into your Kubernetes cluster. In most platforms, you select the cluster, click Connect, and copy the access command.
  • Install the Harness Delegate using the harness-delegate.yaml file you just downloaded. Click Next in the Harness UI, then run the command shown. For example:
    kubectl apply -f harness-delegate.yaml
    You should see output similar to this:
    % kubectl apply -f harness-delegate.yaml
    namespace/harness-delegate-ng created
    clusterrolebinding.rbac.authorization.k8s.io/harness-delegate-ng-cluster-admin created
    secret/ci-quickstart created
    statefulset.apps/ci-quickstart created
    service/delegate-service created
Connect to the Delegate
  • Return to the Harness UI. It might take a few minutes to verify the Delegate. Once it is verified, close the wizard.
  • Back in Delegates Setup, you can select the new Delegate:
    • In the list of Delegates, you can see your new Delegate and its tags.
    • Select the Connect using Delegates with the following Tags option.
    • Enter the tag of the new Delegate and click Save and Continue.
    • Wait for the connection test to complete and then click Finish.
  • Back in About Your Stage, the Connector and repo are displayed.
  • Click Set Up Stage. The new stage is added to the Pipeline.

Step 3: Define the Build Farm Infrastructure

Here you'll define the build farm infrastructure and the stage steps.

  • Under Select a Kubernetes Cluster, click Select.
  • Click New Connector and set up the new Connector as follows:
    • Name: ci-delegate
    • Details: Select Use the credentials of a specific Harness Delegate. You'll add this Delegate next.
    • Delegates Setup: Select the Kubernetes Delegate you added earlier using its Tags.
    • Connection Test: Wait for the test to finish and then click Finish. The new Connector is added to the Kubernetes Cluster field.
  • In Namespace, enter the namespace harness-delegate-ng. Click Next to proceed to the Execution tab.

Now that the build farm infrastructure is set up, you can run unit tests against your code.

Step 4: Build and Run Unit Tests

Next, we'll add a Run step to the stage that will build the code and run a unit test.

  • You should now be in the Execution tab. Click Add step and then select Run (under Build).

The Run step executes one or more commands on a container image. Configure the step as follows.

  • Name = Run Unit Tests.

Configure Run Step

  • Name: Run Unit Tests
  • Container Registry: Click Select and then +New Connector.
  • Select your Connector Type: Docker Registry

You will now create a new Connector to your DockerHub account as follows.

Docker Connector Overview
  • Name: Docker Quickstart
Docker Connector Details
  • Docker Registry URL: https://index.docker.io/v1/
  • Provider Type: Docker Hub
  • Username: The username for your Docker Hub account
  • Password: Create a Harness Secret for the Personal Access Token you use with your Docker Hub account. You can also use your own Secret Manager.
Delegates Setup
  • Delegates Setup: Select the new Delegate you installed previously using its Tags.
  • Wait for the Connector test to complete and then click Finish.
  • You should now be in the Configure Run Step pane, with the new Connector in the Container Registry setting. Configure the step as follows:
    • Name: Run Unit Tests
    • Container Registry: This should show the Docker Hub Connector you just created.
    • Image: golang:1.15
    • Command:
      go get gotest.tools/gotestsum
      gotestsum --format=standard-verbose --junitfile unit-tests.xml || true
      CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo
      The last line contains the go build command to compile the package, along with its dependencies.
    • Report Paths (Optional Configuration): *.xml
    • Click Apply Changes.
    • In the Pipeline Studio, click Save.

This Run step will intentionally fail the test. This will be useful to see how a test failure is recorded in Harness.

Step 5: Build and Push Image to Docker Hub

Next, you'll add a step to build your container and push it to your Docker Hub repo. You'll need a repo in your Docker Hub account to receive the artifact.

  • You should be in the Execution tab of the Built Test and Push Stage. Click Add step, then click Build and Push an Image to Docker Registry.
  • Configure the step as follows:
    • Name: Build and push an image to Docker Registry
    • Docker Connector: Select the Docker Hub Connector you set up previously.
    • Docker Repository: <your_repo>/ciquickstart
      Replace <your_repo> with the name of the Docker Hub repo you are using for this quickstart.
    • Tags: <+pipeline.sequenceId>
      This tag is a built-in Harness variable that represents the Build ID number, for example Build ID: 9. The pipeline uses the Build ID to tag the image that it pushes in stage 1 and pulls in stage 2. You will see the Build ID when you run the pipeline. You will also this variable to identify the image location when you set up the Configure Service Dependency step in the next stage.
    • Click Apply Changes to return to the Pipeline Studio.
  • Click Save to save the Pipeline. You can run the Pipeline now, or you can continue and add the Integration Test stage.

Step 6: Create the Integration Test Stage

Now you have a Stage to clone, build, containerize, and then push your image to Docker Hub. In this step you'll add a Stage to pull that image, run it in a container, and run integration tests on it.

  • Click Add Stage, and select Build.
  • Enter the name Run Integration Test, disable Clone Codebase, and then click Set Up Stage.

Define Stage Infrastructure

Here you configure the stage to use the same infrastructure as the previous stage.

  • Click Infrastructure.
  • Select Propagate from an existing stage.
  • Select the previous stage. This will use the same Kubernetes cluster and namespace as the first stage.
  • Click Next.

Add Built Image from Stage 1 as a Service Dependency

In the Build Test and Push stage, you built your code and pushed your built image to Docker Hub. Now, in Run Integration Test, you will identify the image as a Service Dependency for your test stage. Harness will pull the image onto the container in your infrastructure. Next, it will start the Hello World Server in the image.

  • In the Run Integration Test stage > Execution tab, click Add Service Dependency.
  • Configure the dependency:
    • Dependency Name: Run Hello World Server
    • Container Registry: The same Docker Hub Connector you used in the Build Test and Push stage.
    • Image: <your_repo>/ciquickstart:<+pipeline.sequenceId>
      Replace <your_repo> with the name of the Docker Hub repo you are using for this quickstart.
  • Click Apply Changes.

Notice that the Image field uses the same variable <+pipeline.sequenceId> for the Image that you used in the previous stage. This tells Harness to pull the image with the same tag as the image pushed previously. Here's an example:

Build and Test and Push (stage 1)

Build and Push to Docker Hub step

Run Integration Test (stage 2)

Configure Service Dependency step

Add Integration Test Step

Next, we can run an integration test. We'll simply test the connection to the server.

  • Click Add Step, and then click Run.
  • Configure the step as follows:
    • Name: test connection to server
    • Container Registry: The Docker Hub Connector you've been using.
    • Image: curlimages/curl:7.73.0
    • Command:
      sleep 10
      curl localhost:8080
      curl localhost:8080?Hello!_I_am_a_nice_demo!
    • Click Apply Changes. You are now back the Pipeline Studio.

Step 7: Run the Pipeline

The Build Pipeline is complete. You can now run it.

  • Click ↑Save.
  • Click Run. The Pipeline Inputs settings appear.
  • Under CI Codebase, select Git branch.
  • In Git Branch, enter the name of the branch where the codebase is, such as main.
  • Click Run Pipeline.

Step 8: View the Results

Click each stage's steps to see the logs in real time.

Click Console View to see more details.

You can see the build number in the Build and push image to Docker Hub step used as an image tag. For example, 11:

--destination=myrepo/ciquickstart:11

In the Initialize step of the Run Integration Test stage, you can see the image with the same tag pulled from your Docker Hub repo:

   8 Pulling image "myrepo/ciquickstart:11"
9 Successfully pulled image "myrepo/ciquickstart:11" in 1.878887425s

Now look in your Docker Hub repo. You can see the image and tag:

Click Tests. You can see the failed test from stage 1.

Click Builds. Your build appears.

Congratulations! You have a CI Pipeline that builds and tests your code.

Using YAML

The entire Pipeline is available as YAML, also.

  • In Builds, click more options (︙) and select Edit Pipeline.
  • Click YAML.

You can see the entire Pipeline as YAML. You can edit anything in the Pipeline and run it again.

pipeline:
name: CI Pipeline
identifier: CI_Pipeline
allowStageExecutions: false
projectIdentifier: CI_QuickStart_20220401
orgIdentifier: default
tags: {}
properties:
ci:
codebase:
connectorRef: CI_QuickStart
build: <+input>
stages:
- stage:
name: Build Test and Push
identifier: Build_Test_and_Push
type: CI
spec:
cloneCodebase: true
infrastructure:
type: KubernetesDirect
spec:
connectorRef: cidelegate
namespace: harness-delegate-ng
execution:
steps:
- step:
type: Run
name: Run Unit Tests
identifier: Run_Unit_Tests
spec:
connectorRef: Docker_Quickstart
image: golang:1.15
command: |-
go get gotest.tools/gotestsum
gotestsum --format=standard-verbose --junitfile unit-tests.xml || true
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo
privileged: false
reports:
type: JUnit
spec:
paths:
- "*.xml"
- step:
type: BuildAndPushDockerRegistry
name: Build and push image to Docker Hub
identifier: Build_and_push_image_to_Docker_Hub
spec:
connectorRef: Docker_Quickstart
repo: dockerhubuser/ciquickstart
tags:
- <+pipeline.sequenceId>
- stage:
name: Run Integration Test
identifier: Run_Integration_Test
type: CI
spec:
cloneCodebase: false
infrastructure:
useFromStage: Build_Test_and_Push
serviceDependencies:
- identifier: Run_Hello_World_Server
name: Run Hello World Server
type: Service
spec:
connectorRef: Docker_Quickstart
image: dockerhubuser/ciquickstart:<+pipeline.sequenceId>
execution:
steps:
- step:
type: Run
name: test connection to server
identifier: test_connection_to_server
spec:
connectorRef: Docker_Quickstart
image: curlimages/curl:7.73.0
command: |-
sleep 10
curl localhost:8080
curl localhost:8080?Hello!_I_am_a_nice_demo!


Please Provide Feedback