CI Pipeline Quickstart

Updated 1 week ago by Manish Jaiswal

This quickstart shows you how to build a simple, two-stage CI Pipeline in Harness. Setting up and running the Pipeline will take about 30 minutes.

The Pipeline will build and run a unit test on a codebase, upload the artifact to Docker Hub, and then run integration tests.

You will use publicly-available code, images, and your Github and Docker Hub accounts.

Objectives

You'll learn how to create a CI Pipeline that does the following:

  1. Clone a code repo for an app.
  2. Build the code using a Kubernetes cluster build farm.
  3. Run unit tests in the build farm.
  4. Package the app as a Docker image and upload it to Docker Hub.
  5. Pull the uploaded image to the build farm as a service dependency.
  6. Run an integration test against the app.

Before You Begin

Make sure you have the following set up before you begin this quickstart:

  • Github account: This quickstart clones a codebase from a Github repo. You will need a Github account so Harness can connect to Github.
  • Docker Hub account and repo: You will need to push and pull the image you build to Docker Hub. You can use any repo you want, or create a new one for this quickstart.
  • Kubernetes cluster for Harness Delegate and build farm:
    • You'll need a Kubernetes cluster for Harness to use for the Harness Delegate and as a build farm. Ensure you have a cluster that meets the following requirements:
      • Number of pods: 3 (two pods for the Harness Delegate, the remaining pod for the build farm).
      • Machine type: 4vCPU
      • Memory: 16GB RAM. The build farm and Delegate requirements are low but the remaining memory is for Kubernetes, the Docker container, and other default services.
      • Networking: Outbound HTTPS for the Harness connection, and to connect to Docker Hub. Allow TCP port 22 for SSH.
      • Namespace: When you install the Harness Delegate, it will create the harness-delegate namespace. You'll use the same namespace for the build farm.
  • A Kubernetes service account with permission to create entities in the target namespace is required. The set of permissions should include list, get, create, and delete permissions. In general, the cluster-admin permission or namespace admin permission is enough.
    For more information, see User-Facing Roles from Kubernetes.

Visual Summary

Here's an architecture diagram of the very simple setup we use for Harness CI:

You will install the Harness Delegate in the same cluster you use for the build farm. The Delegate will create the namespace harness-delegate. You'll use that namespace for both the Delegate and build farm. You can change it if you like.

Video Overview

Here's a quick video that provides an overview of Harness CI Enterprise:

Option: Clone the Codebase Repo

For this quickstart, we use a codebase located at:

https://github.com/keen-software/goHelloWorldServer

You can use this repo for the quickstart or clone it and use the clone. You will connect Harness to the repo or the clone.

You can use your own code repo instead. All the steps in this quickstart will work with any code repo in Git.

Step 1: Start a New Pipeline

Pipelines are a collection of one or more stages. They manage and automate builds, testing, deployments, and other important build and release stages.

In Harness, click Projects.

Click + Project to start a new Project.

Enter the following settings and then save:

  • Name: CI Quickstart
  • Organization: default
  • Invite People to Collaborate: You do not need to add yourself.

Your new project is listed.

Click the new Project.

In Enable More Modules, in Continuous Integration, click Enable.

The module is added.

In the module, click Create New Pipeline. Pipeline Studio appears.

Enter the name CI Pipeline and click Start.

As you enter a name for the Pipeline, the ID for the Pipeline is created. A Pipeline name can change, but an ID is permanent. The ID is how you can reference subordinate elements of a Pipeline, such as the names of variables within the Pipeline.

Next, we'll add a stage to the Pipeline to clone a codebase.

Step 2: Clone the Codebase

In this step we'll do the following as part of cloning the codebase:

  • Install the Harness Delegate.
  • Create the GitHub Connector.
  • Connect Harness to the GitHub repo with the codebase.

In Pipeline Studio, click Add Stage, and then click Build.

In Stage Name, enter Build Test and Push.

In Configure Codebase, in Connector, click Select.

Click New Connector.

Click GitHub Connector.

In GitHub Connector, in Name, enter the name CI Quickstart, and then click Continue.

Enter the following settings and click Continue through each step.

  • URL Type: Repository
  • Connection Type: HTTP
  • GitHub Account URL:
    https://github.com/keen-software/goHelloWorldServer.git
  • Username and Password:

Enter the username and password for your GitHub account.

You'll need to create a Harness Secret for the password. Harness secrets are safe. They are stored in the Harness Secret Manager.

You can always use your own Secret Manager with Harness.

In Set Up Delegates, click Install new Delegate.

The Delegate wizard appears.

Click Kubernetes, and then click Continue.

Enter a name for the Delegate, like quickstart, click the Small size.

In Delegate Configurations, select Primary Configuration. This is simply a default. In the future, you can add Delegate Configuration to run scripts on your Delegates and scope them to different Environments.

Click Continue.

Click Download Script. The YAML file for the Kubernetes Delegate will download to your computer as an archive.

Open a terminal and navigate to where the Delegate file is located.

You will connect to your cluster using the terminal so you can simply run the YAML file on the cluster.

In the same terminal, log into your Kubernetes cluster. In most platforms, you select the cluster, click Connect, and copy the access command.

Next, install the Harness Delegate using the harness-delegate.yaml file you just downloaded. In the terminal connected to your cluster, run this command:

kubectl apply -f harness-delegate.yaml

You can find this command in the Delegate wizard:

The successful output is something like this:

% kubectl apply -f harness-delegate.yaml
namespace/harness-delegate unchanged
clusterrolebinding.rbac.authorization.k8s.io/harness-delegate-cluster-admin unchanged
secret/k8s-quickstart-proxy unchanged
statefulset.apps/k8s-quickstart-sngxpn created
service/delegate-service unchanged

In Harness, click Verify. It will take a few minutes to verify the Delegate. Once it is verified, close the wizard.

Back in Set Up Delegates, you can select the new Delegate.

In the list of Delegates, you can see your new Delegate and its tags.

Select the Connect using Delegates with the following Tags option.

Enter the tag of the new Delegate and click Save and Continue.

When you are done, the Connector is tested.

Select the new Connector and click Apply Selected.

Back in About Your Stage, the Connector and repo are displayed.

Click Set Up Stage. The new stage is added to the Pipeline.

Step 3: Define the Build Farm Infrastructure

Here you'll define the build farm infrastructure and the stage steps.

In the Build Test and Push stage, click Infrastructure.

In Select a Kubernetes Cluster, click Select.

Click New Connector.

Name the Connector cidelegate, and click Continue.

In Details, select Use the credentials of a specific Harness Delegate. We will add this Delegate next.

Click Continue.

Select the Kubernetes Delegate you added earlier using its Tags, and then click Save and Continue.

Harness verifies the Connector.

Click Finish.

The new Connector is added to Select a Kubernetes Cluster.

In Namespace, enter the namespace harness-delegate.

Click Next.

You can ignore the Dependencies step for this stage.

Now that the build farm infrastructure is set up, you can run unit tests against your code.

Step 4: Run Unit Tests

Click Add step, and then click Run.

The Run step executes one or more commands on a container image.

In Name, enter Run Unit Tests.

In Container Registry, click Select.

Click New Connector.

Select Docker Registry.

In Name, enter Docker Quickstart.

Enter the following settings and click Continue.

  • Name: Docker Quickstart.
  • Docker Registry URL: https://index.docker.io/v1/
  • Provider Type: Docker Hub
  • Username and password:
    Enter the username and password for your Docker Hub account.
  • You cannot use Anonymous because you will push to Docker Hub.

Back in Set Up Delegates, you can select the new Delegate you added earlier.

In the list of Delegates, you can see your new Delegate and its tags.

Select the Connect using Delegates with the following Tags option.

Enter the tag of the new Delegate and click Save and Continue.

When you are done, the Connector is tested.

Click Finish.

Select the new Connector and click Apply Selected. The new Connector is in the Run step Container Registry setting.

In the Run step, enter the following and click Apply Changes.

  • Image: golang:1.15
  • Command:
go get gotest.tools/gotestsum
gotestsum --format=standard-verbose --junitfile unit-tests.xml || true
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo
  • Report Paths: *.xml

This Run step will intentionally fail the test. That is why || true is added. It will be useful to see how a test failure is recorded in Harness.

Step 5: Build and Push Image to Docker Hub

Next, we'll add a step to build our codebase and push it to your Docker Hub repo. You'll need a repo in your Docker Hub account to receive the artifact.

Click Add step, and then click Build and Push an Image to Docker Registry.

Enter the following settings and click Apply Changes.

Replace <your_repo> with the name of the Docker Hub repo you are using for this quickstart.

  • Step Name: Build and push image to DockerHub
  • Docker Hub Registry: The Docker Hub Connector you set up earlier.
  • Repository: <your_repo>/ciquickstart
  • Tags: <+pipeline.sequenceId>

The <+pipeline.sequenceId> you added as an image tag is used to identify this image in the next stage.

In the Configure Service Dependency step in the next stage, you identify the image location using <+pipeline.sequenceId>.

The <+pipeline.sequenceId> is a built-in Harness variable that represents the Build ID number, for example Build ID: 9.

When the Pipeline is run you will see a Build ID:

And this Build ID will be used to tag the image you push in stage 1 of your Pipeline, and pull in stage 2 of your Pipeline.

Click Save to save the Pipeline. You can run the Pipeline now, or you can continue and add the Integration Test stage.

Step 6: Create the Integration Test Stage

Now we have a stage to clone, test, build or code, and then push our image to Docker Hub.

Next, we'll add a stage to pull that image, run it in a container, and run integration tests on it.

Click Add Stage, and select Build.

Enter the name Run Integration Test, disable Clone Codebase, and then click Set Up Stage.

Define Stage Infrastructure

We'll define the stage's infrastructure the same way we did in stage 1.

Click Infrastructure.

Select Propagate from an existing stage.

Select the previous stage.

This will use the same Kubernetes cluster and namespace as the first stage.

Click Next.

Add Built Image from Stage 1 as a Service Dependency

In stage 1, we built our code and pushed our built image to Docker Hub.

Now, in stage 2, we will identify the image as a Service Dependency for our test stage.

Harness will pull the image onto the container in the stage 2 infrastructure. Next, it will start the Hello World Server in the image.

Click Add service dependency.

Enter the following settings and click Apply Changes.

Replace <your_repo> with the name of the Docker Hub repo you are using for this quickstart.

  • Name: Run Hello World Server
  • Container Registry: The same Docker Hub Connector you used in stage 1.
  • Image:
<your_repo>/ciquickstart:<+pipeline.sequenceId>

Notice we use the same expression <+pipeline.sequenceId> in Image that we used the Tags setting of stage 1's Build and Push to DockerHub step.

This tells Harness to pull the image with the same tag as the image we pushed in stage 1's Build and Push to DockerHub step. Here's an example:

Build and Push step from stage one

Configure Service Dependency step from stage two

Add Integration Test Step

Next, we can run an integration test. We'll simply test the connection to the server.

Click Add Step, and then click Run.

Enter the following settings and click Apply Changes.

  • Name: test connection to server
  • Container Registry: The Docker Hub Connector you've been using.
  • Image: curlimages/curl:7.73.0
  • Command:
sleep 10
curl localhost:8080
curl localhost:8080?Hello!_I_am_a_nice_demo!

The Build Pipeline is complete. You can now run it.

Step 7: Run the Pipeline

Now you can run your Pipeline.

Click Save and Publish.

Click Run. The Pipeline Inputs settings appear.

In CI Codebase, click Git branch.

In Git Branch, enter the name of the branch where the codebase is, such as main.

Click Run Pipeline.

Step 8: View the Results

Click each stage's steps to see the logs in real-time.

Click Console View to see more details.

You can see the build number in the Build and push image to Docker Hub step used as an image tag. For example, 11:

--destination=myrepo/ciquickstart:11

In the Initialize step of the Run Integration Test stage, you can see the image with the same tag pulled from your Docker Hub repo:

   8 Pulling image "myrepo/ciquickstart:11"
9 Successfully pulled image "myrepo/ciquickstart:11"

Now look in your Docker Hub repo. You can see the image and tag:

Click Tests.

You can see the failed test from stage 1.

Click Builds. Your build appears.

Congratulations! You have a CI Pipeline that builds and tests your code.

Using YAML

The entire Pipeline is available as YAML, also.

In Builds, click more options (︙) and select Edit Pipeline.

Click YAML.

You can see the entire Pipeline as YAML. You can edit anything in the Pipeline and run it again.


Please Provide Feedback