Run Docker-in-Docker in a CI Stage

Updated 2 months ago by Doug Bothwell

You can run Docker-in-Docker (dind) in a CI Stage. This is useful whenever you need to run Docker commands as part of your build process. For example, you can build images from two separate codebases in the same Pipeline: one using a step such as Build and Push an Image to Docker Registry, and another using Docker commands in a Run step.

This topic illustrates a simple build-and-push workflow using Docker-in-Docker.

Docker-in-Docker must run in privileged mode to work properly. You need to be careful because this provides full access to the host environment. See Runtime Privilege and Linux Capabilities in the Docker docs.

Before You Begin

To go through this workflow, you need the following:

Step 1: Set Up the CI Stage

In your Harness Pipeline, click Add Stage. Then click Build.

In the Overview tab for the new Build Stage, configure the Stage as follows:

  • For this example, disable Clone Codebase. You will be cloning a different codebase from the one referenced in the Codebase Object.
  • Under Shared Paths, add the following:
    • /var/run
    • /var/lib/docker
  • Under Advanced, add a Stage Variables for your Docker Hub Personal Access Token and any other fields you want to parameterize. For passwords and Personal Access Tokens, select Secret as the variable type.

Step 2: Define the Build Farm Infrastructure

In the CI stage Infrastructure, define the build farm for the codebase. See Set Up Build Infrastructure.

Step 3: Add a dind Service Dependency

In the Execution tab, click Add Service Dependency and configure the dependency as follows:

  • Dependency Name: dind_Service.
  • Container Registry: A Connector to your Docker registry.
  • Image: The image you want to use, such as docker:dind.
  • Optional Configuration: Select Privileged. This is required for Docker-in-Docker.

Step 4: Configure the Run Step

In the Execution tab, add a Run Step and configure it as follows:

  • Container Registry: A Connector to your Docker registry.
  • Image: The same image you specified for the Service Dependency.
  • Command: Enter the shell commands you want to run in the dind container.

Once the container is started, the software inside the container takes time to initialize and start accepting connections. Give the service adequate time to initialize before trying to connect. You can use a while loop, as shown here:


while ! docker ps ;do 
      echo "Docker not available yet"
done
echo "Docker Service Ready"
docker ps

The following example code clones a Git repo, builds an image, and pushes the image to a Docker registry:

apk add git
git --version
git clone https://github.com/$GITHUB_USERNAME/$GITHUB_REPO
cd $GITHUB_REPO

echo $DOCKERHUB_PAT > my_password.txt
cat my_password.txt | docker login --username $DOCKERHUB_USERNAME --password-stdin

docker build -t $DOCKER_IMAGE_LABEL .
docker tag $DOCKER_IMAGE_LABEL $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
docker push $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>

Step 5: Run the Pipeline

Now you can run your Pipeline. You simply need to select the codebase.

  1. Click Save.
  2. Click Run.
  3. If prompted, specify a Git branch, tag, or PR number.
  4. Click Run Pipeline and check the console output to verify that the Pipeline runs as intended.

Configure As Code: YAML

To configure your pipeline as YAML in CI, go to Harness Pipeline Studio, click YAML. Here’s is a working example of the workflow described in this topic. Modify the YAML attributes such as name, identifiers, codebase, connector refs, and variables as needed.

pipeline:
    name: docker-in-docker-test
    identifier: dockerindockertest
    allowStageExecutions: false
    projectIdentifier:   # your Project Id
    orgIdentifier: default           
    description: test of Trigger and Codebase variables
    tags: {}
    properties:
        ci:
            codebase:
                connectorRef:  # Connector to your GitHub account
                repoName: $GITHUB_REPO          
                build: <+input>
    stages:
        - stage:
              name: Build Alpha
              identifier: Build_Test_and_Push
              type: CI
              spec:
                  cloneCodebase: false
                  infrastructure:
                      type: KubernetesDirect
                      spec:
                          connectorRef:    # Connector to you build infrastructure 
                          namespace: harness-delegate-ng
                          automountServiceAccountToken: true
                  execution:
                      steps:
                          - step:
                                type: Run
                                name: build-alpha-service
                                identifier: echotriggervarscustom
                                spec:
                                    connectorRef:   # Connector to your Docker repo 
                                    image: docker:dind
                                    shell: Sh
                                    command: |
                                        while ! docker ps ;do
                                            echo "Docker not available yet"
                                        done
                                        echo "Docker service is up"
                                        docker ps 

                                        apk add git
                                        git --version
                                        git clone https://github.com/$GITHUB_USERNAME/$GITHUB_REPO
                                        cd $GITHUB_REPO

                                        echo $DOCKERHUB_PAT > my_password.txt
                                        cat my_password.txt | docker login --username $DOCKERHUB_USERNAME --password-stdin
                                        docker build -t $DOCKER_IMAGE_LABEL .
                                        docker tag $DOCKER_IMAGE_LABEL $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
                                        docker push $DOCKERHUB_USERNAME/$DOCKER_IMAGE_LABEL:<+pipeline.sequenceId>
                                    privileged: true
                  serviceDependencies:
                      - identifier: dind_service
                        name: dind_service
                        type: Service
                        description: dind service
                        spec:
                            connectorRef:   # Connector to your Docker repo 
                            image: docker:dind
                            privileged: true
                  sharedPaths:
                      - /var/run
                      - /var/lib/docker
              variables:
                  - name: DOCKERHUB_USERNAME
                    type: String
                    value: # username
                  - name: DOCKERHUB_PAT
                    type: Secret
                    value: # Personal Access Token
                  - name: GITHUB_USERNAME
                    type: String
                    value: # username
                  - name: GITHUB_REPO
                    type: String
                    value: # repo
                  - name: DOCKER_IMAGE_LABEL
                    type: String
                    value: # label

See Also

Please Provide Feedback