Serverless Lambda CD Quickstart

Updated 1 month ago by Michael Cretzman

Currently, this feature is behind the feature flag SERVERLESS_SUPPORT. Contact Harness Support to enable the feature.

This quickstart shows you how to deploy a Serverless Lambda application to AWS Lambda using Harness. We'll use a publicly-available serverless.yaml and artifact and deploy them to your AWS Lambda service using a Harness Pipeline.

New to Serverless? See Tutorial: Your First Serverless Framework Project from Serverless.

Objectives

  1. Add a serverless.yaml file and ZIP artifact to a Harness Pipeline stage.
  2. Define your AWS Lambda service as the deployment target.
  3. Deploy the Serverless application to Lambda.

Before You Begin

Review Harness Key Concepts to establish a general understanding of Harness.
  • GitHub account: this quickstart uses a publicly available serverless.yaml file, but GitHub requires that you use a GitHub account for fetching files.
  • Kubernetes Delegate cluster with Serverless installed: the Harness Delegate is a worker process that performs all deployment tasks. For this quickstart, we'll install a Kubernetes Delegate running in a Kubernetes cluster.
    • You can use a cluster hosted on a cloud platform or run one in minikube using Docker Desktop locally. The installation steps are the same.
    • The Delegate host must have Serverless installed. We'll add the Serverless installation script to the Delegate YAML file later in this quickstart.
  • AWS User account with required policy: Serverless deployments require an AWS User with specific AWS permissions, as described in AWS Credentials from Serverless. To create the AWS User, do the following:
    • Log into your AWS account and go to the Identity & Access Management (IAM) page.
    • Click Users, and then Add user. Enter a name. Enable Programmatic access by clicking the checkbox. Click Next to go to the Permissions page. Do one of the following:
      • Full Admin Access: click on Attach existing policies directly. Search for and select AdministratorAccess then click Next: Review. Check to make sure everything looks good and click Create user.
      • Limited Access: click on Create policy. Select the JSON tab, and add the JSON using the following code from the Serverless gist:
      IAMCredentials.json
      {
      "Statement": [
      {
      "Action": [
      "apigateway:*",
      "cloudformation:CancelUpdateStack",
      "cloudformation:ContinueUpdateRollback",
      "cloudformation:CreateChangeSet",
      "cloudformation:CreateStack",
      "cloudformation:CreateUploadBucket",
      "cloudformation:DeleteStack",
      "cloudformation:Describe*",
      "cloudformation:EstimateTemplateCost",
      "cloudformation:ExecuteChangeSet",
      "cloudformation:Get*",
      "cloudformation:List*",
      "cloudformation:UpdateStack",
      "cloudformation:UpdateTerminationProtection",
      "cloudformation:ValidateTemplate",
      "dynamodb:CreateTable",
      "dynamodb:DeleteTable",
      "dynamodb:DescribeTable",
      "dynamodb:DescribeTimeToLive",
      "dynamodb:UpdateTimeToLive",
      "ec2:AttachInternetGateway",
      "ec2:AuthorizeSecurityGroupIngress",
      "ec2:CreateInternetGateway",
      "ec2:CreateNetworkAcl",
      "ec2:CreateNetworkAclEntry",
      "ec2:CreateRouteTable",
      "ec2:CreateSecurityGroup",
      "ec2:CreateSubnet",
      "ec2:CreateTags",
      "ec2:CreateVpc",
      "ec2:DeleteInternetGateway",
      "ec2:DeleteNetworkAcl",
      "ec2:DeleteNetworkAclEntry",
      "ec2:DeleteRouteTable",
      "ec2:DeleteSecurityGroup",
      "ec2:DeleteSubnet",
      "ec2:DeleteVpc",
      "ec2:Describe*",
      "ec2:DetachInternetGateway",
      "ec2:ModifyVpcAttribute",
      "events:DeleteRule",
      "events:DescribeRule",
      "events:ListRuleNamesByTarget",
      "events:ListRules",
      "events:ListTargetsByRule",
      "events:PutRule",
      "events:PutTargets",
      "events:RemoveTargets",
      "iam:AttachRolePolicy",
      "iam:CreateRole",
      "iam:DeleteRole",
      "iam:DeleteRolePolicy",
      "iam:DetachRolePolicy",
      "iam:GetRole",
      "iam:PassRole",
      "iam:PutRolePolicy",
      "iot:CreateTopicRule",
      "iot:DeleteTopicRule",
      "iot:DisableTopicRule",
      "iot:EnableTopicRule",
      "iot:ReplaceTopicRule",
      "kinesis:CreateStream",
      "kinesis:DeleteStream",
      "kinesis:DescribeStream",
      "lambda:*",
      "logs:CreateLogGroup",
      "logs:DeleteLogGroup",
      "logs:DescribeLogGroups",
      "logs:DescribeLogStreams",
      "logs:FilterLogEvents",
      "logs:GetLogEvents",
      "logs:PutSubscriptionFilter",
      "s3:CreateBucket",
      "s3:DeleteBucket",
      "s3:DeleteBucketPolicy",
      "s3:DeleteObject",
      "s3:DeleteObjectVersion",
      "s3:GetObject",
      "s3:GetObjectVersion",
      "s3:ListAllMyBuckets",
      "s3:ListBucket",
      "s3:PutBucketNotification",
      "s3:PutBucketPolicy",
      "s3:PutBucketTagging",
      "s3:PutBucketWebsite",
      "s3:PutEncryptionConfiguration",
      "s3:PutObject",
      "sns:CreateTopic",
      "sns:DeleteTopic",
      "sns:GetSubscriptionAttributes",
      "sns:GetTopicAttributes",
      "sns:ListSubscriptions",
      "sns:ListSubscriptionsByTopic",
      "sns:ListTopics",
      "sns:SetSubscriptionAttributes",
      "sns:SetTopicAttributes",
      "sns:Subscribe",
      "sns:Unsubscribe",
      "states:CreateStateMachine",
      "states:DeleteStateMachine"
      ],
      "Effect": "Allow",
      "Resource": "*"
      }
      ],
      "Version": "2012-10-17"
      }
      • View and copy the API Key and Secret to a temporary place. You'll need them when setting up the Harness AWS Connector later in this quickstart.

Limitations

  • Harness supports all language runtimes that Serverless supports.
  • Harness supports ZIP file artifacts only.
  • Harness does not support Docker images yet.

Step 1: Create the Deploy Stage

Pipelines are collections of stages. For this quickstart, we'll create a new Pipeline and add a single stage.

Create a Project for your new CD Pipeline: if you don't already have a Harness Project, create a Project for your new CD Pipeline. Ensure that you add the Continuous Delivery module to the Project. See Create Organizations and Projects.

In your Harness Project, click Deployments, and then click Create a Pipeline.

Enter the name Serverless Quickstart and click Start.

Your Pipeline appears.

Click Add Stage and select Deploy.

Enter the name Deploy Service, make sure Service is selected, and then click Set Up Stage.

The new stage settings appear.

In About the Service, click New Service.

Give the Service the name quickstart and click Save.

Let's take a moment and review Harness Services and Service Definitions (which are explained below). Harness Services represent your microservices/apps logically.
You can add the same Service to as many stages as you need. Service Definitions represent your artifacts, manifests, and variables physically. They are the actual files and variable values.
By separating Services and Service Definitions, you can propagate the same Service across stages while changing the artifacts, manifests, and variables with each stage.

Once you have created a Service, it is persistent and can be used throughout the stages of this or any other Pipeline in the Project.

Step 2: Add the Manifest

Next, we can add a serverless.yaml for our deployment. We'll use publicly-available serverless.yaml file available from Harness.

In Service Definition, in Deployment Type, click Serverless Lambda.

In Manifests, click Add Manifest.

Select Serverless Lambda Manifest, and click Continue.

In Specify Serverless Lambda Manifest Store, click GitHub, and then click New GitHub Connector.

The Git Connector settings appear. Enter the following settings.

Name: serverless.

URL Type: Repository.

Connection Type: HTTP.

GitHub Repository URL: https://github.com/wings-software/harness-docs.git.

Username: Enter your GitHub account username.

In Personal Access Token, click Create or Select a Secret.

Click New Secret Text.

In Secret Name, enter a name for the secret like github-pat.

In Secret Value, paste in a GitHub Personal access token.

When you're logged into GitHub, these are typically listed at https://github.com/settings/tokens. For steps on setting up a GitHub PAT, see Creating a personal access token from GitHub.

Ensure you PAT has the repo scope selected:

Select Connect through Harness Platform.

Verify the connection.

Click Finish.

Back in Specify Serverless Lambda Manifest Store, click Continue.

In Manifest Details, enter the following.

Manifest Identifier: serverless.

Git Fetch Type: Latest from Branch.

Branch: main.

Folder Path: serverless/artifacts.

In Advanced, you can see Serverless Config File Path. Use this setting when your Serverless manifest is not named serverless.yml|.yaml|.js|.json. The is the same as the --config option in serverless deploy. See AWS - deploy from Serverless.

The serverless.yaml manifest is added to Harness.

Step 3: Add the Artifact

Currently, Harness supports ZIP file artifacts only. Harness does not support Docker images yet.

Next, we'll add a publicly-available artifact to your Service. The artifact is a zip file with a Javascript function hosted in Artifactory.

We'll add a new Artifactory Connector and install a Harness Kubernetes Delegate in a Kubernetes cluster. The Delegate is a worker process that performs the deployment operations. The Delegate will use the URL and credentials you provide in the Connector to connect to Artifactory and fetch the artifact at runtime.

In Artifact, click Add Primary.

In Specify Artifact Repository Type, click Artifactory, and click Continue.

In Artifactory Repository, click New Artifactory Connector.

In Create or Select an Existing Connector, click New Artifactory Connector.

Enter a name for the Connector, such as JFrog Serverless. Click Continue.

In Details, in Artifactory Repository URL, enter https://harness.jfrog.io/artifactory/.

In Authentication, select Anonymous.

In Delegates Setup, click Install new Delegate.

The Delegate wizard appears.

Click Kubernetes, and then click Continue.

Enter a name for the Delegate, like serverlesslocal, click the Laptop size.

With the Laptop size, you can use a local minikube cluster or a small cluster hosted in a cloud platform. In you use minikube, start minikube with the following resources: minikube start --memory 3g --cpus 2.

Click Continue.

Click Download YAML file. The YAML file for the Kubernetes Delegate will download to your computer.

Installing Serverless on the Delegate

Now we need to edit the YAML to install Serverless when the Delegate pods are created.

Open the Delegate YAML in a text editor.

Locate the Environment variable INIT_SCRIPT in the StatefulSet.

...
- name: INIT_SCRIPT
value: ""
...

Replace the value with the follow Serverless installation script.

...
- name: INIT_SCRIPT
value: |-
#!/bin/bash
echo "Start"
export DEBIAN_FRONTEND=noninteractive
echo "non-inte"
apt-get update
echo "updagte"
apt install -yq npm
echo "npm"
npm install -g [email protected]
echo "Done"
...

Save the YAML file as harness-delegate.yml.

Serverless is installed using the StatefulSet INIT_SCRIPT environment variable instead of simply installing it in the Delegate pods via shell because the StatefulSet ensures that the Delegate pods always have Serverless installed. If a pod goes down and is replaced, the StatefulSet will install Serverless in that new pod.

Install and Register the Delegate

Open a terminal and navigate to where the Delegate file is located.

You will connect to your cluster using the terminal so you can simply run the YAML file on the cluster.

In the same terminal, log into your Kubernetes cluster. In most platforms, you select the cluster, click Connect, and copy the access command.

Next, install the Harness Delegate using the harness-delegate.yml file you just downloaded. In the terminal connected to your cluster, run this command:

kubectl apply -f harness-delegate.yml

You can find this command in the Delegate wizard:

In Harness, click Verify. It will take a few minutes to verify the Delegate. Once it is verified, close the wizard.

Back in Set Up Delegates, in the list of Delegates, you can see your new Delegate and its tags.

Select the Connect using Delegates with the following Tags option.

Enter the tag of the new Delegate and click Save and Continue.

In Connection Test, you can see that the connection is successful. Click Finish.

Add the Artifact

Back in Artifactory Repository, click Continue.

Enter the following artifact settings and click Submit. The following image shows how the Artifactory settings corresponded to Artifact Details.

Repository: lambda.

Artifact Directory: serverless.

Artifact Details: Value.

Artifact Path: handler.zip.

When you click one of the settings, the Harness Delegate fetches artifact metadata from Artifactory.

Click Submit.

The artifact is added to the Service.

Now the Service is configured, we can define the target for our deployment.

Click Continue to view the Infrastructure.

Step 3: Define Amazon Web Services Deployment Target

In Infrastructure, we'll add an AWS Connector to connect Harness with your Lambda service.

In Infrastructure Details, in Specify your environment, click New Environment. Just like with a Service, you can create a new Environment or selecting an existing one. We'll create a new one.

In New Environment, enter a name, select Pre-Production, and click Save. The new Environment appears.

In Infrastructure Definition, click AWS.

In Amazon Web Services Details, click in Connector.

In Create or Select an Existing Connector, click New Connector.

Enter the following and click Save and Continue.

Name: AWS Serverless.

Credentials: AWS Access Key.

Enter the AWS access key for the AWS User you created with the required policies in Before You Begin.

Enter the secret key as a Harness Text Secret.

The Harness Delegate will use these credentials to authenticate Harness with AWS at deployment runtime.

Delegates Setup: Only use Delegates with all of the following tags.

Select the Delegate you added earlier in this quickstart.

The Connection Test verifies the connection. Click Finish.

Back in Amazon Web Services Details, in Region, enter the region for your AWS Lambda service, such as us-east-1.

In Stage, enter the name of the stage in your service that you want to deploy to, such as dev. This is the same as the --stage option in the serverless deploy command.

When you run your deployment, you will see these settings used in logs. For example: serverless deploy list --stage dev --region us-east-1.

Click Continue. The Execution steps appear.

Step 4: Add a Serverless AWS Lambda Deploy Step

In Execution, you add the steps that define how Harness deploys your Serverless Lambda service.

Harness automatically add two Serverless Lambda steps to Execution:

  • Serverless Lambda Deploy: this step performs the deployment.
  • Serverless Lambda Rollback: this step performs rollback in the event of a deployment failure. To see this step, toggle the Execution/Rollback setting.

In Execution, click Serverless Lambda Deploy.

Click the Advanced tab and select the Delegate that you installed in Delegate Selector.

If you only have one Delegate installed in your Project, then this isn't necessary. But if you have multiple Delegates, you want to make sure the Serverless Lambda Deploy step uses the Delegate where you installed Serverless.

Click Apply Changes.

Now you're ready to deploy.

Serverless Deploy Command Options

In Serverless Deploy Command Options, you can add any serverless deploy command options. See the Serverless AWS - Deploy doc for the list of options.

In the deployment logs, you will see the options added to the serverless deploy command.

For example, if you add --conceal in Serverless Deploy Command Options you will see the following:

serverless deploy --stage dev --region us-east-1 --conceal

Step 5: Deploy and Review

Save you Pipeline and then click Run, and then Run Pipeline.

The Pipeline executes.

In the Serverless AWS Lambda Deploy step, click Input to see the deployment inputs:

Click Output to see what is deployed:

Click Details or Console View to see the logs.

In the logs you can see the successful deployment.

Deploying..

Serverless Deployment Starting..
serverless deploy --stage dev --region us-east-1
Serverless: Deprecation warning: Support for Node.js versions below v12 will be dropped with next major release. Please upgrade at https://nodejs.org/en/
More Info: https://www.serverless.com/framework/docs/deprecations/#OUTDATED_NODEJS
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service artifactFile file to S3 (721 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: myfunction
stage: dev
region: us-east-1
stack: myfunction-dev
resources: 11
api keys:
None
endpoints:
GET - https://85h6zffizc.execute-api.us-east-1.amazonaws.com/tello
functions:
hello: myfunction-dev-hello
layers:
None

Deployment completed successfully.

Congratulations! You have successfully deployed a function using Serverless Lambda and Harness.

Clean Up

To delete the Harness Delegate from your Kubernetes cluster, you delete the StatefulSet for the Delegate. Once created, the StatefulSet ensures that the desired number of pods are running and available at all times. Deleting the pod without deleting the StatefulSet will result in the pod being recreated.

For example, if you have the Delegate pod name quickstart-vutpmk-0, you can delete the StatefulSet with the following command:

$ kubectl delete statefulset -n harness-delegate-ng quickstart-vutpmk

Note that the -0 suffix in the pod name is removed for the StatefulSet name.

Notes

Now that you're done the quickstart, here is some more information to help you extend your Harness Serverless Lambda deployments.

Serverless Manifest Supports Harness Secrets and Expressions

The serverless.yaml file you use with Harness can use Harness secret and built-in expressions.

Expression support lets you take advantage of Runtime Inputs and Input Sets in your serverless.yaml files. For example, you could use a Stage variable as a runtime input to change plugins with each stage deployment:

service: <+service.name>
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- <+stage.variables.devplugins>

See:

Rollback Timestamps

In a Serverless CLI rollback (serverless rollback --timestamp timestamp), you would have to manually identify and select the timestamp of the last successful deployment. This can be difficult because you need to know which timestamp to use. With multiple developers deploying, there is the possibility of rolling back to the wrong version.

Harness avoids this issue by automatically identifying the last successful deployment using its timestamp. During the event of a rollback, Harness will automatically rollback to that deployment.

You can see the timestamps in the deployment logs:

Serverless: Listing deployments:
Serverless: -------------
Serverless: Timestamp: 1653065606430
Serverless: Datetime: 2022-05-20T16:53:26.430Z
Serverless: Files:
Serverless: - compiled-cloudformation-template.json
Serverless: - myfunction.zip
Serverless: -------------
Serverless: Timestamp: 1653344285842
Serverless: Datetime: 2022-05-23T22:18:05.842Z
Serverless: Files:
Serverless: - artifactFile
Serverless: - compiled-cloudformation-template.json
Serverless: -------------
Serverless: Timestamp: 1653415240343
Serverless: Datetime: 2022-05-24T18:00:40.343Z
Serverless: Files:
Serverless: - artifactFile
Serverless: - compiled-cloudformation-template.json

If there is no change in code, Serverless doesn't deploy anything new. In the logs you will see Serverless: Service files not changed. Skipping deployment....

While this is somewhat similar to how rollback is performed in Serverless CLI, but Harness performs rollback automatically and always uses the timestamp of last successful deployment.

During a Harness rollback, you can see the timestamp used to rollback to the last successful deployment (rollback --timestamp 1653415240343 --region us-east-1 --stage dev):

Rollback..

Serverless Rollback Starting..
serverless rollback --timestamp 1653415240343 --region us-east-1 --stage dev
Serverless: Deprecation warning: Support for Node.js versions below v12 will be dropped with next major release. Please upgrade at https://nodejs.org/en/
More Info: https://www.serverless.com/framework/docs/deprecations/#OUTDATED_NODEJS
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Updating Stack...

Rollback completed successfully.

Versioning

Serverless Lambda deployments are versioned using the timestamp of their deployment. This versioning has no relation to the versioning in AWS Lambda.

Plugin Support

Harness supports Serverless plugins in your serverless.yaml file.

You simply add the plugin using the standard Serverless plugins format and Harness adds the plugin at runtime.

service: <+service.name>
frameworkVersion: '2 || 3'

provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
events:
- httpApi:
path: /tello
method: get
package:
artifact: <+artifact.path>
plugins:
- [email protected]


Please Provide Feedback