Verify Deployment with Prometheus

Updated 3 weeks ago by Renu Bhaskaran

Harness CV integrates with Prometheus to:

  • Verify that the deployed service is running safely and performing automatic rollbacks.
  • Apply machine learning to every deployment to identify and flag anomalies in future deployments.

This topic covers how to add and configure Prometheus as a Health Source for the Verify step.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

In this topic:

Before You Begin

Add Prometheus as a verification provider

Review: CV Setup Options

To use the Verify step, you will need a Harness Change Intelligence Monitoring Source. In the simplest terms, a Change Intelligence Monitoring Source is basically a mapping of a Harness Service to a service monitored by your APM or logging tool.

You can set up a Change Intelligence Monitoring Source in the Change Intelligence module or in the Verify step in a CD stage. The process is the same.

No matter where you set up the Monitoring Source, once it's set up, it's available to both Change Intelligence and CD modules.

In this topic, we set up the Change Intelligence Monitoring Source as part of the Verify step setup.

Step 1: Add Verify Step

There are two ways to add the Verify step:

  • When selecting the stage deployment strategy:
    The Verify step can be enabled in a CD stage the first time you open the Execution settings and select the deployment strategy. When you select the deployment strategy you want to use, there is also an Enable Verification option. Select the Enable Verification option.
    Harness will automatically add the Verify step. For example, here is a stage where Canary strategy and the Enable Verification option were selected.
  • Add the Verify step to an existing Execution setup: You can also add the Verify step to the Execution section of a CD stage in a Pipeline you previously created. Simply click Add Step after the deployment step, and then select Verify.

Step 2: Enter a Name and Timeout

In Name, enter a name for the step.

In Timeout, enter a timeout value for the step.

You can use:

  • w for weeks
  • d for days
  • h for hours
  • m for minutes
  • s for seconds
  • ms for milliseconds

The maximum is 53w.Timeouts can be set at the Pipeline level also.

Step 3: Select a Continuous Verification Type

In Continuous Verification Type, select a type that matches your deployment strategy.

Step 4: Create a Monitored Service

In Monitored Service, click Click to autocreate a monitored service.

Harness automatically creates a Monitored Service using a concatenation of the Service and Environment names. For example, a Service named todolist and an Environment named dev will result in a Monitored Service named todolist_dev.

If the stage Service or Environment settings are Runtime Inputs, the Monitored Service and Health Sources settings will show up in the Runtime Input settings when you run the Pipeline.

Step 5: Add Health Sources

A Health Source is basically a mapping of a Harness Service to the service in a deployment environment monitored by an APM or logging tool.

In Health Sources, click Add. The Add New Health Source settings appear.

  1. In Select health source type, select Prometheus.
  2. In Health Source Name, enter a name for the Health Source.
  3. Under Connect Health Source, click Select Connector.
  4. In Connector settings, you can either choose an existing connector or click New Connector.
  1. After selecting the connector, click Apply Selected. The Connector is added to the Health Source.
  1. In Select Feature, select the Prometheus feature to be used.
  2. Click Next. The Customize Health Source settings appear.
The subsequent settings in Customize Health Source depend on the Health Source Type you selected. 

You can customize the metrics to map the Harness Service to the monitored environment.

In Query Specifications and Mapping settings, do the following:

  1. Click Map Metric(s) to Harness Services drop down.
  2. Enter a name for the metric in Metric Name.
  3. Enter a name for the Prometheus group in Group Name.
  4. Click Build your Query drop down.
  5. In Prometheus Metric, select the desired metric.
  6. In Filter on Environment, select a filter.
  7. In Filter on Service, select a filter. To add more filters, click Additional Filter which is optional.
  8. To add an aggregator for the metric, click Aggregator which is also optional.
  9. Click Risk Profile drop down.
  10. In Risk Category, select a risk type.
  11. In Deviation Compared to Baseline, select one of the options based on the selected risk type.

Once the records are fetched, click Submit. The Health Source is displayed in the Verify step.

You can add one or more Health Sources for each APM or logging provider.

Step 6: Select Sensitivity

In Sensitivity, select High, Medium, or Low based on the risk level used as failure criteria during the deployment.

Step 7: Select Duration

Select how long you want Harness to analyze and monitor the logs/APM data points. Harness waits for 2-3 minutes to allow enough time for the data to be sent to the APM/logging tool before it analyzes the data. This wait time is a standard with monitoring tools.

The recommended Duration is 10 min for logging providers and 15 min for APM and infrastructure providers.

Step 8: Specify Artifact Tag

In Artifact Tag, use a Harness expression to reference the artifact in the stage Service settings.

The expression <+serviceConfig.artifacts.primary.tag> refers to the primary artifact.

Option: Advanced Settings

In Advanced, you can select the following options:

See Advanced Settings.

Step 9: Deploy and Review Results

After setting up the Verify step, click Apply Changes.

Click Run to run the pipeline.

In Run Pipeline, select the tag for the artifact if a tag was not added in the Artifact Details settings.

Click Run Pipeline.

When the Pipeline is running, click the Verify step.

You can see that the verification takes a few minutes.

Once verification is complete, the Verify step shows the following:

The risk level might initially display a number of violations, but the red and orange colored host often change to green over the duration.

Summary

The Summary section shows the number of logs that are in violation.

Console View

Click Console View or simply click View Details in Summary to take a deeper look at verification.

You can use the search option to search for any specific metric or transaction you want.

Please Provide Feedback