Set Up Cloud Cost Management for AWS

Updated 2 weeks ago by Archana Singh

Harness Cloud Cost Management (CCM) monitors and provides visibility into the cloud costs of your Amazon Web Services (AWS) across your cloud infrastructure and AWS services, such as EC2, S3, RDS, Lambda, and so on. CCM also allows you to optimize your instances, auto-scaling groups (ASGs), and EKS clusters using intelligent cloud AutoStopping rules.

You can set up CCM for your AWS resources, in a simple two-step process:

  1. Create a Cost and Usage Report (CUR). Harness CCM uses a secure, cross-account role with a restricted policy to access the cost and usage reports and resources for cost analysis.
  2. Create a Cloudformation stack to provision IAM Roles and corresponding policies to grant access for the required features. CCM offers the following features:

    Cost Visibility (Required)

    This feature is available by default and requires access to the CUR report. Provides the following capabilities:

    • Insights into AWS costs by services, accounts, etc.
    • Root cost analysis using cost perspectives
    • Cost anomaly detection
    • Governance using budgets and forecasts
    • Alert users using Email and Slack notification

    AWS ECS and Resource Inventory Management (Optional)

    This feature provides visibility into your EC2, EBS volumes, and ECS costs. The insights provided by inventory management can be consumed by Finance teams to understand the resource utilization across the board.

    AWS resource optimization using AutoStopping rules (Optional)

    This feature allows you to enable Intelligent Cloud AutoStopping for your AWS instances and auto-scaling groups. For more information, see Create AutoStopping Rules for AWS.

    • Orchestrate VMs and ASGs based on idleness
    • Run your workloads on fully orchestrated spot instances
    • Provides Granular savings visibility

    Cloudformation template has policies corresponding to all the permissions (Visibility, Inventory, and Optimization). However, it is important to note that the permissions (policies) of the selected features will only be applied.
After enabling CCM, it takes about 24 hours for the data to be available for viewing and analysis.

In this topic:

Before You Begin

Review: AWS Connector Requirements

  • For CCM, AWS connectors are available only at the Account level in Harness.
  • You can create an AWS connector in the master or linked account. CCM requires one connector per AWS account (master or linked).

Review: Cost and Usage Reports (CUR) and CCM Requirements

  • If you have a consolidated billing process enabled, then CCM needs read-only access to the cost and usage reports (CUR) stored in the S3 bucket in the master or payer account. This gives access to the cost data for all the accounts (linked/member) in the organization.
  • If you don't have consolidated billing enabled at the organization level then you can create the CUR at a linked account level.
  • If you have provided CUR access to the master account then you do not need to provide billing details for each linked account. CCM requires one connector per AWS account (master or linked).

    It is recommended to create a CUR at the master account to avoid the CUR creation step for each linked account.
  • If you do not have access to the master account, you can create an AWS connector in the linked account for which you have the required access.
  • If you have created a billing report for your AWS account ID once then you can use the same CUR again for the AWS connector. You do not need to create CUR again for the same account.

Review: AWS Access Permissions

CCM requires the following permissions:

Cost Visibility

The cost visibility policy performs the following actions:

  • List CUR reports and visibility into the organization Structure
  • Get objects from the S3 bucket configured in the CUR
  • Put objects into Harness S3 bucket

      HarnessBillingMonitoringPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Condition: CreatingHarnessBillingMonitoringPolicy
    Properties:
    Description: Policy granting Harness Access to Collect Billing Data
    PolicyDocument:
    Version: 2012-10-17
    Statement:
    - Effect: Allow
    Action:
    - 's3:GetBucketLocation'
    - 's3:ListBucket'
    - 's3:GetObject'
    Resource:
    - !Join
    - ''
    - - 'arn:aws:s3:::'
    - !Ref BucketName
    - !Join
    - /
    - - !Join
    - ''
    - - 'arn:aws:s3:::'
    - !Ref BucketName
    - '*'
    - Effect: Allow
    Action:
    - 's3:ListBucket'
    - 's3:PutObject'
    - 's3:PutObjectAcl'
    Resource:
    - 'arn:aws:s3:::ce-customer-billing-data-prod*'
    - 'arn:aws:s3:::ce-customer-billing-data-prod*/*'
    - Effect: Allow
    Action:
    - 'cur:DescribeReportDefinitions'
    - 'organizations:Describe*'
    - 'organizations:List*'
    Resource: "*"
    Roles:
    - !Ref HarnessCloudFormationRole

AWS ECS and Resource Inventory Management

The inventory management policy performs the following actions:

  • ECS Visibility - For Granular Cluster Cost Breakdown
  • EC2, EBS Visibility - Inventory Management
    HarnessEventsMonitoringPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Condition: CreateHarnessEventsMonitoringPolicy
    Properties:
    Description: Policy granting Harness Access to Enable Event Collection
    PolicyDocument:
    Version: 2012-10-17
    Statement:
    - Effect: Allow
    Action:
    - 'ecs:ListClusters*'
    - 'ecs:ListServices'
    - 'ecs:DescribeServices'
    - 'ecs:DescribeContainerInstances'
    - 'ecs:ListTasks'
    - 'ecs:ListContainerInstances'
    - 'ecs:DescribeTasks'
    - 'ec2:DescribeInstances*'
    - 'ec2:DescribeRegions'
    - 'cloudwatch:GetMetricData'
    - 'ec2:DescribeVolumes'
    - 'ec2:DescribeSnapshots'
    Resource: '*'
    Roles:
    - !Ref HarnessCloudFormationRole

AWS Resource Optimization Using AutoStopping Rules

The AutoStopping policy performs the following actions:

  • Create an IAM role for optimization
  • Permissions for creating AutoStopping Rules
     HarnessOptimizationLambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Condition: CreateHarnessOptimisationPolicy
    Properties:
    RoleName: !Ref LambdaExecutionRoleName
    AssumeRolePolicyDocument:
    Version: 2012-10-17
    Statement:
    - Effect: Allow
    Principal:
    Service: "lambda.amazonaws.com"
    Action: 'sts:AssumeRole'
    Path: /ce-optimization-service-role/

    HarnessOptimsationLambdaPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Condition: CreateHarnessOptimisationPolicy
    Properties:
    Description: Policy granting Harness Access to Enable Cost Optimisation
    PolicyDocument:
    Version: 2012-10-17
    Statement:
    - Effect: Allow
    Action:
    - 'ec2:CreateNetworkInterface'
    - 'ec2:CreateNetworkInsightsPath'
    - 'ec2:CreateNetworkInterfacePermission'
    - 'ec2:CreateNetworkAcl'
    - 'ec2:*'
    - 'ec2:CreateNetworkAclEntry'
    - 'logs:CreateLogGroup'
    - 'logs:CreateLogStream'
    - 'logs:PutLogEvents'
    Resource: "*"
    Roles:
    - !Ref HarnessOptimizationLambdaExecutionRole

    HarnessOptimisationPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Condition: CreateHarnessOptimisationPolicy
    Properties:
    Description: Policy granting Harness Access to Enable Cost Optimisation
    PolicyDocument:
    Version: 2012-10-17
    Statement:
    - Effect: Allow
    Action:
    - elasticloadbalancing:*
    - ec2:StopInstances
    - autoscaling:*
    - ec2:Describe*
    - iam:CreateServiceLinkedRole
    - iam:ListInstanceProfiles
    - iam:ListInstanceProfilesForRole
    - iam:AddRoleToInstanceProfile
    - iam:PassRole
    - ec2:StartInstances
    - ec2:*
    - iam:GetUser
    - ec2:ModifyInstanceAttribute
    - iam:ListRoles
    - acm:ListCertificates
    - lambda:*
    - cloudwatch:ListMetrics
    - cloudwatch:GetMetricData
    - route53:GetHostedZone
    - route53:ListHostedZones
    - route53:ListHostedZonesByName
    - route53:ChangeResourceRecordSets
    - route53:ListResourceRecordSets
    - route53:GetHealthCheck
    - route53:GetHealthCheckStatus
    - cloudwatch:GetMetricStatistics
    Resource: "*"
    Roles:
    - !Ref HarnessCloudFormationRole

Step: Connect CCM to AWS Cloud Provider

To enable CCM for your AWS services (such as EC2, S3, RDS, Lambda, and so on), you simply need to connect Harness to your AWS accounts.

Perform the following steps to connect CCM to the AWS cloud provider.

Step 1: Overview

  1. In Account Setup, in Account Resources, click Connectors.
  2. In Connectors, click + Connector.
  3. In Cloud Costs, click AWS.
  4. In AWS Connector, in Overview, enter the Connector Name. The name will appear in CCM Perspectives to identify this cloud provider.
  5. In Specify the AWS account ID, enter your AWS account ID and click Continue. To find your AWS account ID, see Finding your AWS account ID.

Step 2: Cost and Usage Report

Cost and Usage Report (CUR) provides detailed billing data across AWS accounts to help you analyze your spending. You need to enter the cost and usage report name and cost and usage S3 bucket name in Harness. To get these details, do the following:

  1. In Cost and Usage Report, click Launch AWS console to log into your AWS account.
  2. In AWS Cost and Usage Reports, click Create Report.
  3. Enter the Report Name. This is the CUR name that you need to enter in Harness.
  4. In Additional report details, select the checkbox Include resource IDs to include the IDs of each individual resource in the report.
  5. In Data refresh settings, select the checkbox Automatically refresh your Cost & Usage Report when charges are detected for previous months with closed bills.
  6. Click Next.

    When you are done with the Report content step, it will look something like this:
  7. In the S3 bucket, click Configure.
  8. In Configure S3 Bucket, in Create a bucket, enter the S3 bucket name. This is the cost and usage S3 bucket name that you need to enter in Harness. For more information on S3 bucket naming requirements, see Amazon S3 Bucket Naming Requirements.
  9. Select Region from the drop-down list and click Next. It is recommended to select US East (N. Virginia).
  10. In Verify policy, select the checkbox I have confirmed that this policy is correct and click Save.
  11. Enter the report path prefix that you want to be prepended to the name of your report.
  12. Select Hourly in Time granularity.
  13. Select Overwrite Existing Report in Report versioning.
  14. Do not select any value in Enable report data integration for.
  15. Select GZIP in the Compression type.
  16. Click Next.

    When you are done with the Delivery options step, it will look something like this:
  17. Review your report details and click Review and Complete.
    Your report is listed in AWS Cost and Usage Reports.
  18. Enter the Cost and Usage Report Name (as entered in step 3) and Cost and Usage S3 Bucket Name (as entered in step 8) in Harness.

Step 3: Select Features

Select the Cloud Cost Management features that you would like to use on your AWS account. Based on your selection Harness requires specific permissions for the cross-account role. See Review: AWS Access Permissions.

CCM offers the following features:

Cost Visibility (Required)

This feature is available by default and requires access to the CUR report. Provides the following capabilities:

  • Insights into AWS costs by services, accounts, etc.
  • Root cost analysis using cost perspectives
  • Cost anomaly detection
  • Governance using budgets and forecasts
  • Alert users using Email and Slack notification

AWS ECS and Resource Inventory Management (Optional)

This feature provides visibility into your EC2, EBS volumes, and ECS costs. The insights provided by inventory management can be consumed by Finance teams to understand the resource utilization across the board.

AWS resource optimization using AutoStopping rules (Optional)

This feature allows you to enable Intelligent Cloud AutoStopping for your AWS instances and auto-scaling groups. For more information, see Create AutoStopping Rules for AWS.

  • Orchestrate VMs and ASGs based on idleness
  • Run your workloads on fully orchestrated spot instances
  • Granular savings visibility

Make your selection and click Continue.

Step 4: Create Cross-Account Role

Harness uses the secure cross-account role to access your AWS account. The role includes a restricted policy to access the cost and usage reports and resources for the sole purpose of cost analysis and cost optimization.

  1. In Create Cross Account Role, click Launch Template in AWS console.
  2. In Quick create stack, in Capabilities, select the acknowledgment, and click Create stack.
    It is recommended that you do not modify any value in the Quick create stack page.


    The value for BillingEnabled, EventsEnabled, and OptimizationEnabled varies depending on the features that you have selected in the Select Features step.
  3. In the Stacks page, from the Outputs tab copy the Value of CrossAccountRoleArn Key.
  4. In Role ARN, enter the Cross-Account Role ARN that you copied from the Outputs tab (previous step) in Harness.
  5. The External ID is generated dynamically for your account. For example, harness:111111111111:lnFZRF6jQO6tQnB9xxXXXx .
    Do not modify the value of External ID.
  6. Click Save and Continue.

Step 5: Test Connection

The validation and verification happen in this step. Once the validation and verification are completed, click Finish.

Your connector is now listed in the Connectors.

Next Steps


Please Provide Feedback