Harness Cloud Cost Management (CCM) monitors and provides visibility into the cloud costs of your Amazon Web Services (AWS) across your cloud infrastructure and AWS services, such as EC2, S3, RDS, Lambda, and so on. CCM also allows you to optimize your instances, auto-scaling groups (ASGs), and EKS clusters using intelligent cloud AutoStopping rules.
You can set up CCM for your AWS resources, in a simple two-step process:
Create a Cost and Usage Report (CUR). Harness CCM uses a secure, cross-account role with a restricted policy to access the cost and usage reports and resources for cost analysis.
Create a Cloudformation stack to provision IAM Roles and corresponding policies to grant access for the required features. CCM offers the following features:
Cost Visibility (Required)
This feature is available by default and requires access to the CUR report. Provides the following capabilities:
Insights into AWS costs by services, accounts, etc.
Root cost analysis using cost perspectives
Cost anomaly detection
Governance using budgets and forecasts
Alert users using Email and Slack notification
This feature will give you cost insights that are derived from the CUR. For deep Kubernetes visibility and rightsizing recommendations based on the historical utilization and usage metrics, set up Kubernetes connectors. See Set Up Cloud Cost Management for Kubernetes.
AWS ECS and Resource Inventory Management (Optional)
This feature provides visibility into your EC2, EBS volumes, and ECS costs. The insights provided by inventory management can be consumed by Finance teams to understand the resource utilization across the board.
Breakdown by ECS cluster cost, Service, Task, Launch Type (EC2, Fargate)
AWS resource optimization using AutoStopping rules (Optional)
This feature allows you to enable Intelligent Cloud AutoStopping for your AWS instances and auto-scaling groups. For more information, see Create AutoStopping Rules for AWS.
Orchestrate VMs and ASGs based on idleness
Run your workloads on fully orchestrated spot instances
Provides Granular savings visibility
Cloudformation template has policies corresponding to all the permissions (Visibility, Inventory, and Optimization). However, it is important to note that the permissions (policies) of the selected features will only be applied.
After enabling CCM, it takes about 24 hours for the data to be available for viewing and analysis.
The same connector cannot be used in NextGen and FirstGen. For information on creating an AWS connector in the FirstGen see Set Up Cost Visibility for AWS.
For CCM, AWS connectors are available only at the Account level in Harness.
Review the AWS connector requirements for different CCM features:
Cost Visibility: You can create an AWS connector for the master or linked account. CCM requires one connector per AWS account (master or linked) for cost visibility.
AWS ECS and Resource Inventory Management: You need to create an AWS connector for each linked account. For inventory management, CCM requires a connector for all the linked accounts.
AWS Resource Optimization Using AutoStopping Rules: You need to create an AWS connector for each linked account. For resource optimization using AutoStopping Rules, CCM requires a connector for all the linked accounts.
Review: Cost and Usage Reports (CUR) and CCM Requirements
If you have a consolidated billing process enabled, then CCM needs read-only access to the cost and usage reports (CUR) stored in the S3 bucket in the master or payer account. This gives access to the cost data for all the accounts (linked/member) in the organization.
If you don't have consolidated billing enabled at the organization level then you can create the CUR at a linked account level.
If you have provided CUR access to the master account then you do not need to provide billing details for each linked account. CCM requires one connector per AWS account (master or linked).
It is recommended to create a CUR at the master account to avoid the CUR creation step for each linked account.
If you do not have access to the master account, you can create an AWS connector in the linked account for which you have the required access.
If you have created a billing report for your AWS account ID once then you can use the same CUR again for the AWS connector. You do not need to create CUR again for the same account.
Review: AWS Access Permissions
CCM requires the following permissions:
If you don't have access to create a cost and usage report or run a CloudFormation template, contact your IT or security teams to provide the required permissions.
Cost Visibility
The cost visibility policy performs the following actions:
List CUR reports and visibility into the organization's Structure
Get objects from the S3 bucket configured in the CUR
To enable CCM for your AWS services (such as EC2, S3, RDS, Lambda, and so on), you simply need to connect Harness to your AWS accounts.
Perform the following steps to connect CCM to the AWS cloud provider.
Step 1: Overview
In Account Setup, in Account Resources, click Connectors.
In Connectors, click + Connector.
In Cloud Costs, click AWS.
In AWS Connector, in Overview, enter the ConnectorName. The name will appear in CCM Perspectives to identify this cloud provider.
In Specify the AWS account ID, enter your AWS account ID and click Continue. To find your AWS account ID, see Finding your AWS account ID.
Step 2: Cost and Usage Report
Cost and Usage Report (CUR) provides detailed billing data across AWS accounts to help you analyze your spending. You need to enter the cost and usage report name and cost and usage S3 bucket name in Harness. To get these details, do the following:
In Cost and Usage Report, click Launch AWS console to log into your AWS account.
In AWS Cost and Usage Reports, click Create Report.
Enter the Report Name. This is the CUR name that you need to enter in Harness.
In Additional report details, select the checkbox Include resource IDs to include the IDs of each individual resource in the report.
In Data refresh settings, select the checkbox Automatically refresh your Cost & Usage Report when charges are detected for previous months with closed bills.
Click Next.
When you are done with the Report content step, it will look something like this:
In the S3 bucket, click Configure.
In Configure S3 Bucket, in Create a bucket, enter the S3 bucket name. This is the cost and usage S3 bucket name that you need to enter in Harness. For more information on S3 bucket naming requirements, see Amazon S3 Bucket Naming Requirements.
Select Region from the drop-down list and click Next. It is recommended to select US East (N. Virginia).
In Verify policy, select the checkbox I have confirmed that this policy is correct and click Save.
Enter the report path prefix that you want to be prepended to the name of your report.
Select Hourly in Time granularity.
Select Overwrite Existing Report in Report versioning.
Do not select any value in Enable report data integration for.
Select GZIP in the Compression type.
Click Next.
When you are done with the Delivery options step, it will look something like this:
Review your report details and click Review and Complete.Your report is listed in AWS Cost and Usage Reports.
Enter the Cost and Usage Report Name (as entered in step 3) and Cost and Usage S3 Bucket Name (as entered in step 8) in Harness.
Step 3: Select Features
Select the Cloud Cost Management features that you would like to use on your AWS account. Based on your selection Harness requires specific permissions for the cross-account role. See Review: AWS Access Permissions.
CCM offers the following features:
Cost Visibility (Required)
This feature is available by default and requires access to the CUR report. Provides the following capabilities:
Insights into AWS costs by services, accounts, etc.
Root cost analysis using cost perspectives
Cost anomaly detection
Governance using budgets and forecasts
Alert users using Email and Slack notification
This feature will give you cost insights that are derived from the CUR. For deep Kubernetes visibility and rightsizing recommendations based on the historical utilization and usage metrics, set up Kubernetes connectors. See Set Up Cloud Cost Management for Kubernetes.
AWS ECS and Resource Inventory Management (Optional)
This feature provides visibility into your EC2, EBS volumes, and ECS costs. The insights provided by inventory management can be consumed by Finance teams to understand the resource utilization across the board.
Breakdown by ECS cluster cost, Service, Task, Launch Type (EC2, Fargate)
AWS resource optimization using AutoStopping rules (Optional)
This feature allows you to enable Intelligent Cloud AutoStopping for your AWS instances and auto-scaling groups. For more information, see Create AutoStopping Rules for AWS.
Orchestrate VMs and ASGs based on idleness
Run your workloads on fully orchestrated spot instances
Granular savings visibility
Make your selection and click Continue.
Step 4: Create Cross-Account Role
Harness uses the secure cross-account role to access your AWS account. The role includes a restricted policy to access the cost and usage reports and resources for the sole purpose of cost analysis and cost optimization.
In Create Cross Account Role, click Launch Template in AWS console.
In Quick create stack, in Capabilities, select the acknowledgment, and click Create stack.
It is recommended that you do not modify any value in the Quick create stack page.
The value for BillingEnabled, EventsEnabled, and OptimizationEnabled varies depending on the features that you have selected in the Select Features step.
In the Stacks page, from the Outputs tab copy the Value of CrossAccountRoleArn Key.
In Role ARN, enter the Cross-Account Role ARN that you copied from the Outputs tab (previous step) in Harness.
The External ID is generated dynamically for your account. For example, harness:111111111111:lnFZRF6jQO6tQnB9xxXXXx .
Do not modify the value of External ID.
Click Save and Continue.
Step 5: Test Connection
The validation and verification happen in this step. Once the validation and verification are completed, click Finish.
Your connector is now listed in the Connectors.
Step: Create Multiple Connectors in an AWS Account
Harness CCM also provides the flexibility to create multiple Connectors using a stack set configured at the master account level. It involves the following steps:
Create a stack set
Create an API Key in Harness
Add an Admin Role
Run the cURL Command to create connectors using the Roles created in the AWS accounts via API
Step 1: Create a Stack Set in AWS
Perform the following steps to create a stack set in AWS:
In Choose a template, in Permissions, select Service-managed permissions.
In Prerequisite - Prepare template, select Template is ready.
In the Specify template, in the Template source, select Amazon S3 URL.
In the Amazon S3 URL enter the following URL and click Next. https://continuous-efficiency-prod.s3.us-east-2.amazonaws.com/setup/ngv1/HarnessAWSTemplate.yaml
In Specify StackSet details, in StackSetname, enter a stack set name. For example, harness-ce-iam-stackset.
In Parameters, specify the following details:
Set BillingEnabled to false.
Leave the BucketName empty.
Set EventsEnabled to true.
In ExternalID enter your <Harness Account ID>, for example, harness:111122225555 .
In LambdaExecutionRoleName, enter Lambda execution role name, for example, HarnessCELambdaExecutionRole. The Lambda execution role name must begin with Harness.
Set OptimizationEnabled to true.
PrincipalBilling is auto-generated for your AWS account. Do not edit the Principal Billing details. For example, arn:aws:iam::123451231355:root.
In RoleName, enter the role name, for example, HarnessCERole. The role name must begin with Harness e.g., HarnessCERole, HarnessManagedRole.
Once you've entered all the details, click Next.
In Configure StackSet options, in Managed execution, select Active and click Next.
In Set deployment options, in Add stacks to stack set, select Deploy new stacks.
In Deployment targets, select Deploy to organization (recommended). You can select Deploy to Organizational Units (OUs) to limit the monitoring clusters to a particular OU or a subset of linked accounts.
In Automatic deployment, select Enabled.
In Account removal behavior, select Delete Stacks.
Select a region from the drop-down list and click Next.
In Deployment Options, in Region Concurrency, select Sequential.
Review the details, select acknowledgment, and click Submit.
Step 2: Create an API Key in Harness
In Harness, click Home.
In Account Setup, click Access Control.
Click Service Accounts and then click the service account to which you want to add a new API Key. For step-by-step instructions to add a new Service Account, see Add and Manage Service Accounts.
In the Service Account's settings page, click API Key.
In the New API Key settings, enter Name, Description, and Tags.
Click Save. The new API Key is created.
Once you've created an API Key for your Service Account, generate a Token for this API Key.
To generate a Token for this API Key, click Token below the API Key you just created.
In the New Token settings, enter Name, Description, and Tags.
To set an expiration date for this token, select Set Expiration Date.
Enter date in Expiration Date (mm/dd/yyyy).
Click Generate Token.
Your new Token is generated.
You cannot see this token value after you close this dialog. Make sure to copy and store the generated token value securely.
You need to enter this Token when running your cURL command.
Step 3: Add an Admin Role to the Service Account
Ensure that you've added the Admin role to this Service Account. For more information, see Add and Manage Roles.
Step 4: Run the cURL Command
Run the following command for each AWS Account ID and IAM Role Pair:
curl -i -X POST \ 'https://app.harness.io/gateway/ng/api/connectors?accountIdentifier=<CustomerHarnessAccountID>' \ -H 'Content-Type: application/json' \ -H 'x-api-key: <Enter your API Key Token>' \ -d '{ "connector":{ "name":"AWSConnector-<AWSAccountId>", "identifier":"AWSConnector_<AWSAccountId>", "spec":{ "crossAccountAccess":{ "crossAccountRoleArn":"<Enter the Role Created in the Account>", "externalId":"<Enter ExternalID oused when creating the IAM Role>" }, "awsAccountId":"<AWSAccountId>", "curAttributes":{ "reportName":"", "s3BucketName":"" }, "featuresEnabled":[ "VISIBILITY", "OPTIMIZATION" ] }, "type":"CEAws" } }'
