Optimize Kubernetes Costs with Node Pool Recommendations

Updated 1 month ago by Archana Singh

One of the most impactful ways to reduce spend on Kubernetes infrastructure is to make sure your clusters are optimally sized for the workloads and node pools they run. Harness CCM recommends optimal compute resources for your workloads and node pools. It can help you reduce costs and improve performance, by analyzing your historical utilization metrics. CCM recommends you choose the optimal resource configuration based on your utilization data and requests metrics of Pods.

This topic describes how CCM computes node pool recommendations and how you can use them to potentially reduce monthly costs.

Before using recommendations in your cluster environment, ensure that you evaluate their impact thoroughly. The person reviewing the recommendations should be able to understand the impacts identified in the recommendations, as well as the impact on the infrastructure and business.


Using recommendations without proper assessment could result in unexpected changes, such as issues with system performance or poor reliability.

Before You Begin

Review: How are Node Pool Recommendations Computed?

The node pool recommendations are computed by analyzing historical utilization data and requests metrics of Pods. CCM recommends the optimal resource configurations for the Spot and On-demand instances. It uses the following parameters to determine the maximum node counts:

  • Total CPUs
  • Total memory
  • Max CPUs
  • Max Memory

The recommendations are calculated by aggregating the resource utilization and request across all pods running across nodes in the node pool. You can select the number of days to compute recommendations based on the utilization data. The available options are last 1 day, 7 days, and 30 days.

CPU, Memory: Max at a given instant [Aggregated Max(resource Utilization, requests)]

Let's try to understand how the recommendations are computed using the following example. Assume there are two nodes in a node pool, each with two pods running.

Time Instant 1:
Node1:
Pod1:
Util → 2.1 vCPU, Request → 4 vCPU
Pod2:
Util → 1.4 vCPU, Request → 1 vCPU

Node2:
Pod1:
Util → 1.1 vCPU, Request → 4 vCPU
Pod2:
Util → 0.9 vCPU, Request → 1 vCPU

Time Instant 2:
Node1:
Pod1:
Util → 2.2 vCPU, Request → 4 vCPU
Pod2:
Util → 1.5vCPU, Request → 1 vCPU
Node2:
Pod1:
Util → 2.0 vCPU, Request → 4 vCPU
Pod2:
Util → 0.3 vCPU, Request → 1 vCPU

The recommendations are calculated as the following:

Time Instant 1: [ Max (2.1, 4) + Max (1.4, 1)+ Max (1.1, 4)+ Max (0.9,1)] = [4 + 1.4 + 4 + 1] => 10.4

Time Instant 2: [ Max (2.2, 4) + Max (1.5, 1)+ Max (2, 4)+ Max (0.3,1)] = [4 + 1.5 + 4 + 1] => 10.5

So the Max CPU recommendation would be: Max (10.4, 10.5) = 10.5 vCPU

Similarly, for memory, nodes will be the max number of nodes in the observation interval.

Node pool recommendations also offer the flexibility of tuning the recommendations by setting the resource configuration preferences. See Tune Recommendations.

Step: View Recommendations

Once you enable CCM, it may take up to 48 hours for the recommendations to appear in Cloud Costs. It depends on the time at which CCM receives the utilization data for the workload.
  1. In Cloud Costs, click Recommendations.

    The recommendations page displays the following information:
    • A breakdown of all the available recommendations.
    • Potential Monthly Savings across your Kubernetes clusters if you apply the recommendations.
    • Forecasted Monthly Spend across your Kubernetes clusters if you do not apply the recommendations.
  2. The Recommendation Breakdown displays the following information:

    Monthly Savings

    Potential monthly savings for your resource, if you apply the recommendations.

    Resource Name

    Name of the resource for which CCM displays the recommendation.

    Resource Type

    Type of the resource for which the CCM has provided the recommendations. For example, node pool or workload.

    Monthly Cost

    Potential Monthly Savings for the resource, if you apply the recommendations.

    Recommendation Type

    Type of the recommendation for your resource. For example, rightsizing or resizing. Based on your resource type, CCM recommends to rightsize or resize your CPU, memory, or node counts.

  3. Click the recommendation for which you want to view the details. You can use a filter to select the resource or recommendation for which you want to view the details.
    You can filter by:
    • Name: Each Kubernetes namespace in the cluster.
    • Resource Type: The type of resources for which the recommendation is displayed. Currently, CCM supports node pool and workload.
    • Namespace: Each Kubernetes namespace in the cluster.
    • Cluster Name: Each Kubernetes cluster in your infrastructure.
    • Monthly savings greater than: Filter by potential monthly savings greater than the specified amount. For example, all the recommendations that provide potential monthly savings of more than $1000.
    • Monthly cost greater than: Filter by forecasted monthly spend greater than the specified amount. For example, all the recommendations that provide forecasted monthly spend more than $1000.
  4. A recommendation for the selected resource is displayed.
  5. Select the number of days to compute recommendations based on the utilization data. You can select the last day, 7 or 30 days.
  6. You can use this information to optimize your resources to potentially reduce your monthly cloud costs.

Step: Tune Recommendations

CCM provides you the option to set your preferences to tune your recommendations. You can modify the value of CPU, memory, node counts, and instances. You can also add a buffer to the resource preferences. In such a scenario, CCM generates context-aware cluster sizing recommendations based on the resource configurations that you provide.

To tune your recommendations, set your preferences for the following resources:

  • CPU
  • Memory
  • Node count
  • Instances

Option: Preferred Resource Needs

The value for CPU and memory is auto-populated by aggregating the resource utilization and request across all pods running across nodes in the node pool. See How are Node Pool Recommendations Computed?.

CPU (vCPU): Set preferences for the CPU.

RAM (GiB): Set preferences for memory.

Buffer: This section allows you to add an additional buffer to the resources for calculating the recommendations.

By default, the recommendation computation adds a 15% buffer to the recommended resources. This option allows you to add an additional buffer to your resources.

Once you apply the preferences, modified values are used for calculating the recommendations.

Option: Largest Workload Requirements

The maximum workload that is observed based on the historical utilization that is scheduled on the nodes is used to calculate the value for this field. You can customize your CPU (vCPU) and RAM options (GiB).

Option: Preferred Minimum Node Count

This option allows you to set the number of minimum nodes that are used for the recommendation. Use the + button to increase the node count and click - to decrease the node count.

Option: Preferred Instance families

You can use this option to limit the recommendations to a certain machine type/instance family. For example, you have RIs/commitments on a specific machine type and only want to use that instance family.

  1. In Preferred Instance families, click add preferred instance families.
  2. Select your preferences and click Save Preferences. The category of the instance families will depend on the cloud provider type. The following example shows the GKE.
  3. Once, you've made all the changes, click Apply Preferences or click Reset to default.

Next Steps


Please Provide Feedback