CPU limits und requests, how to avoid CPU throttling

If you look at the usual utilization of a kubernetes cluster, you can quickly see that a lot of resources are idling around. Nevertheless, you may receive a warning that the cluster’s resources are at their limit. How does this fit together?

The issue is non-optimized resource allocation. Resource limits help with budget control by preventing a single application from consuming more resources than allocated, which is critical in cloud environments where resources translate directly into costs.

Pitfalls of wrong CPU limits/requests

By limiting resource usage, organizations can predict and control the compute costs associated with their Kubernetes clusters. It is essential for companies that these limits are set so that they do not go bankrupt with the next cloud provider bill (which they can enforce via policies, but that’s another topic).

If the requests and limits are set inaccurately, it can happen that the cluster enters a state in which nothing can be deployed while the workload is in the idling state.

Another pitfall can be CPU throttling, which occurs when a container tries to use more CPU resources than are allocated to it. A CPU request limit is similar to a time slice on a CPU. What you set is a time slice for how long a workload can process things. on that device. CPU throttling happens when the time slice is not enough. This process involves temporarily reducing the container’s CPU usage, effectively slowing down the application’s execution. Applications might experience increased response times or reduced throughput, which can impact end-user experience and overall system efficiency.

Are Kubernetes CPU Limits an Antipattern?

Setting CPU limits on Kubernetes might sometimes be considered an antipattern, primarily because it can lead to undesirable CPU throttling. If limits are set too low, they can constrain application performance unnecessarily, affecting service responsiveness and potentially leading to timeouts and error escalations.

The challenge is in accurately predicting the appropriate CPU limits that match an application’s demands without under or overprovisioning.

Recommendations

In my opinion this task is quite daunting and hard. And can only be solved with good observability tooling.

My recommendation is to start with no CPU limits for workloads, just set CPU requests. And set the requests to a value that corresponds to the usual state when running for several hours. If you know the maximum CPU requests from your observability tooling and have clear information, set CPU limits.

If your company pays for the CPUs consumed (especially because you are not onPrem), and you want to have some safety, use the LimitRanges and maxLimitRequestRatio values. For more information, here’s a link.

If your workload is very CPU sensitive, it may be good to have a closer look into the CPU Manager feature which constraints workloads to specific CPUs.

Cheats for Admins

Cluster administrators also have another good tool to ensure that their cluster costs do not lead to a state where the company could go bankrupt: The Cluster Resource Override Operator. The Cluster Resource Override Operator exactly does what it means. https://docs.openshift.com/container-platform/4.17/nodes/clusters/nodes-cluster-overcommit.html#nodes-cluster-resource-override_nodes-cluster-overcommit

apiVersion: operator.autoscaling.openshift.io/v1
kind: ClusterResourceOverride
metadata:
    name: cluster 
spec:
  podResourceOverride:
    spec:
      memoryRequestToLimitPercent: 50 
      cpuRequestToLimitPercent: 25 
      limitCPUToMemoryPercent: 200

This is a nice and effective way. But it is an opt-in for namespaces. To use it, you have to define a label for all namespaces.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    clusterresourceoverrides.admission.autoscaling.openshift.io/enabled: "true"

You can ensure that via ProjectTemplates and polices, but that’s another topic.

API Certificate fix for OpenShift

I run my own small OpenShift cluster, which is hosted on hetzner (installation: https://github.com/RedHat-EMEA-SSA-Team/hetzner-ocp4). Recently my internal API certificate expired. Here is the error message:

openshift message: ‘Existing private key is not up to date for spec: [spec.privateKey.size]

I thought this could be solved with the Ansible renew certificate playbook, but no. I found the solution here: https://github.com/cert-manager/cert-manager/issues/5183

You have to set spec.privateKey.rotationPolicy on the Certificate resource to Always (oc edit certificate api-server-certificate -n openshift-config), then the internal rotation of the certificates works. I may have broken something myself by testing to get to this state, but that fixed it anyway.

Sonos video about disastrous app release

I ran into an interesting video about Sonos and what may happen if you ignore warnings from your software engineers: https://www.youtube.com/watch?v=yvGY5vkONds

Tool recommendations

Here is a list of useful tools that you should take a look at: