After a bit of a delay from the original targeted release date, Kubernetes’ latest version, v1.29, has been officially released! The preceding Kubernetes version, 1.28 was launched in August 2023, with a few security-related changes made to RBAC user attributes, CRD validation rules, validation admission policy, and more. The number of security-related changes in the current Kubernetes version 1.29 release is substantially larger, and range from features meant to reduce disruptive downtime, improve cloud interoperability, achieve better observability and fix potential security flaws. Generally bug fixes in Kubernetes are reserved for CVE publications; these Kubernetes features give the Kubernetes community additional functionality, ease of use and better security in the Kubernetes project.
- Latest Kubernetes version features that improve uptime and reliability for SREs and engineers
- Kubernetes Component Health Service Level Indicators (SLI)
- Kubelet Resource Metrics Endpoint
- Cloud Dual-Stack --node-ip Handling
- Make Kubernetes aware of the LoadBalancer behaviour
- Allow replacement of Pods in a Job when fully terminated - for machine learning frameworks
- Split Image Filesystem into readable and writeable in alpha
- Multiple Service CIDRs in alpha
- Latest Kubernetes version features that help Security teams
- Support User Namespaces in Alpha
- Structured Authorization Configuration
- ReadWriteOncePod PersistentVolume AccessMode
- Sidecar container changes in Beta
- Common Expression Language (CEL) for Admission Control & CRD Validation Expression Language
- Interactive(-i) flag to kubectl delete for user confirmation
- Reduction of secret-based service account tokens
- Bound service account token improvements in alpha
- KMS v2 Improvements
- Cluster Trust Bundles
- How to check your current Kubernetes version
- Conclusion
Need better visibility into your K8s security risk? Get the checklist
Latest Kubernetes Version Features That Improve Uptime and Reliability for SREs and Engineers
Kubernetes security is a shared responsibility between security and engineering teams. SREs and engineers that care more about Kubernetes features impacting uptime, reliability and building consistent deployments view Kubernetes security through the lens of consistency across hybrid environments, accidental oversubscription of rate limits, and observability. Many of the Kubernetes 1.29 release features were specifically meant to ensure reliable and predictable performance.
Kubernetes Component Health Service Level Indicators (SLI)
Observability is critical to maintaining uptime, and this enhancement aims to create quicker intervals for consuming Kubernetes SLIs, which are metrics for the status of Kubernetes components (for example, the health of etcd). In Kubernetes 1.28, by default, Kubernetes publishes SLIs for each component. But the data is surfaced in an unstructured format that requires monitoring agents to both interpret and act. With a more structured interface to SLIs, the idea is that observability will become easier, in service of teams’ larger mission to maintain a set of service level objectives for availability.
Kubelet Resource Metrics Endpoint
Observability in Kubernetes has long been a challenge. The Kubelet resource metrics endpoint, as one might suspect, serves to improve the resource and monitoring metrics of the Kubelet. The proposed change involves creating a purpose-built resource metric API so that the Kubelet Summary API doesn’t have to do both resource and monitoring metrics at the same time - which it does today, causing multiple problems around granularity and also overloading of the metrics server that is used by Kubelets to store the API data.
Cloud Dual-Stack --node-ip Handling
This enhancement is all about making dual-stack a possibility when Kubernetes is deployed with a cloud provider. Dual-stack support in Kubernetes means simultaneous support for IPv4 and IPv6 in networking. Today, the kubelet supports dual-stack behavior in a non cloud environment, but not when it comes to cloud providers.
Make Kubernetes aware of the LoadBalancer behavior
When interfacing with a cloud provider, today, the kube-proxy is either binding the External IPs of the LoadBalancer Service to each node or redirecting packets to go around the LoadBalancer, which doesn’t honor the intended behavior of the load balancer. This could result in issues like inaccurate health checks, or bypassing cloud features that act at the level of the load balancer.
The only current solution is to employ a workaround that enables the cloud provider to disable the kube-proxy’s actions. With this enhancement, the need for hacks and workarounds would be removed, lowering overall risk of performance implications. In this enhancement, the cloud provider would be able to more accurately interface with the kube-proxy so the LoadBalancer wouldn’t have to be bypassed or bound to each node via External IP.
Allow replacement of Pods in a Job when fully terminated - for machine learning frameworks
In machine learning frameworks, the current process of pod termination and re-deployment is a problem, because when a pod begins the termination process, but isn’t yet fully terminated, the pod being deployed in its place doesn’t start right away. It has to wait for the termination process, which isn’t immediate. But if the pod is starting before the other pod has fully terminated, it might cause an undesired uptick in resource utilization. This enhancement calls for better visibility into the terminating state so users can better control resource utilization and control how they want the system to work.
In this enhancement, the idea is that users would get both visibility into, and the option to specify, whether they want a replacement pod to be created when the previous pod is fully terminated, or in the process of terminating. This is specifically created for machine learning frameworks that require unique pods for processing.
Split Image Filesystem into readable and writeable in alpha
In Kubernetes, there is a layer that doesn’t change very much (the image layer), which is referred to as the readable layer, and the container and node layers, which can be referred to as the writable layer, because it contains the temporary files required for the kubelet to run.
In this enhancement, which is in alpha in version 1.29, the goal is to enhance the performance of the kubelet by separating readable and writable file systems, while ensuring the writable files required for running the kubelet are available on the kubelet itself.
This enhancement could prevent an unwanted decline in performance of the Kubelet and make garbage collecting more transparent, because today when the image disk gets too full the garbage collector doesn’t have any context on which files it is taking from, as they are both stored in the same place.
Multiple Service CIDRs in alpha
In networking, uptime and availability can be limited by the fact that Kubernetes services like LoadBalancer, ClusterIP and NodePort are limited to the number of IPs that are available. This causes several availability and security issues:
- The logic of allocating Services IPs could possibly leak ClusterIPs
- The Service IP range is not exposed, so no other components can use it
- Users can increase the availability of IPs assigned to Services, and clusters can run out of available IPs
This enhancement proposes more scalable, flexible and dynamic logic for services to use and receive IP addresses.
Latest Kubernetes Version Features That Help Security Teams
In a shared responsibility model for Kubernetes security, teams in IT Security will be more focused on Kubernetes features such as policy enforcement, access, secrets, and the ability to deploy security-related plugins and tools into their clusters.
Support User Namespaces in Alpha
Today, multiple over permissioning issues arise in Kubernetes because there are no different user/group definitions or distinctions from inside the container versus on the node. For example, to have privileged capabilities with cluster admin, you are also granted extra privileges on the host or other pods. Or there isn’t a way to take an action on the host just within a certain namespace. Or to run a container as root you also get privileges on the host.
In this enhancement, the goal is to use user and group IDs that can map from the container to the host, improving isolation. For example, mapping root privileges in the container to unprivileged user and group IDs in the node so you don’t automatically get root privileges in the container and privileged access on the host. A side benefit of this is then that pods could have capabilities that would only be valid in the pod, instead of always having some 1:1 mapping of capabilities in the pod across to the container.
There have been multiple vulnerabilities caused by the broad permission granted across containers and pods, that could be remedied by enabling or disabling user namespaces in the pod.spec, as well as supporting pods inside user namespaces.
Structured Authorization Configuration
This enhancement adds more capabilities for authorization webhooks to the API server, for example:
- Cluster administrators can add more than one webhook to the authorization chain
- You could configure the authorization requirements within a config file
- It is possible to create clear guidelines and power for denying a request (or skipping a part in the chain)
The proposal includes defining a config file format that would contain the relevant information about the overall authorization chain, while allowing for implementations that differ based on users and resources.
ReadWriteOncePod PersistentVolume AccessMode
PersistentVolumes are a cluster resource and an important part of Kubernetes storage and the CSI driver. There have been many storage-related Kubernetes vulnerabilities in 2023, including the CSI driver Kubernetes vulnerability and a Kubernetes vulnerability in the in-tree plugins for Windows nodes.
To-date, to access persistent volumes, the options are limited for use cases of multiple users needing access on the same node. In this enhancement, which is a graduating enhancement first introduced in 2021, in version 1.22, a ReadWriteOncePod access mode for PersistentVolumes solves the current problem (which has caused many vulnerabilities) around restricting access to a single node, but then allowing multiple pods to access that node. This ReadWriteOnce mode allows access to the node from only a single pod. This requires changes across the storage ecosystem including the CSI, driver, and sidecars.
Sidecar container changes in Beta
Many service mesh, security and monitoring solutions rely on sidecars, but today sidecars rely on a clunky implementation with limitations:
- Sidecars don’t start until the container starts, so their logging, security or service mesh capabilities are incomplete
- The regular pod isn’t allowed to terminate until the sidecar terminates - so if the sidecar is running continuously, the pod termination won’t occur and interferes with the Kubernetes deployment lifecycle
In this enhancement, one could easily inject sidecars into Pods, allow sidecar containers that run longer than regular containers, and make the sidecar container startup sequence more in-line with the regular pods.
Common Expression Language (CEL) for Admission Control & CRD Validation Expression Language
In version 1.25, it was proposed that CEL be used as a way to support validation rules for CRDs. Validation rules for CRDs mean that CRD creation can be approved or denied based on validation rules instead of complex webhooks. In version 1.28, there were more improvements made to the capability, and in version 1.29 the goal is generally to provide an alternative to webhooks with in-tree extensions, and to make this core functionality available in an easily consumable way in GitOps and the CI/CD process.
For the CRD piece, the goal is to allow for validating policies to be handled in-tree for the CRD, versus using webhooks to apply validation rules for a CRD. Our blog for Kubernetes version 1.28 goes into detail around the benefits and drawbacks of webhooks versus admission policy.
Interactive(-i) flag to kubectl delete for user confirmation
In Kubernetes, the command ‘kubectl delete’ can delete a Kubernetes Deployment. The results of the Kubectl delete command are irreversible, so the idea is to mitigate accidental deletes by showing a preview to the user of all the resources planning to be deleted beforehand. This enhancement intends to create a flag, interactive(-i), that asks for confirmation from a user before proceeding to enforce the command kubectl delete.
Reduction of secret-based service account tokens
The current state of generating secret-based service account tokens is less secure than than the bound token option introduced in version 1.22. This enhancement aims to reduce the exposure created by the generation of secret-based service account tokens. For example, it aims to remove unused, auto-generated secret-based service account tokens, and the auto-creation of secrets for service accounts.
Bound service account token improvements in alpha
Service accounts are the machine or API users in RBAC - aka the most likely ones to be overlooked or forgotten. This enhancement allows for better tracking and quicker transparency into the token usage of service accounts. For example, this enhancement allows somebody to understand who made an original access request from a service account through the token; by binding info about the node for which the token is being used, to the JWT token itself.
This means it’s easier to check if the host identity is indeed the one tied to the token, and determine where the original request for a serviceaccount token came from, without having to perform complicated cross-referencing.
KMS v2 Improvements
In Kubernetes, a KMS (Key Management Service) allows you to encrypt Kubernetes API data using a key stored in an external key service. Today, the process for rotating a KMS (Key Management Service) key is error-prone; for example, the health check for the connection from kube-apiserver to the KMS plugin has to be done by decrypting the data, and there is no easy way to check for key rotation. And when you request to the API server for encrypted secrets you can reach rate limits which can take too much time.
In this proposal, the goals are to improve this time lag for clusters with a large number of encrypted items, enable key rotation that is either fully or partially automated without having to restart the API server, and improve the KMS plugin health check process.
Cluster Trust Bundles
Generally, a trust anchor refers to the top most certificate in a certificate chain - for example, a certificate in a root store (more common) or a self-signed cert by a developer (less common). In Kubernetes, today there is no standardized mechanism for trust anchors to be distributed to workloads, even though there is a flexible, pluggable way for workloads to request X.509 certificates (learn more about X.509 certificates in Kubernetes with this Kubernetes Security checklist).
The goal of this proposal is to make it easier for signers in a Kubernetes cluster to publish their relevant trust anchors for the workloads in that cluster, and for workloads to access those trust anchors. This will make the overall use of certificates more secure and transparent.
How to Check Your Current Kubernetes Version
The kubectl version –short command will give you the current Kubernetes version running, both the client and the server version. If you want to know the version for the kubelet, which could be different if you are running Kubernetes on-premises, use the kubectl get nodes command, which will show you the version of the kubelet for each node running on it. This is important not to forget, especially in the case of updating versions to fix a vulnerability like the most recent one CVE-2023-5528, which has mostly to do with in-tree storage plug-ins.
The release notes for the current Kubernetes version are not going to be the same place to find out about the most recent Kubernetes CVEs - and as mentioned above, any patch releases or patch versions are not necessarily indicated in this release.
Conclusion
Kubernetes version 1.29 is absolutely packed with security features that are relevant to all teams responsible for Kubernetes security. Upgrading is not a trivial process, but with these new capabilities, there is plenty to gain on the other side in terms of improved uptime, observability and reduction of the overall attack surface. If your team is running a fair amount of clusters today and struggling to gain proper visibility around your security posture, get in touch with KSOC today to see how we can help with Kubernetes risk visibility in real-time, as well as creating a basic asset inventory.
