Kubernetes Security Trends for the Year Ahead

There are two pitfalls to avoid when writing a “year ahead” post. The first is writing a really boring article that just says “everything next year will be the same as last year.” Cool, nobody needed to read that. The second is making all dark horse predictions and advising your readers to bet the farm on things that are unlikely to (all) come true. We’re going to try something a little different in our year ahead post: while recognizing some likely thematic continuity between 2022 and 2023, we’re going to highlight the questions Kubernetes security teams need to be asking as these trends dominate the cloud native landscape in the year ahead.

You Get an eBPF, Everybody Gets an eBPF

One of the hottest topics at KubeCon NA in 2022 was eBPF, a technology that allows performing certain operations directly in the kernel space. The excitement about the possibilities this brings was almost palpable. Already underway are efforts to manage networking (see Istio Ambient’s roadmap) and to do monitoring/security analysis via eBPF. 2023 is likely to see this work move into more and more areas of the Kubernetes stack. One way of thinking about this: Kubernetes is often analogized as the “operating system of the cloud.” By integrating Kubernetes with the kernel, that analogy is becoming more reality.

eBPF is touted for providing both performance and security improvements, but it is not without security risks. Will this be another case of development outpacing security (and security playing catchup for years to come) or will the inherent elevated privilege that comes with eBPF prompt security to be a first-order consideration in the work moving forward? As the excitement continues to grow, here are three concerns Kubernetes security teams are going to need to keep an eye on:

1. Once you’ve moved an operation into the Kernel with eBPF, most traditional monitoring tools (whether security or otherwise) are no longer able to see or track it. That’s concerning when you think about operations like networking suddenly being invisible to your security team. Now, of course, there are ways of monitoring the kernel and most of the tools that are making this eBPF transformation have their own internal ways of producing metrics and logs. So security teams and tools are going to need to hook into those and rely less on traditional security agents that monitor “normal” (ie, non-kernel) parts of the Kubernetes stack.
2. Running tasks in the kernel grants them elevated privileges, making them harder to control. There are built-in protections in the eBPF framework, but these largely focus on protecting the kernel itself from denial of service events (whether an attack or the result of bad programming) and do not necessarily address concerns about confidentiality of data integrity. Before you grant a module the extra power that comes with eBPF, you should make sure you understand the risks you’re assuming to your systems and data. As platform engineering teams flock to eBPF-based solutions, security teams are going to want to ask questions about how well these solutions have been designed and vetted for security concerns.
3. Vulnerabilities, including critical ones, have been found in eBPF implementations already. More will undoubtedly continue to be found. It’s only a matter of time before one becomes widely exploited (whether against Kubernetes or another technology stack exploring eBPF). If and when this happens, security teams need to understand what their patching strategy is going to be. This is new technology and many security teams will lack subject matter expertise. How much trust will you place in your operations team’s understanding of eBPF when responding to a serious vulnerability or potential exploitation? In that situation, are you comfortable updating in place or would you prefer to roll the affected nodes to make sure nothing with elevated privileges has lingering persistence? These sorts of questions are best thought out in advance and incorporated into your playbooks rather than left to decide in the moment.

SBOMs Become a Gatekeeper

SBOMs have been part of mainstream conversations around software supply chain security since the Solar Winds breach in late 2020. They became an on-the-horizon requirement for Federal contractors in 2021 with Biden’s EO. In 2022 numerous tools focused on either CICD or dependency analysis began rolling out the capability to generate and even sign SBOMs. Many other security tools, such as KSOC, began rolling out the capability to read and ingest SBOMs, helping teams make decisions about the artifacts they have running in their clusters. This momentum is likely to continue in 2023, even as Congress has just stripped the SBOM requirement out of the most recent NDAA. Generating SBOMs appears to be following the 80/20 rule: about 80% of the problem has been solved, the nuances and edge cases that compose the final 20% might take years to completely iron out.

The next big frontier with SBOMs, though, is how to use them. Here are three capabilities for using SBOMs that Kubernetes security teams will want to explore in 2023:

1. Analyzing what’s running in your cluster. For example, you may may want to discover which artifacts running in your cluster contain a particular vulnerable package in their dependency trees. Several tools, including KSOC, now have the capability to help with this (and more developing it).
2. Evaluating the contents of an internal artifact repository. Similarly to the above, this could be used to find artifacts containing a particular vulnerable package. This approach could, in theory, be used to prevent certain vulnerabilities being deployed by either labeling those vulnerable artifacts or yanking them from the repository altogether.
3. Dynamic SBOM evaluation at admissions. The idea here would be using an admissions controller to check an artifacts SBOM on the fly before deployment and make an allow/disallow decision based on its contents. To date, this has been experimented with in very limited ways such as (a) verifying that a (signed) SBOM exists for the workload (but not evaluating its contents) or (b) checking for the presence of a single vulnerable package (ie, Log4J) in that SBOM. More generalizable policies, such as “No Packages with Known Critical Vulnerabilities” have not yet been widely implemented, but we hope to see work in this area during the year ahead.
   
   

TLDR

The Kubernetes community is going to continue pursuing eBPF solutions to many parts of the Kubernetes stack. There’s a lot to be excited about in that, but security teams need to understand the risks of moving so much to the kernel and adapt their practices accordingly. Similarly, security teams have been talking about the need for SBOMs for a few years now. SBOM generation is becoming a more and more common practice, now it’s up to security teams to know what to do with them.

The KSOC platform provides customers visibility into their clusters, context around security issues, and flexibility in how to respond to them. For more information on KSOC’s cloud native Kubernetes security platform, schedule a demo.