Introduction
In our article “To DIY or not to DIY”, we wrote about the central value proposition of any managed service:
The claim any managed service makes is that they are handling certain problems for you, freeing up your resources to deal with others.
As more and more organizations move their workloads to managed services in the cloud, they often assume that their cloud provider will handle security for them. While it's true that using a managed service may improve your security posture in certain respects, it's important to remember that these providers cannot “do” security for you. The ultimate responsibility for securing your applications and data still falls on you. In this article, we’ll explain why.
The Limits of a Managed Kubernetes Provider
For a cloud provider to offer something as a managed service, it has to meet a few criteria. It must be stable, meaning that their customers can rely on it. It must remove some management burden from their customers (otherwise, what’s the value proposition?). And most importantly for our purposes today: it must be something the provider can offer in a scalable way (otherwise, what’s the value for them?). This final point is what lies beneath the “Shared Responsibility Model” of security in the cloud.
The reason for this is that scalability requires consistency. If you are providing a managed widget deployment service, but every widget requires unique configuration, infrastructure, and special attention, you’re either (a) going to go out of business or (b) going to have to charge so much for this managed service that customers will decide to just manage it themselves. A managed service only makes sense if the service on offer is consistent for every customer, creating an “economy of scale” that allows the provider to control their costs while offering an affordable option to their clients.
To illustrate with a simple-ish example: think of a managed database service. There are many aspects to deploying a database, but some of them can be turned into a consistent pattern— ie, deploying the “host”, it’s operating system, and the database software itself. In contrast, there will be certain other pieces which vary from database to database, such as the table schema and actual data content. A managed service can take on those consistent parts, but no managed provider will “manage” your data for you— it’s literally your “customer data”, you own it and are responsible for it.
When mapping this concept to Kubernetes, the notion of “customer data” is a bit fuzzier. Your deployed applications and services clearly fit into this category, but so might things that modify or extend the control plane your provider is theoretically managing for you (for example, a service mesh or the CNI), including some things your provider may have installed for you, but that you can change afterwards. This results in three categories of things that fall outside the scope of “management” for a managed Kubernetes service: (1) customer-specific workloads (the analog to customer data), (2) cluster configurations that have to be established by the customer, and (3) cluster configurations that customers may change, and therefore make “un-manageable.”
EKS Security as an example
As a case study, we’ll examine Amazon’s managed Kubernetes service, EKS. EKS bakes certain security decisions into the platform, but there are still plenty of things that fall into the categories above.
Kubernetes misconfigurations
As a paradigmatic example of a security configurations that has to be left to the customer: AWS allows you to expose your Kubernetes API to the internet. Employing Kubernetes security best practices and taking the API off the internet requires some alternative mechanism for connecting to the control plane. What that looks like will vary from customer to customer based on things like the identity provider you use, where you store your cluster configs, what CICD or gitops tooling you use, etc. AWS cannot make those decisions for you (though they might hope you use their own tooling), and without knowing what you will decide they almost have to default to putting the API on the internet, leading many customers to unwittingly adopt a serious Kubernetes security vulnerability right out of the gate.
Kubernetes tools
There are also examples of the third category: things that, if AWS (or any other managed provider) made decisions about, they wouldn’t be able to manage for you after deployment. These include admissions control policies and Role Based Access Control (RBAC). It would be possible, certainly, to deploy a cluster for you that included something like Kyverno or OPA Gatekeeper and a default set of rules. Or to start the cluster with a default set of RBAC policies and roles you could assign. But as soon as you began to change and tweak those rules/roles, the managed provider would cease to be able to “manage” them for you (lest they break the rule logic you had created), they would become “customer data.” So most providers, including AWS EKS, choose not to provide even a starter set and leave it entirely up to you to configure admissions control and RBAC for your cluster. These are increasingly critical components of any Kubernetes deployment, so this represents a major responsibility for the customer. AWS isn’t being stingy here, its simply not possible for a managed provider to efficiently include these security components as part of a scalable/consistent pattern that will be valuable to all their customers.
Kubernetes runtime security
Finally, there are of course your unique workloads: the applications and services you are deploying on your cluster. EKS (or any other provider) cannot meaningfully manage what you deploy to your cluster or how you configure it. These workloads will likely represent the largest attack surface on your cluster, so their contents and configurations are extremely important to Kubernetes security. But since they fall squarely in the realm of customer data, the security of those workloads falls entirely on your shoulders.
You can’t rely on managed cloud providers for security
Managed providers take advantage of the economy of scale that can be achieved by abstracting certain aspects or patterns common to every deployment of a service. They then leave the unique bits— the customer data— up to you, the customer, to provide and manage. For Kubernetes this becomes a bit fuzzy because the “customer data” consists of running workloads and configurations that may alter the control plane. Much of what goes into securing a Kubernetes cluster— including things like authentication, RBAC, admissions control, the use of a service mesh, and ultimately the configuration and security of the workloads you are running— falls into the customer data category. As a result, while managed providers may bake some hardening into their deployment patterns for Kubernetes, they cannot “do” security for you. As a result, security is always a shared responsibility between the cloud provider and you, the customer.
What Kubernetes security should look like
Kubernetes is dynamic and workloads spin up and down in a matter of minutes with thousands of deployments each day. Today’s Kubernetes Security Posture Management (KSPM) tools, including anything provided by the cloud providers themselves, take an interval scanning approach to Kubernetes. This means misconfigurations are scanned for on a polling interval which could range from minutes to hours. By definition, the alerts you get from an interval scanning method in Kubernetes are inactionable and impossible to operationalize.
