A recent article showed that over 900,000 Kubernetes endpoints (API servers) are sitting publicly on the internet. If you think that’s bad, remember that a Kubernetes API server being public is only about 5-10% of the security concerns that happen in many production environments when it comes to Kubernetes and container security.
Because of the vast security holes in most production environments, the overall concerns cannot be lumped up into one section of the cloud-native world. Instead, it needs to be split so each team can contribute to mitigating as much of the risk as possible.
The split, in many cases, ends up being Kubernetes Security and Container Security.
In this blog post, you’ll learn about what Kubernetes security is, what Container security is, and how they tie together.
Have you ever created a container image, whether it was with a Dockerfile or a CloudNative Buildpack and asked yourself where is the base container image coming from?
Many engineers have gotten comfortable with the idea of pulling ubuntu:latest or golang:latest from Docker Hub to build a container image with their application on it, but the reality is, those base container images are being built by other engineers. Because of that, you should still be concerned about what’s happening during the build process of the container image.
The screenshot below is an output from running docker scan ubuntu:latest , which is a vulnerability scan command utilizing Snyk as a backend. As you can see, there are 21 vulnerabilities found. Although many of these are considered low and medium vulnerabilities, that doesn’t mean they aren’t high vulnerabilities for certain security teams.
Container security is all about two things:
There’s also talk from teams about ensuring that the application is secure during this process, but the truth is, ensuring that an application is secure should happen a few steps before a container image is even thought of. Once the code passes certain security-related scans, then the application can be containerized.
In short, ensuring an app is secure should happen before the containerization process.
Kubernetes security at a high level breaks down into two pieces:
Dev security is all about securing Pods. Inside of a Pod is a container, and that Pod’s responsibility is to run the container so users or other Pods can access the application running inside of said Pod. With Pod security, there are many aspects including who has access to the Pod, what Pods have access to other Pods, and where Pods are running. For example, if Pods are deployed via a higher level controller, like a Deployment or DaemonSet, and a service account isn’t specified in any of the Kubernetes Manifests, that means the default service account has access to every single application running on the Kubernetes cluster. If that service account gets compromised, so does every single containerized app.
Ops security is all about securing the clusters (servers) and networking. From a cluster perspective, even though Kubernetes abstracts many infrastructure pieces, there are still servers running the environment. Even if you’re using a cloud service like Azure Kubernetes Service or Elastic Kubernetes Service and the control plane is abstracted away from you, there are still servers running the Worker Nodes. Because of that, how the Worker Nodes are operating and who has access to them is crucial to think about. From a networking perspective, there are so many pieces. For example, pod-to-pod (east west traffic) is completely unencrypted. Service-to-Service communication is completely unencrypted. That means you may want to think about a Service Mesh or a network framework like Calico that’s security-focused. When you have a frontend application running as a Service with a load balancer in front of it, you have to ask yourself how people are accessing the load balancer. Is it exposed over port 80 with zero SSL?
There’s a lot to think about when it comes to Kubernetes security and many practitioners believe that there should be a full team in itself that’s solely focused on the security aspects of running Kubernetes.
The truth is, you can’t have one without the other. You can’t use Kubernetes without a containerized application to orchestrate, and you can’t deploy a containerized application and expect it to scale without an orchestrator like Kubernetes. Because of that, security must be thought of separately when it comes to Kubernetes and Containers, but at the same time, it must be thought of in tandem.
Because one must come before the other, many security professionals like to think about container security before Kubernetes security. The reason why is because without containers, you would have nothing to orchestrate in a Kubernetes environment. If containers and orchestration are new to you or your environment, your first step will most likely be containerizing an application. Once the application is containerized, it can then be managed by an orchestration platform like Kubernetes.
With that being said, your best bet is to secure and mitigate as many vulnerabilities as possible from a containerization perspective. Before deploying the containerized apps to Kubernetes, take on a security-first mindset and mitigate Kubernetes vulnerabilities. Once the Kubernetes vulnerabilities that you want to worry about in your environment are mitigated, then you can deploy containerized apps.
Although this process will take longer than simply containerizing an app, deploying it to Kubernetes, and worrying about security later, you’ll have a much larger headache of dealing with a breach if you don’t follow proper security practices from the beginning.
KSOC’s upcoming blogs will be going into far more detail around this topic, but to get your mind thinking about it before we go more in-depth, you should have a few key components in mind when it comes to Kubernetes security.
The first is cluster sprawl. There are teams that have one Kubernetes cluster, but more likely than not, they’ll need more than one. There are a lot of pieces that go into having multiple clusters, including multi-tenancy for many resources (users, applications, etc.) and where the clusters run. With cluster sprawl comes configuration sprawl, which then comes vulnerabilities if multi-cluster environments aren’t thought of with a security-first mindset.
The second is Kubernetes benchmarks, which typically comes down to some known standards around CVSS scoring, CIS, NIST standards, and OWASP. Most of the standard severity scores come from the National Vulnerability Database (NVD), which most tools like Checkov and Snyk use to match vulnerabilities with severity. CTO and Co-Founder of KSOC Jimmy Mesta recently wrote the OWASP Top 10 For Kubernetes which you can find here.
The third is access control and RBAC. RBAC is a topic in Kubernetes that a literal book can be written around because of the vast amount of information. The reality is that without proper RBAC, clusters and Pods are open to a massive amount of vulnerabilities. A common example is how organizations run Pods. Pods are run in a Kubernetes cluster with a Kubernetes service account, and many organizations use the default service account, which is a service account that has access to every Pod on the cluster. When it comes to application and namespace segregation, using the default service account can open many security vulnerabilities.
If you’re interested in going further into these topics, check out what KSOC is doing by taking Kubernetes security to the next level with automatic remediation so you don’t have to worry about “what’s next” after a scan is complete.