“Shifting Security Left” has become the mantra of DevOps teams everywhere. The objective is to incorporate security “early” in the development lifecycle so that failing security “gates” do not force projects to significantly backtrack before release. This same mindset can apply to the design of platforms such as Kubernetes, and you can’t get much farther left than the system architecture that is informing its development. Here are three concepts to add to your Kubernetes security best practices, so system and security architects can implement a Kubernetes platform securely from the start: Role Based Access Control, Admissions Control, and Service Mesh design.
Role Based Access Control has become the industry standard for managing authorizations in a Kubernetes cluster. RBAC can be applied to both human users and the service accounts associated with workloads running in the cluster. It is configured primarily through four Kubernetes objects:
In addition to these foundational constructs, you can also aggregate roles together using an aggregationRule field in a ClusterRole object. ClusterRoles matching the selectors defined in this field are “merged” in the new role, giving you ways of creating more complex roles out of numerous simpler ones.
RBAC provides a robust and widely accepted way of managing authorization within your cluster. Planning your RBAC deployment does need to take into consideration the possibility of a “break-glass” scenario: if, somehow, the roles and role assignments needed for cluster administration were deleted, your cluster may become unmanageable. To avoid this, assign certain “break-glass” user accounts to the system:masters group. Because members of this group have essentially unrevocable super-admin powers, these accounts should never be used for normal day-to-day operations. Instead, they should be held in reserve with authentication material carefully stored in a secure manner and only used when other means of controlling the cluster have been lost.
A related consideration for RBAC implementation is the Kubernetes authentication method for users accessing the cluster. If users are not securely authenticated, they may be able to assume roles for which they were not assigned, defeating the purpose of any RBAC implementation. Built-in Kubernetes authentication methods are mostly designed for the workload/service-account use case. As such, they tend to rely on single-factor shared secrets. For human users, you should investigate integration with external authentication systems— such as your existing SSO solution— so that you can more easily enforce mapping cluster identities to a single user and harden authentication with MFA.
An Admissions Controller integrates with the Kubernetes API to function as a gatekeeper for incoming container images. It evaluates the images based on a configured list of rules and determines if they can be admitted to the cluster or should be rejected. The simplest implementation would be a straight deny-list (“block these images, allow all others”), but this will quickly become either obsolete or unsustainably long and unmanageable. You might also consider a straight allow-list: “admit these images, reject all others.” Ideally, the admitted images should be identified by an image digest (a sha256 or similar), since container image tags are mutable. This approach is doubtless more reliably secure, but quickly becomes difficult to manage as your cluster grows in size and complexity (and your security team has less ground-truth knowledge about what images should be running for all workloads). Eventually, you will want logical rules such as “reject containers with a vulnerable version of log4j”. This is the most dynamic and maintainable approach to admissions control, but also requires a fairly deep level of knowledge about your supply chain that may not be available in all organizations.
In addition to deciding whether images can or should be admitted, Admissions Controllers can also apply certain security policies to the admitted images. This allows you to enforce things like limiting what containers are allowed to run as root/privileged users. When you combine the admissions and security policy functions of admissions controllers, you can think of them as analogous to an authentication and authorization engine for containers. Two popular policy engines for Kubernetes admission control are Kyverno and Open Policy Agent (OPA).
If Admissions Control is about controlling what containers can run in your cluster and with what privileges, a Service Mesh determines what they can access once they are there. Though Kubernetes has built-in mechanisms for service discovery, a service mesh enables additional layers of security. First, a service mesh establishes mutual TLS (mTLS) between each service, validating their identities securely. This can be used to support a zero-trust architecture within your cluster, ensuring that connections between services cannot be spoofed, intercepted, or replayed. Second, a service mesh can be used to enforce fine-grained rules about what services are able to speak to one another across your cluster. In this capacity, it provides effectively an Access Control List (ACL) for your workloads. Finally, many service meshes can also be used to either replace or compliment Kubernetes’ native Ingress/Egress controls. This allows a service mesh to take on some of the work of a load balancer, but can also be used as a basic firewall, limiting ingress/egress to designated external endpoints. You can of course open up such ingress/egress to the world, but you could also use it to require traffic to pass through a proxy server first. This wouldensure only users who have authenticated with your IdP have access to the service.
Architecting these systems well from the start is vital, but there will still be a need for ongoing monitoring of your cluster state and adjustment of your implementations as your clusters change or grow. KSOC allows you to enforce Kubernetes security best practices along these three dimensions (and more). For example, KSOC can aggregate and easily find over-permissions in Kubernetes RBAC, providing the visibility you need to compare the intended architecture of your cluster with its actual state. Kubernetes admission controllers can be tricky to enforce, given that they can make breaking changes in application delivery and deployment. That’s why we provide a dry run view that shows what we would block, based on your policies, in a single, searchable place to share with the SRE team in preparation for enforcement. Most importantly, KSOC shows your posture in real-time, not in a point-in-time scan that might miss half your deployments. This gives you the concrete data to catch potentially serious incidents, as well as design future refinements to your security architecture.
A good security architecture for Kubernetes will map users with the right roles and permissions, provide assurances about the quality and security of workloads running the cluster, and limit internal traffic based on expected patterns of behavior to prevent services being abused for lateral movement throughout the cluster. Kubernetes RBAC, admissions control, and service meshes give you the tools to achieve these goals. Of course, security is not a once-and-done endeavor, and real-time security is not an option in an environment as dynamic as Kubernetes. Contact us for a demo of how KSOC can help your team along these dimensions, and more.