How many of us actually know how the Kubernetes pod lifecycle, or the Kubernetes container lifecycle, actually works? The Kubernetes workload is often described as ‘ephemeral, ’short-lived’ and there is no shortage of repeats for the ‘cattle versus pets’ description of servers in cloud native app development. In this blog, we will discuss the lifecycle of the Kubernetes workload and why it matters.
See why real-time KSPM is a requirement
The top 14 KSPM misconfigurations across hardening and incident response make real-time Kubernetes Security Posture Management (KSPM) a non-negotiable for any Kubernetes security program.
Pods can be confusing little things. They are the atomic unit of Kubernetes, and like atoms in physics there are certain sub-atomic “particles” within them. In the case of pods, these are containers and init-containers. The room for confusion is in conflating containers and pods into an equivalent unit. Indeed, most guides (ie, hello, world demos) use single-container pods with no init-containers, making it easy to think of a pod as just another word for a container. However, because pods operate at a higher level of abstraction, mapping container constructs onto a pod does not always work. This article will look at one such mis-match: the stages of the Pod lifecycle.
Let’s start by making sure we are all on the same page regarding the stages of a container’s lifecycle. Kubernetes defines three basic phases for a container: Waiting, Running, and Terminated. The status of the containers in a pod can be seen by using the kubectl describe pod pod-name command. Waiting is a slightly complex phase, as it encompasses everything that happens before a container is “running,” including waiting to be scheduled, pulling an image, and the startup process (things like applying a Kubernetes secret object). Kubernetes uses a separate Reason field to track why a container is in the waiting phase. Running and Terminated are more or less what they say on the tin, the only nuance is that Terminated encompasses both containers that have exited normally and those that have existed due to some sort of error. As with the Waiting phase, kubectl reveals a Reason field and the exit code for terminated containers to shed more light on this.
In the most basic case— a single container pod— the container phases map pretty easily to pod lifecycle phases. The pod lifecycle begins with a Pending phase— in a single-container pod this essentially encompasses the Waiting phase of the container lifecycle. Both pods and containers have a Running phase, and in a single-container pod these are essentially equivalent (we’ll get to some nuance here in a moment). Finally, Kubernetes provides two pod lifecycle phases that encompass the Terminated container status: Succeeded, which in a single-container pod indicates the container has exited normally/successfully, and Failed, which would indicate an exit in an error state.
Viewed from the perspective of a single-container pod, this is an almost 1:1 mapping, and one might be tempted to conflate the container and pod lifecycle phases. Remember, though, that pods are a different level of abstraction from containers. This might seem like a distinction without a difference, but it turns out to be quite important. Let’s look at why.
At first glance, perhaps the most important distinction between a Pod and a container is that Pods can contain more than one container, including init-containers. The biggest consequence of this for our purposes is around defining the Running phase of the Pod lifecycle. It turns out that the Running phase for a pod does not indicate that the container is up and running, but rather that at least one of the containers has made it at least as far as the Running phase of the container lifecycle. There are some nuances here to unpack. At least one container means that the containers in the pod can be at multiple stages in the container lifecycle. In fact, they could be at any phase in the container lifecycle. As an example, you could have containers that are Running while others are still Waiting (perhaps still pulling their image). This case makes a fair amount of sense— we intuitively expect a larger image to take longer to pull than a smaller one, so the fact that the smaller container might make it to the Running phase first is not surprising. Perhaps more surprising, however: at least as far as the Running phase means that we can also have containers that have Terminated in a Pod that is at the Running lifecycle phase so long as those containers are in the process of being restarted.
At first glance, this seems counter-intuitive, but it stems from the level of abstraction that a pod represents. A Pod is not a container (or even a collection of containers), it is a Kubernetes API object. It represents an instance of an application or service composed of one or more containers. As a Kubernetes API object, it also represents the “desired state” of the cluster, not the “actual state” of the containers declared in the Pod manifest. In other words, the Running phase of a Pod’s lifecycle is only indirectly connected to the status of the containers in the Pod. What the pod’s lifecycle represents is whether the API object “pod”, representing a desired instance of an application, has been successfully instantiated in the cluster. Instantiated here means something like: scheduled, images pulled, and manifest in the hands of the kubelet so that the desired state of deployment can be maintained. In other words, it is answering the question “is this object now being managed by Kubernetes?” Kubernetes does not guarantee that an object it is managing is actually running or available at any given moment (in fact, it explicitly treats many objects as ephemeral), only that the cluster is continuously striving to make it running/available according to the specifications in the object’s manifest.
Seen at this level of abstraction, we need new definitions for the pod lifecycle stages:
These pod lifecycle stages are taken from the point of view of the Kubernetes API, which is concerned about “desired state.” That’s not necessarily the most helpful view from the perspective of a user, though, who mostly wants to know the “actual state” of the objects in a cluster (ie, can I connect to this object right now?). In addition to seeing the status of actual containers in the pod using kubectl, Kubernetes also provides “Pod Conditions” that provide additional information about the Pod’s readiness for use (it’s “actual state”). These are:
If you want to understand the lifecycle of your Kubernetes clusters in detail, you will want to keep the Kubernetes audit logs. Kubernetes keeps audit logs for terminated containers and performs methods to ensure this doesn’t consume all available storage on the node. However, this is not a fail-safe method. For example, if a container fails many times or the pod is otherwise not on the same node, the containers’ and their corresponding logs are lost.
You will want to have some observability solution or other way to keep your pod logs so you can understand what is happening in your clusters over time, above what Kubernetes offers by default. This is called cluster level logging, and the storage and lifecycle of this storage will be separate from that of your actual environment.
Kubernetes allows you to trigger events at certain points in the lifecycle. You can do this right after a container is created or right before a container is terminated. There are difference between init containers and lifecycle hooks here, in accordance with the the nuances described above, so it’s important to keep those in mind.
In understanding the Kubernetes workload lifecycle, pods are often thought of as another name for containers. But understood correctly pods are Kubernetes API objects representing “desired state” for the most atomic unit of work on a cluster. As a result, the phases or stages of the Pod lifecycle indicate the status of that API object (is it being “managed” by Kubernetes to achieve desired state?), not the “actual state” of the workload or its readiness to perform work. This is a subtle but important distinction that points to the level of abstraction for Kubernetes objects and its promise not that a workload is available at any given moment but that the cluster is striving to make it available.
KSOC is the first and only real-time security solution for Kubernetes that takes into account the Kubernetes lifecycle. Given how complicated it is and how quickly workloads cycle through different stages on any given day, it is impossible to secure Kubernetes without also taking into account the lifecycle. A point in time Kubernetes scanner creates inactionable results by definition because the workloads will have changed since the scan was performed. And it is impossible to figure out where to remediate in the environment when the container accessible through the manifest misconfiguration no longer exists. See this blog for more information on how the lifecycle can make Kubernetes security actionable and accurate.
If you are ready to see real-time Kubernetes security in action, contact us for a demo.