In this section

Linux Container Escape Investigation: Reading a Kubernetes Node's Records After nsenter

Module 0

Introduction

A container is not a small machine. It is a set of ordinary processes on the host, given their own view of the filesystem, the process list and the network by the kernel's namespaces, and limited by cgroups.

There is one kernel, and it belongs to the host. That is why a container's activity can be read from the node it runs on, and why a container that gets out reaches everything on that node.

This lesson reads INC-NE-2026-0422, a container escape on SRV-NGE-MCR-APP01, a Kubernetes node the case describes as Ubuntu 22.04. The runtime monitor alerted on a pod running a shell and touching the host's filesystem, and a live acquisition was taken at 04:00. Every record in the lesson came from the node.

1. Applicationthe container's own log: requests, errors, what the app printed2. Containerthe runtime's records: images, pod specs, container IDs3. Host kernelprocesses, cgroups, namespaces, audit records: one kernel for all4. Clusterthe orchestrator's audit log: API calls and who made them

Four layers of records about one container, and the node holds or reaches all of them.

The figure is the map Module 8 works from. Each layer keeps its own records, and the node is where all four can be read: the application's log through the runtime, the runtime's records directly, the kernel's records because it is the node's kernel, and the cluster's audit log because the node runs a control plane in this case.

01

What the Node Sees

Container processes in the host's process list

The host's process list at 04:00 includes the container runtime and everything it started:

  1200      1  root   04-20 14:22  /usr/bin/containerd
 25100   1200  root   04-20 14:22  nginx: master process /usr/sbin/nginx
 25200   1200  65534  04-20 14:22  /bin/prometheus --config.file=/etc/prometheu
 28400   1200  root   04-20 14:22  node /app/server.js
 28500   1200  1000   04-20 14:22  node /app/api.js
 32001  28400  root   04-22 02:49  /bin/bash

containerd is PID 1200, and its children are the containers' main processes: nginx, Prometheus, and two Node.js applications. To the host they are ordinary processes with host PIDs. The web front end, webapp-frontend-7b4d9, is PID 28400, node /app/server.js, running as root. And one process stands out: /bin/bash, PID 32001, a child of that node process, started at 02:49:33 and still running at collection.

That is the first lesson of Module 8 in one record. The container's shell is visible from the host, with its host PID, its parent and its start time, because the host's kernel runs it. Inside the container it would have had a different PID and seen only the container's processes; from the node, it is one more line in the list.

The runtime's own records identify which container each process belongs to: the container ID, the image, the namespace, the host PID of its main process. That mapping is how an investigator turns a suspicious host PID into a pod name and an image, and it is what Module 8's lesson on which container is this teaches.

Without that mapping a host PID means little, because containers are created and destroyed constantly and the same PID can belong to different containers on different days. Here the mapping is direct: PID 28400 is the main process of container a8f7e6d5c4b3, image nge-registry.northgateeng.com/webapp-frontend:2.4.1.

The user column deserves a note. The front end's main process runs as root on the host, because the container ran as root and the node's kernel sees the same user ID. The API container beside it runs as user 1000. That difference is the first of three this lesson keeps coming back to.

The start times say something as well. Every container on the node started on 20 April, the day the node was last brought up. The bash shell is the only process in this part of the list that started on 22 April, which makes it stand out without reading its command at all.

Module 8 uses that contrast constantly: a container's processes usually start with the container, and anything that starts later inside it is either an application spawning workers or something worth explaining.

02

The Way Into the Container

A template injection in the application's own log

The application inside the container wrote its requests to standard output, which the runtime kept as the container's log. From 02:48 on 22 April:

02:48:11 stdout POST /api/render 200 12ms - body: {"template":"{{constructor....
02:48:22 stdout POST /api/render 200 8ms - response: uid=0(root) gid=0(root) ...
02:49:11 stdout POST /api/render 200 15ms - body: {"template":"{{constructor....
02:49:33 stderr Connection to 198.51.100.77:9443 established

The first request at 02:48:11 posted a template to /api/render containing JavaScript that reached the process object and ran the command id. The application logged the request body, and eleven seconds later logged the response: uid=0(root). The second request ran bash with a reverse shell, and at 02:49:33 the container's standard error recorded a connection to 198.51.100.77:9443 established.

This is server-side template injection: the application rendered user input as a template, and the template language could reach the runtime. It is the web door from lesson 0.4, opened inside a container instead of on a plain server. The difference is what the intruder arrived as: not www-data, but root, because the container ran its application as root.

The container's log held all of this only because the application logged request bodies and responses, which most applications do not. The log has 44 entries in the collection, and without the two that carry the template, the entry would have to be inferred from the shell's existence and its start time.

Module 8's lesson on what the runtime recorded covers where a container's logs live on the node, and how long they last once the container is gone.

The time matters too. The shell in the host's process list started at 02:49:33, the same second as the connection the container logged. Two records from two layers, the application's and the kernel's, agree on the moment the intruder got a shell.

The template body also shows the intruder testing before acting. The first request ran id and nothing else; the second, forty-nine seconds after the response, opened the shell.

That pattern, a harmless command to confirm execution and identity, then the real one, is common enough to be worth looking for in any application log that records request bodies. It also dates the moment the intruder learned they were root, which explains why the escape that followed needed no escalation step at all.

03

The Pod That Could Reach the Host

Three settings in one specification

A shell as root inside a container is not yet root on the host, and on a well-specified pod it never becomes one. What decides whether it can become that is the pod's specification. The three pods on the node:

pod                      hostPID  privileged  user  capabilities
webapp-frontend-7b4d9    yes      yes         0     ALL
webapp-api-3c1f8         no       no          1000  none
nginx-ingress-k2d9f      no       no          101   NET_BIND_SERVICE

The front end pod has hostPID set, so its processes share the host's process namespace and can see every process on the node, PID 1 included. It is privileged, which removes most of the isolation the runtime normally applies. Its application runs as user 0. And it has every capability.

The API pod beside it has none of these: no hostPID, not privileged, user 1000, no capabilities, a read-only root filesystem.

What let this container reach the host

Three settings, one escape

runAsUser 0

the application ran as root inside the container

hostPID

the container could see every process on the node, including PID 1

privileged, all capabilities

nothing stopped it entering the host's namespaces

The neighbor

the API pod beside it had none of the three

Each setting alone weakens the boundary; together they remove it. With hostPID the container can see the host's PID 1; with privilege and all capabilities nothing stops a process joining PID 1's namespaces; and as root it has the permissions to do it. The case's own root-cause finding names exactly these three.

The ingress pod is a third data point: it uses the host's network, which it needs to receive traffic, but nothing else, and it runs as an unprivileged user with one narrow capability. That is what a pod with a real reason to touch the host looks like. The comparison with the API pod is the useful part.

The same application team, the same namespace, the same node, and one pod built safely beside one built unsafely. Module 8's lesson on reach beyond the boundary reads specifications like these to say, before any record of an escape is found, whether a compromised container could have reached the host. Here the answer was yes before the intruder arrived.

Specifications like this usually exist for a reason someone once had. hostPID and privileged are sometimes set for monitoring agents or debugging and then copied into templates where they do not belong.

The investigation's job is not to guess the reason but to record that the setting was there, when the pod was created with it, and which other pods share the template, because they share the exposure. Module 8 treats the specification as evidence in its own right.

The specification also explains why the alert was about host filesystem access. A privileged container with hostPID can reach the host's filesystem through PID 1's view of it, without any volume mounted for the purpose, so the runtime monitor saw a pod reading host paths it had no mount for.

04

Crossing the Boundary

nsenter, setns, and a cgroup that changed

The host's audit records, written by auditd on the node itself, show the crossing in four lines:

02:55:11 setns  28901 28400 nsenter /kubepods/besteffort/pod-w 
02:55:12 execve 28902 28901 bash    /                          /bin/bash
02:55:33 openat 28902 28901 cat     /                          /etc/shadow
02:56:11 openat 28902 28901 cat     /                          /root/.kube/config

At 02:55:11 a process called nsenter, PID 28901, whose parent is 28400, the container's node process, called setns successfully. Its cgroup is still the pod's. One second later its child, bash, was executed, and its cgroup is /: the host's. Then cat read /etc/shadow, and cat read /root/.kube/config.

setns is the system call that moves a process into an existing namespace, and nsenter is the standard tool that calls it.

Pointed at PID 1's namespaces, from a container that can see PID 1 and has the privilege to join them, it gives a shell that is on the host in every sense that matters. The cgroup column is what shows it happened: a process whose parent chain leads into the pod, running in the host's cgroup.

These records exist because the node ran auditd with rules that tagged them, the keys container_escape and credential_access in the full records. That is a deliberate choice, not a default, and it is the reason this escape is visible at all.

Module 8's lesson on one process in two views covers what the same events look like on a node without those rules: a bash with an odd parent and no explanation.

The detection gap is in the case's brief: the alert came from the runtime monitor, not from these audit rules, and the case puts the gap between the escape and the alert at twenty-seven minutes. The records existed from the first second; nothing was reading them. That is a pattern this course has now met on four hosts, and Module 11 is where it is addressed.

The second file read matters more than the first. /etc/shadow holds the node's password hashes. /root/.kube/config holds the credentials the node's root uses to administer the cluster. Reading it is the moment the intrusion stopped being about one node.

The audit records also show why the history file alone would have been weaker. Root's history on the node starts with cat /etc/shadow; it has no line for nsenter, because nsenter was run from the container's shell, whose history, if it kept one, lived inside the container's filesystem.

Only the audit records join the container's process to the host's shell. Module 8's lesson on what remains after a container stops is about where that inside history goes once the container is gone.

05

From Node to Cluster

What root on the node did with the cluster's credentials

The cluster's own audit log, kept by the control plane on this node, recorded what came next:

02:57:11 sa webapp:webapp-sa    list   secrets  (cluster)
02:58:22 system:admin           get    secrets  production
03:05:11 system:admin           create cronjobs kube-system
03:08:22 system:admin           create pods     kube-system

At 02:57:11 the log records the web application's service account listing secrets across the whole cluster, and being allowed to, although root's history shows kubectl run with the node's administrator configuration at the same second; the two records disagree about the identity, and a finding should say so.

A minute later, the cluster administrator identity read the secrets in the production namespace. At 03:05 and 03:08 the same identity created a CronJob and a pod in kube-system, the namespace for the cluster's own components. Every call came from 127.0.0.1, the node itself, which is what an API call made from a shell on a control-plane node looks like.

On the host, root's history and the audit records show the rest: the secrets saved to a file, an etcd snapshot taken, both compressed and posted to 198.51.100.77, and a systemd unit, node-exporter.service, written at 03:05 to run /usr/local/bin/.node_exporter.

At collection that process was PID 31447, running as root with an established connection to the same address. The name imitates a real monitoring agent, the same trick lesson 0.6 found on two servers.

The whole sequence took thirteen minutes from the escape to the second kube-system object. So one injectable template in one container ended with the cluster's secrets and its datastore copied out, persistence on the node, and persistence in the cluster.

The cluster-level CronJob is the one most likely to survive a cleanup of the node, because it lives in the cluster's state and will be scheduled on whichever node the cluster chooses.

Each of those outcomes needs its own cleanup. The node's unit and binary are host work. The CronJob and the stray pod are cluster work.

The copied secrets mean every credential in the production namespace has to be rotated, because the intruder now holds them all, and the etcd snapshot means the same for anything else the cluster stored. None of that is visible from inside the container where it started.

That is the point Module 8 makes about containers and hosts. The boundary is a configuration, and when it fails the investigation has to follow the intruder into every layer the node can reach: the host, its credentials, and the cluster those credentials control.

06

What Module 8 Builds On It

Containers on the Linux host

Module 8 reads a container's evidence from the node: what the node records, the writable layer as a diff, identifying which container a process belongs to, reading image layers, one process seen in two views, the network namespace, reach beyond the boundary, what remains after a container stops, what the runtime recorded, and a workflow.

What Module 8 builds

Module 8

8.1 to 8.3

what the node records, the writable layer as a diff, which container is this

8.4 to 8.6

reading the layers, one process in two views, the network namespace

8.7 to 8.9

reach beyond the boundary, after it stops, what the runtime recorded

8.10

the container workflow

This case touches most of those. The host process list is which container is this and one process in two views. The pod specifications are reach beyond the boundary. The container log is what the runtime recorded. The audit records are the escape itself, and the cluster audit log is where the module hands over to cloud and cluster courses that cover the control plane in depth.

The module also keeps the distribution split in view, because container runtimes and their defaults differ between Ubuntu and RHEL nodes: Docker against Podman, AppArmor against SELinux, and different paths for the runtime's records. The habits to carry into it are the ones this lesson used. Read containers as host processes, with host PIDs, users and start times.

Map host PIDs to containers through the runtime's records, never by guessing from a process name. Read the pod specification before deciding whether an escape was possible. Follow PIDs and cgroups across the boundary, one parent at a time. And when the node is breached, follow its credentials into the cluster.

None of those habits needs an orchestrator to be present. A single host running a handful of containers with Docker or Podman has the same layers, minus the cluster, and the same boundary that a specification can weaken. Module 8 works on both, because many of the containers an investigator meets run on ordinary servers rather than in clusters.

It also covers what happens after a container is deleted, which is often before anyone investigates: the layers, logs and records that remain on the node, and those that go with the container. In this case the container was still running at 04:00; on many nodes it would have been rescheduled away, and the investigation would start from the node's leftovers.

Module 8's labs run containers on a lab host you control, with safe and unsafe specifications side by side, so that each boundary can be tested and each crossing recorded.

Do

Practice

Decide where the process left the container
prepared evidence

The prepared evidence below is the host's audit records from 02:55. Decide which record is the escape, which is the first process running in the host's cgroup, and which records show credentials being read. Each answer comes with its reasoning, so check it against the rows rather than only the choice.

Follow the PID and PPID columns from row to row, and watch the cgroup column. A process whose parent is in a pod's cgroup but which itself runs in /, the host's, has crossed the boundary.

Expect only a few rows. Audit rules that record everything would drown the node, so the rules that produced these were targeted, and that is why each row matters. Then read the paths in the last column. Once a process is on the host, what it reads first says what it came for, and here both files were credentials.

Then, on a lab host with a container runtime that you can rebuild afterwards, start one container with hostPID and privileged set and one without, and compare what each can see of the host's process list.

Run nsenter from the first and look at the new shell's cgroup in /proc, then try the same from the second and read the error. The difference between the two is the whole of this lesson's root cause.

A container is processes on the host's kernel, and the node can read them. Here the node's records showed a template injection in the container's log, a pod specified so that it could reach the host, nsenter crossing into the host's namespaces with the cgroup changing from the pod's to the host's, and the node's cluster credentials used to copy the cluster's secrets and plant persistence in it.

The next lesson moves to the running host itself: collecting from a server that cannot be switched off, and reading its memory for what the disk does not hold.