cgroups fundamentals

The Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O) of a collection of processes.

What it is

While Linux Namespaces isolate what a process can see (providing a dedicated network stack and process tree), Control Groups (cgroups) limit what a process can use.

Developed by engineers at Google in 2006 (originally called “process containers”), cgroups are a feature of the Linux kernel that allows an administrator to allocate strict quotas of physical resources (CPU, Memory, Network Bandwidth, Disk I/O) to specific groups of processes. Without cgroups, containerization as we know it (Docker, Kubernetes) would be impossible, as a single runaway process could consume 100% of a physical server’s resources, starving all other containers on the node.

Why it matters

If you deploy a Java application and a Node.js application to the same physical server, the Java JVM might decide it wants to allocate 32GB of RAM. If the server only has 32GB total, the Node.js application will crash with an Out of Memory (OOM) error, even though it did nothing wrong.

cgroups mathematically enforce resource boundaries. By placing the Java application into a cgroup with a strict 4GB memory limit, the Linux kernel will forcibly kill the Java application (OOMKill) the moment it attempts to allocate 4.01GB, preserving the remaining 28GB for the rest of the system.

How to implement

cgroups are typically managed invisibly by container runtimes (like containerd or cri-o) and orchestrated by Kubernetes using “Resource Requests and Limits.” However, under the hood, cgroups are exposed to the user space via a virtual filesystem, usually mounted at /sys/fs/cgroup/.

To manually create and enforce a cgroup on a Linux machine without Docker:

Create the Group: Create a new directory inside the virtual filesystem. The kernel automatically populates it with configuration files.
```
mkdir /sys/fs/cgroup/memory/my_app_group
```

Define the Limit: Write the maximum byte limit into the configuration file.

# Limit to 500 Megabytes
echo 500000000 > /sys/fs/cgroup/memory/my_app_group/memory.limit_in_bytes

Assign the Process: Write the Process ID (PID) of the application into the tasks file.
```
echo 12345 > /sys/fs/cgroup/memory/my_app_group/tasks
```

The kernel immediately restricts process 12345. If it requests more than 500MB, the kernel terminates it.

Common mistakes

CPU Limits (Throttling vs Killing)

Memory cgroups and CPU cgroups behave fundamentally differently.

If a process breaches its Memory limit, the kernel terminates it instantly (OOMKill), because memory cannot be compressed.
If a process breaches its CPU limit, the kernel does not kill it. Instead, the kernel throttles it, putting the process to sleep for a few milliseconds, slowing it down until its average CPU usage falls below the quota. Many developers misdiagnose latency spikes in Kubernetes; the application isn’t slow because the database is slow, it is slow because the cgroup CPU throttle is constantly pausing the execution threads.

Ignorant Runtimes (The JVM Problem)

Older application runtimes (like Java 8) were not “cgroup aware.” When the JVM booted, it looked at the physical server’s total RAM (e.g., 64GB) instead of looking at the cgroup limit (e.g., 2GB). The JVM would proudly set its internal heap size to 16GB, attempt to allocate it, and be instantly shot in the head by the kernel. Modern runtimes automatically detect cgroup limits and scale their internal memory management accordingly.

Verification

To verify if a container is being throttled or killed by cgroups in a Kubernetes environment, you must inspect the kernel’s tracking metrics.

View the exact reason a Pod terminated in Kubernetes:

kubectl get pod my-app -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}'
# If it returns "OOMKilled", the memory cgroup limit was breached.

To see if a running container is currently being subjected to CPU throttling by the cgroup, execute into the container and read the kernel statistics directly from the virtual filesystem:

kubectl exec -it my-app -- cat /sys/fs/cgroup/cpu/cpu.stat
# Look for 'nr_throttled' (number of times the process was paused) 
# and 'throttled_time' (total nanoseconds spent paused).

Sources & further reading

Linux Kernel Documentation: cgroups - Linux Kernel Organization