2. Container Architecture, Standards and Linux Internals

Introduction and Context

Containerization is often described as a lightweight alternative to virtual machines, but that description is incomplete. A container is not a miniature computer. It is a process, or group of processes, whose view of the operating system has been selectively isolated and whose resource consumption is controlled by the host kernel. Understanding this distinction is essential for designing secure platforms, diagnosing failures, selecting runtimes, and operating Kubernetes clusters responsibly.

Modern container platforms combine several layers of technology. Linux namespaces provide isolation; control groups regulate and measure resource usage; capabilities and security mechanisms constrain privileged operations; layered filesystems make images efficient to store and distribute; and container runtimes translate image and execution specifications into operating-system operations. Standards developed by the Open Container Initiative (OCI) provide common interfaces and formats, while registries distribute immutable image artifacts. Around these foundations, tools such as Docker, containerd, CRI-O, Kubernetes, and cloud platforms create higher-level workflows.

This chapter examines these foundations as an integrated architecture. It also distinguishes ordinary containers from virtual machines and stronger sandboxed runtimes. The objective is not merely to memorize components, but to understand what each component guarantees, what it does not guarantee, and how those distinctions influence engineering decisions.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain how namespaces, control groups, capabilities, and Linux security mechanisms contribute to container isolation.

2. Describe image layers, copy-on-write filesystems, and the relationship between image storage and container writable storage.

3. Distinguish image formats, runtime specifications, low-level runtimes, high-level runtimes, and registries.

4. Explain the purpose of OCI image, runtime, and distribution standards.

5. Compare containers, virtual machines, and sandboxed container runtimes in terms of isolation, performance, portability, and operational cost.

6. Apply these concepts to security, capacity planning, troubleshooting, and platform architecture.

Containers as Kernel-Mediated Process Isolation

A traditional application runs in a process on a host operating system. It can normally see host processes, use host networking, inspect filesystem paths, and consume resources subject only to ordinary operating-system permissions. A container changes this situation by placing the process in controlled operating-system contexts.

The container image supplies user-space content: executable files, libraries, configuration files, and metadata. The host supplies the kernel. Consequently, containers generally require kernel compatibility with the application. A Linux container does not carry an independent Linux kernel inside its image. This explains both the efficiency of containers and an important security fact: the container shares the host kernel, so a kernel vulnerability may affect the isolation boundary.

Container isolation is therefore layered rather than absolute. Namespaces control what a process can see. Control groups control how much of selected resources it can consume. Linux permissions, capabilities, seccomp filtering, and mandatory access-control systems restrict what it can do. The runtime assembles these mechanisms according to a specification and launches the process.

Namespaces: Constructing a Different System View

A namespace gives a process an isolated view of a particular global system resource. Linux supports several namespace types, each addressing a different aspect of process context.

The PID namespace provides an isolated process-number hierarchy. A process may appear as process 1 inside its container while having a different process identifier on the host. The first process in a PID namespace has special responsibilities: it can become the parent of orphaned processes and is expected to respond appropriately to termination signals. Poor signal handling by a container’s initial process can therefore cause graceful shutdown problems.

The mount namespace provides an independent view of mounted filesystems. It enables a container to see its own root filesystem and selected mounts without exposing the complete host mount arrangement. Mount isolation is a view-control mechanism, not by itself a complete access-control policy; permissions and other security mechanisms remain important.

The network namespace provides separate network interfaces, routing tables, ports, and firewall contexts. A container may have an interface connected through a virtual Ethernet pair to a bridge on the host. Container networking systems use this foundation to implement private networks, service connectivity, and address translation.

The UTS namespace isolates the hostname and domain name visible to processes. The IPC namespace isolates several forms of interprocess communication, including System V IPC objects and POSIX message queues. The user namespace maps user and group identifiers between the container and host. A process can therefore be root inside a user namespace while corresponding to an unprivileged identifier outside it. This mapping is an important defense, although it must be configured carefully.

The cgroup namespace provides a containerized view of control-group paths. The time namespace can isolate certain system clocks. These namespaces should not be understood as independent security walls. They work together, and a weakness or misconfiguration in one area may undermine the overall design.

Control Groups: Resource Governance Rather Than Visibility

Control groups, commonly called cgroups, organize processes into hierarchical groups and apply resource policies to them. They can limit, prioritize, account for, and sometimes freeze processes. Typical controls include CPU time, memory, block-I/O activity, process counts, and device access.

A memory limit, for example, establishes a boundary for the group’s chargeable memory. If the workload exceeds the available allowance, the kernel may reclaim memory or invoke an out-of-memory response. This is not the same as an application-level memory limit: the kernel may terminate a process without giving the application an opportunity to recover unless the application and platform are designed for that event.

CPU controls may assign a relative weight or impose a quota over a period. A CPU quota can prevent a noisy workload from consuming unlimited processor time, while a weight influences competition among groups. Resource requests and limits in orchestrators are higher-level abstractions that are eventually translated into such operating-system controls, although exact behavior depends on the runtime, kernel, and configuration.

Cgroups also provide accounting information. Operators can use usage statistics to detect throttling, memory pressure, unexpected process growth, or capacity imbalance. Thus cgroups serve both enforcement and observability functions. They do not isolate files, network identities, or kernel operations; those tasks belong to other mechanisms.

Capabilities and the Meaning of Privilege

The traditional Unix model treats root as broadly privileged. Linux capabilities divide many root powers into smaller units, such as changing network configuration, mounting filesystems, administering processes, or changing system time. A container can run as a user with a reduced capability set rather than receiving every privileged operation.

Capability management includes permitted, effective, inheritable, and bounding sets. The details matter because a capability available in one set may become effective under particular circumstances. A secure design removes capabilities that the application does not need and avoids the broad “privileged container” mode unless there is a compelling, reviewed requirement.

Capabilities are not a substitute for user identity. They should be combined with a non-root user, read-only filesystems where practical, restricted device access, seccomp profiles, and mandatory access controls such as SELinux or AppArmor. Security is a composition problem: each control reduces a different class of risk.

Seccomp, LSMs, and Device Boundaries

Seccomp, particularly seccomp-BPF, filters system calls that a process may invoke. A runtime can deny calls that are unnecessary or dangerous for ordinary workloads. A filter reduces attack surface but may interfere with applications requiring unusual kernel interfaces. Profiles should therefore be tested against actual workload behavior.

Linux Security Modules provide additional policy enforcement. SELinux uses labels and policy rules; AppArmor uses profile-based restrictions. These mechanisms can constrain file access, capabilities, and other operations even when ordinary Unix permissions would allow them.

Device access deserves special attention. Devices are represented through kernel interfaces and may provide powerful capabilities. A container that can access a host device may be able to bypass intended isolation. Runtime and orchestrator configurations should expose only explicitly required devices and should treat device passthrough as a high-risk decision.

Layered Images and Filesystem Semantics

A container image is a packaged description of a user-space filesystem and its execution metadata. Images are commonly constructed from layers. Each layer records filesystem changes relative to a lower layer: additions, modifications, and deletions. Shared lower layers allow many images to reuse common operating-system components.

At runtime, a storage driver presents lower, read-only image layers together with a container-specific writable layer. This often uses a union or overlay model. When a process modifies a file originating in a lower layer, the file may be copied into the writable layer before modification. This copy-on-write behavior reduces initial storage use but can introduce performance costs for workloads that modify many files.

Deleting a file from a lower layer does not remove it from the earlier layer. Instead, a deletion marker, often called a whiteout, hides it in the merged view. Therefore, careless image construction can leave sensitive or unnecessary data in earlier layers even when it is absent from the final filesystem view. Secrets should never be placed in an image layer merely because a later command deletes them.

Image layers are not the same as application data volumes. A writable container layer is normally disposable and tied to the container lifecycle. Persistent application data should be stored in a volume or an external storage service. Separating immutable application content from mutable state improves replacement, scaling, backup, and recovery.

A well-designed image uses a suitable base, installs only required packages, removes temporary artifacts within the same build stage where appropriate, and runs as a non-root user. Multi-stage builds separate compilation tools from the smaller production image. Image digests, rather than mutable tags alone, provide stronger reproducibility because a digest identifies content cryptographically.

From Image to Running Process: Runtime Architecture

The word “runtime” refers to several different layers. Confusing them causes design and troubleshooting errors.

A low-level runtime creates and starts the container process according to a runtime specification. Examples in the Linux ecosystem include implementations associated with the OCI runtime specification, such as runc and alternatives designed with stronger isolation. The low-level runtime interacts directly with namespaces, cgroups, mounts, capabilities, and process execution.

A high-level container runtime manages images, snapshots, lifecycle operations, and communication with low-level runtimes. containerd is a prominent example. CRI-O is designed primarily to provide a Kubernetes Container Runtime Interface implementation using OCI-compatible components. Docker Engine presents a broader developer-facing experience and historically uses multiple layers internally, including an image and lifecycle management layer.

Kubernetes normally does not directly invoke a developer command such as `docker run`. A kubelet communicates through the Container Runtime Interface with a node-level runtime. That runtime obtains images, creates filesystem snapshots, configures namespaces and cgroups, and asks a low-level runtime to execute the container process. Networking and storage are integrated through separate Kubernetes and runtime mechanisms.

This layered architecture provides replaceability. An OCI-compatible image may run across different compliant implementations, but compatibility is not magic. Kernel version, architecture, storage driver, security profile, networking, and runtime configuration can still affect behavior.

OCI Standards and the Container Supply Chain

The Open Container Initiative defines open specifications intended to prevent container technology from becoming dependent on one proprietary implementation. Its major areas include the image specification, runtime specification, and distribution specification.

The image specification describes how image content and configuration are represented. A manifest identifies layers and configuration, while content-addressable digests identify exact blobs. The runtime specification describes the configuration needed to run a container, including process, mounts, namespaces, user identity, and resource settings. The distribution specification describes interactions for transferring content between clients and registries.

A registry stores and serves image artifacts. It may be public, private, self-hosted, or provided by a cloud platform. A repository commonly contains multiple tags, while a tag is only a mutable human-readable reference. Pulling by digest improves supply-chain determinism. Signed metadata, provenance attestations, vulnerability reports, and admission policies can add trust signals, but none eliminates the need for careful source and build controls.

A secure supply chain begins before the image reaches a registry. Source dependencies should be reviewed, build environments should be controlled, credentials should not be embedded in layers, and generated artifacts should be traceable to source revisions. Registry access should use authentication, authorization, transport encryption, retention rules, and audit logging. In production, deployment policy may reject images with unacceptable vulnerabilities, untrusted provenance, or disallowed base images.

Containers, Virtual Machines, and Sandboxed Runtimes

A virtual machine emulates or virtualizes hardware and runs a complete guest operating system with its own kernel. A hypervisor isolates virtual machines and schedules virtual CPUs and memory. This generally creates a stronger boundary than an ordinary container, but it requires more storage, boot time, and memory overhead.

A container shares the host kernel and packages user space. It usually starts quickly and supports high workload density, but kernel sharing creates a larger consequence if the host kernel or runtime isolation is compromised. Containers are therefore well suited to microservices, build jobs, CI pipelines, and portable application packaging, provided the threat model is compatible.

Sandboxed container runtimes add another isolation layer. They may use a lightweight virtual machine, a specialized kernel, or system-call interception to reduce the consequences of container escape. Examples in the broader ecosystem include microVM-based approaches and user-space kernel designs. Such runtimes may improve tenant isolation but can introduce compatibility limitations, startup overhead, debugging complexity, or reduced access to certain kernel features.

The selection should be based on workload and threat model rather than on a simplistic hierarchy. Questions include:

• Is the workload trusted, semi-trusted, or hostile?
• Are tenants competing within one node?
• Does the application require special devices or kernel interfaces?
• Is rapid startup more important than a stronger boundary?
• Can the organization operate additional monitoring and patching layers?

An Applied Case: Separating a Multi-Tenant Build Platform

Background

A software company operates a shared continuous-integration platform. Teams submit builds that execute compiler tools, package managers, and test suites. The platform previously used ordinary containers on long-lived worker nodes. Build jobs were short, but they were supplied by many teams and occasionally processed untrusted pull requests.

Situation and Problem

The security team observed excessive memory consumption, inconsistent build performance, and concern that a compromised build could attack the host or inspect artifacts belonging to another job. Simply increasing node capacity would address performance symptoms but not the isolation problem.

Analysis

The architecture review identified several distinct concerns. Cgroups were needed to bound CPU, memory, process count, and ephemeral storage. Separate PID, mount, network, and user namespaces were required to prevent unnecessary visibility. Containers needed to run without root and with a reduced capability set. Build workspaces had to be ephemeral, while approved caches could be mounted selectively. Because jobs were semi-trusted and the consequence of a host compromise was high, ordinary namespace isolation alone was judged insufficient for the most exposed workloads.

Approach and Solution

The company divided jobs into trust classes. Internal, reviewed builds ran in ordinary OCI-compatible containers with strict seccomp and mandatory-access-control profiles. Builds triggered by external contributions ran in a sandboxed runtime using an additional virtualization boundary. Each job received a dedicated temporary workspace and explicit resource limits. Images were pulled by digest from an authenticated registry, scanned before use, and admitted only when provenance and policy requirements were satisfied.

The platform also improved process supervision so that job termination propagated correctly, configured log and artifact limits, and monitored cgroup metrics. The organization documented that image scanning was not a substitute for runtime isolation and that sandboxing did not remove the need for patching the host and runtime.

Outcome

The revised design reduced cross-job visibility, made resource consumption more predictable, and aligned isolation strength with workload risk. It also increased operational complexity: startup latency, runtime compatibility testing, and debugging effort became more significant. The company accepted these costs for externally influenced builds while retaining a faster path for trusted jobs.

Learning Points

The case demonstrates that container architecture is an exercise in layered risk management. Resource limits solve a capacity problem, namespaces solve visibility problems, capabilities and seccomp reduce privilege, and sandboxed runtimes address stronger isolation requirements. No single mechanism provides all of these properties.

Challenges, Risks, and Common Errors

A frequent mistake is treating a container as a security boundary equivalent to a virtual machine. Containers can be secure, but their security depends on kernel patching, runtime configuration, identity mapping, profiles, image hygiene, and operational discipline.

Another mistake is using the `privileged` setting to solve an application problem without analyzing the required capability or device. Privileged configurations may expose broad kernel functionality. Running as root inside the container can also create unnecessary risk, especially when user namespaces are absent or filesystem mounts are writable.

Resource limits may be omitted, incorrectly sized, or applied without observing real usage. A limit that is too low causes avoidable termination; a limit that is too high fails to protect the node. CPU throttling and memory pressure should be distinguished from application defects.

Mutable tags, unverified registries, and unpinned dependencies weaken reproducibility. A build that succeeds today may silently use different content tomorrow. Conversely, excessive pinning without an update process can leave vulnerable components in service. Reproducibility and patchability must be managed together.

Layered filesystems also create operational surprises. Heavy writes to the container layer can be slow, and data stored there may disappear when the container is replaced. Applications should use appropriate persistent storage and should be designed to handle termination and restart.

Governance, Professional Practice, and Emerging Directions

Container governance includes ownership of base images, vulnerability response, registry permissions, image retention, runtime configuration, and exception management. Organizations should define who may publish images, which registries are trusted, how images are signed or attested, and how quickly critical vulnerabilities must be addressed.

A mature platform treats infrastructure configuration as code and continuously tests runtime policies. Admission controls can enforce non-root execution, required resource limits, approved registries, signature verification, and prohibited host mounts. Observability should include process-level logs, image identity, runtime events, cgroup metrics, and security alerts.

Emerging developments include rootless container execution, improved user-namespace support, confidential-computing technologies, microVM-based workloads, eBPF-based observability and security tools, and richer software-supply-chain attestations. These developments do not eliminate fundamental trade-offs. Stronger isolation may affect performance or compatibility, while richer policy may increase administrative burden. Professional judgment remains necessary.

Best-Practice Checklist

A production container platform should generally:

• Use minimal, maintained images and build them through reproducible pipelines.
• Pin production artifacts by digest and maintain a controlled update process.
• Run applications as non-root users with only necessary capabilities.
• Apply seccomp, SELinux or AppArmor, and restricted device access where appropriate.
• Configure meaningful CPU, memory, process, and storage limits.
• Keep mutable application data outside the disposable container layer.
• Protect registries with authentication, authorization, encryption, auditing, and retention policies.
• Separate trusted and untrusted workloads according to a documented threat model.
• Patch host kernels, runtimes, orchestrators, and images.
• Monitor both resource behavior and security events.

Chapter Summary

Containers are kernel-mediated process environments rather than independent computers. Linux namespaces isolate views of processes, mounts, networks, users, and related resources, while cgroups regulate and account for resource consumption. Capabilities, seccomp, mandatory-access-control systems, and device restrictions reduce the operations available to containerized processes.

Layered images provide efficient distribution and reuse, but copy-on-write storage is not a substitute for persistent volumes and does not erase sensitive data from earlier layers. Container architecture commonly separates image management, high-level lifecycle management, and low-level process execution. OCI standards establish portable image, runtime, and distribution formats, while registries support artifact distribution and supply-chain governance.

Containers generally provide greater density and faster startup than virtual machines because they share the host kernel. Virtual machines provide an independent guest kernel and often a stronger isolation boundary. Sandboxed runtimes add further protection for hostile or semi-trusted workloads at the cost of complexity and possible compatibility limitations. Sound engineering decisions depend on threat model, workload behavior, operational requirements, and governance—not on performance claims alone.

Key Terms

Namespace: A Linux kernel mechanism that gives processes an isolated view of selected system resources.

PID namespace: A namespace that isolates process identifiers and process visibility.

Mount namespace: A namespace that provides an independent view of mounted filesystems.

Network namespace: A namespace that isolates interfaces, routes, ports, and network configuration.

User namespace: A namespace that maps container user and group identifiers to host identifiers.

Control group (cgroup): A kernel mechanism for grouping processes and controlling or accounting for resource usage.

Linux capability: A discrete unit of privilege separated from the traditional all-powerful root identity.

Seccomp: A Linux facility for restricting the system calls a process may invoke.

Mandatory access control: Policy-based access enforcement implemented through systems such as SELinux or AppArmor.

Copy-on-write: A storage technique in which shared content is copied only when a modification occurs.

Image layer: A filesystem change set that forms part of a container image.

Whiteout: A layer marker indicating that a file from a lower layer should be hidden.

Container runtime: Software that prepares and executes containers using operating-system isolation mechanisms.

OCI: The Open Container Initiative, which defines open standards for images, runtimes, and distribution.

Registry: A service that stores and distributes container images and related artifacts.

Image digest: A content-derived identifier that references an exact artifact.

Rootless container: A container operated without requiring host-level root privileges.

Virtual machine: A virtualized computer running a guest operating system and its own kernel.

Sandboxed runtime: A container execution environment that adds a stronger isolation layer, often through virtualization or specialized kernel techniques.

Software supply chain: The processes, artifacts, identities, and controls involved in building, distributing, and deploying software.

Review Questions

Short Answer Questions

1. What is the relationship between a container image and the host kernel?

2. How does a PID namespace differ from a cgroup?

3. What purpose do Linux capabilities serve?

4. Why should persistent application data normally be stored outside the writable container layer?

5. What are the principal areas covered by OCI specifications?

Long Answer Questions

1. Explain how namespaces, cgroups, capabilities, seccomp, and mandatory access controls work together to isolate a container.

2. Describe the lifecycle of an image from a registry to a running process, identifying the roles of high-level and low-level runtimes.

3. Compare containers and virtual machines in terms of kernel sharing, startup, resource density, portability, and security boundaries.

4. Explain layered filesystems, copy-on-write behavior, whiteouts, and the implications for image design and data persistence.

5. Discuss the governance controls required for a secure container image supply chain.

Analytical / Application Questions

1. A containerized service repeatedly exceeds its memory limit and is terminated, although the host has unused disk space. Analyze the likely role of cgroups and propose an investigation plan.

2. An organization must run externally submitted build jobs on shared infrastructure. Decide whether ordinary containers, virtual machines, or a sandboxed runtime is most appropriate, and justify the decision using threat model and operational trade-offs.

3. A developer deletes a credential file in a later Dockerfile instruction and assumes it is absent from the image history. Explain the layered-filesystem problem and recommend a safer build and credential-management approach.

22. Multiple Choice Questions with Answers

Question 1

Which mechanism primarily controls the processes and resource view visible inside a container?

A. Namespaces

B. Image registries

C. OCI manifests

D. Copy-on-write layers

Correct Answer: A. Namespaces

Question 2

A container consistently consumes excessive memory and affects other workloads. Which Linux facility is most directly used to impose and measure a memory boundary?

A. UTS namespace

B. Control group

C. Image manifest

D. Registry token

Correct Answer: B. Control group

Question 3

What is the main security concern with running a container in privileged mode?

A. It prevents image layers from being shared

B. It disables all network connectivity

C. It may grant broad access to host kernel functionality

D. It forces the container to use a virtual machine

Correct Answer: C. It may grant broad access to host kernel functionality

Question 4

Why does deleting a secret in a later image layer fail to remove it securely from the image history?

A. Registries cannot store deletion operations

B. Earlier layers may still contain the original file content

C. Namespaces automatically recreate deleted files

D. Cgroups preserve deleted files in memory

Correct Answer: B. Earlier layers may still contain the original file content

Question 5

Which statement best distinguishes a container from a virtual machine?

A. A container normally shares the host kernel, while a virtual machine runs a guest kernel

B. A container always has stronger isolation than a virtual machine

C. A virtual machine cannot have persistent storage

D. A container requires a complete guest operating system

Correct Answer: A. A container normally shares the host kernel, while a virtual machine runs a guest kernel

Question 6

What is the principal purpose of an image digest?

A. To provide a mutable human-readable deployment label

B. To identify exact artifact content

C. To allocate CPU shares

D. To create a network namespace

Correct Answer: B. To identify exact artifact content

Question 7

Which component most directly creates namespaces, mounts, capabilities, and the container process according to a runtime specification?

A. Low-level runtime

B. Image registry

C. Source-code repository

D. Container log driver

Correct Answer: A. Low-level runtime

Question 8

A workload is semi-trusted and runs on a shared node where escape consequences are serious. Which architectural choice most directly adds an isolation boundary beyond ordinary namespaces?

A. Increasing the image tag length

B. Using a sandboxed runtime

C. Adding more image layers

D. Removing all cgroup limits

Correct Answer: B. Using a sandboxed runtime

Question 9

Which practice best supports reproducible production deployments?

A. Using only the latest mutable tag

B. Pulling images by digest through a controlled registry

C. Storing credentials in deleted Dockerfile layers

D. Running every process as host root

Correct Answer: B. Pulling images by digest through a controlled registry

Question 10

What is the primary function of seccomp in container security?

A. It filters permitted system calls

B. It distributes images between registries

C. It creates persistent volumes

D. It assigns container hostnames

Correct Answer: A. It filters permitted system calls