Compute, Virtualization, Containers and Networking

Introduction

Cloud-based artificial intelligence (AI) systems depend on more than models and data. They require suitable compute resources, reliable execution environments, and networks that can move data efficiently between users, storage systems, training workers, and accelerators. Cloud computing provides these capabilities as configurable services rather than as permanently owned hardware.

This chapter examines the infrastructure foundations of AI workloads: virtual machines, hypervisors, containers, orchestration, CPU and accelerator selection, autoscaling, virtual networks, routing, load balancing, DNS, content delivery networks, and private connectivity. The emphasis is not merely on identifying technologies, but on understanding how infrastructure choices affect performance, cost, security, reliability, and maintainability.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain how virtualization and hypervisors support cloud computing.

2. Compare virtual machines, containers, and serverless execution environments.

3. Select CPU, GPU, and TPU resources for representative AI workloads.

4. Describe orchestration, scheduling, autoscaling, and resource isolation.

5. Design basic virtual networks using regions, availability zones, subnets, routes, and security controls.

6. Explain the roles of load balancers, DNS, content delivery networks, and private connectivity.

7. Evaluate infrastructure trade-offs in an AI deployment scenario.

1. Compute Abstractions for Cloud AI

A compute resource executes instructions. In AI systems, compute may be used for data preparation, feature engineering, model training, hyperparameter search, inference, monitoring, and application logic. These activities have different resource profiles.

Batch preprocessing may require many CPU cores and high-throughput storage. Deep-learning training often benefits from GPUs or other accelerators, high memory bandwidth, and fast interconnects. Online inference may require low latency, predictable availability, and efficient scaling. Consequently, there is no universally best compute instance.

Cloud providers expose compute through abstractions such as virtual machines (VMs), containers, managed container platforms, and serverless functions. Each abstraction trades control for operational simplicity.

Virtual Machines

A virtual machine is a software-defined computer with virtual CPUs, memory, storage, and network interfaces. It runs a guest operating system, which is separated from other VMs on the same physical host. A VM image commonly contains a bootable operating system and configuration metadata.

VMs provide strong isolation and substantial control. An organization can select an operating system, install specialized libraries, attach persistent disks, configure kernel parameters, and run long-lived services. They are suitable for GPU drivers, custom inference servers, legacy software, and workloads requiring operating-system-level control.

The main disadvantages are boot time, image maintenance, and resource overhead. Each VM includes a guest operating system, and patching many individually managed VMs can be operationally expensive.

Hypervisors

A hypervisor, also called a virtual machine monitor, allocates physical resources among virtual machines. A Type 1 hypervisor runs directly on hardware; a Type 2 hypervisor runs above a host operating system. Large cloud platforms primarily use Type 1 designs or hardware-assisted virtualization.

The hypervisor virtualizes processor execution, memory, storage, and networking. Modern processors provide hardware support for virtualization, reducing the cost of transitions between guest and host execution. Nevertheless, virtualization can introduce overhead, especially for input/output-intensive workloads. Technologies such as paravirtualized network and storage drivers reduce this overhead by allowing the guest to use interfaces designed for virtualized environments.

Important VM concepts include:

• vCPU: A virtual processor exposed to a VM. Its exact relationship to a physical core depends on the provider and instance type.
• Memory allocation: RAM assigned to the VM, often determining which applications can run without paging.
• Ephemeral storage: Storage whose persistence may not survive VM termination.
• Persistent volume: Provider-managed storage that remains independent of a VM lifecycle.
• Image: A reusable template containing an operating system and software configuration.
• Snapshot: A point-in-time representation of a disk or volume.

A VM does not automatically guarantee dedicated physical performance. Users must inspect provider specifications for CPU sharing, network bandwidth, storage throughput, and accelerator attachment.

2. Containers and Orchestration

A container packages an application, its libraries, and configuration while sharing the host operating system kernel. Unlike a VM, a container does not normally contain a complete guest kernel. Linux containers use mechanisms such as namespaces for isolation and control groups (cgroups) for resource accounting and limitation.

Containers generally start faster and use fewer resources than VMs. They provide reproducible packaging: the same image can be tested, deployed, and rolled back across environments. This is valuable for machine-learning services whose dependencies include frameworks, CUDA libraries, tokenizers, model servers, and system packages.

Containers are not identical to virtual machines. Kernel vulnerabilities, incorrect privileges, exposed sockets, or unsafe images can weaken isolation. Containers should therefore be treated as an application packaging and process-isolation mechanism, not as an absolute security boundary.

Container Images and Registries

A container image consists of immutable layers. A registry stores and distributes images. Good practice includes pinning image versions, scanning images for vulnerabilities, minimizing installed packages, avoiding secrets in image layers, and signing images where supported. Reproducibility improves when a build identifies exact dependency versions rather than using unbounded tags such as `latest`.

Orchestration

When many containers must be scheduled across multiple machines, manual management becomes impractical. An orchestrator places workloads, restarts failed containers, exposes services, manages configuration, and supports rolling updates. Kubernetes is a widely used example, although cloud providers also offer managed container services with different abstractions.

Common orchestration concepts include:

• Pod or task: The smallest deployable unit in a platform.
• Node: A machine capable of running workloads.
• Deployment: A desired state specifying replicas and an image version.
• Service: A stable logical endpoint for a changing set of workload instances.
• Scheduler: A component that selects suitable nodes according to resource requests, constraints, and affinity rules.
• Control plane: Components that maintain and reconcile desired state.

A workload should declare CPU, memory, and accelerator requests. If requests are omitted or inaccurate, scheduling and autoscaling decisions may be unreliable. GPU workloads also require compatible device plugins, drivers, runtime libraries, and often node pools reserved for accelerators.

VMs, Containers, and Serverless: A Comparison

VMs offer the greatest operating-system control and are useful for specialized or long-running workloads. Containers improve portability and deployment speed but require careful image and runtime security. Serverless functions remove server management and can scale automatically, but they usually impose limits on execution duration, memory, networking, and accelerator access. A managed machine-learning endpoint may provide a higher-level alternative, hiding much of the infrastructure while exposing model deployment and scaling controls.

3. CPU, GPU, and TPU Acceleration

Central Processing Units

CPUs are general-purpose processors with strong performance for branching logic, operating-system services, data loading, preprocessing, and many classical machine-learning algorithms. CPU performance depends on core count, clock behavior, cache hierarchy, instruction-set extensions, memory bandwidth, and workload parallelism.

Adding cores does not always produce proportional speedup. If a fraction of a program is serial, the potential benefit of parallel execution is limited. Amdahl’s law expresses this idealized constraint:

\[

S(N)=\frac{1}{(1-p)+p/N}

\]

where \(S(N)\) is speedup using \(N\) processors, \(p\) is the fraction of execution that can be parallelized, and \(1-p\) is the serial fraction. The model assumes equal processor performance and negligible parallelization overhead; real systems usually achieve less speedup.

For example, if 90% of a workload is parallelizable and 10 processors are used, the ideal speedup is:

\[

S(10)=1/(0.1+0.9/10)=1/0.19\approx5.26.

\]

Thus, ten processors do not yield ten times the performance because the serial portion remains.

Graphics Processing Units

GPUs contain many arithmetic units designed for high-throughput parallel operations. Matrix multiplication and convolution, central to deep learning, can map effectively to GPU hardware. GPU systems also include high-bandwidth memory and specialized tensor computation units.

GPU performance depends on more than theoretical arithmetic throughput. Batch size, memory access patterns, kernel efficiency, host-to-device transfer, precision, and inter-GPU communication are important. A model may be compute-bound when arithmetic is the limiting factor, or memory-bound when data movement is the limiting factor.

Mixed-precision computation uses lower numerical precision for selected operations while retaining higher precision where stability requires it. It can improve throughput and reduce memory use, but it must be validated because numerical error, overflow, or underflow may affect training.

Tensor Processing Units and Other Accelerators

TPUs and similar accelerators are specialized for tensor operations. They can provide high performance for supported frameworks and computational patterns, but may require compiler-compatible graphs, specific software stacks, and careful consideration of data movement. Field-programmable gate arrays and inference accelerators may be attractive for low-latency or energy-efficient deployments.

Accelerator selection should consider:

1. Supported software frameworks and operators.

2. Available memory and memory bandwidth.

3. Interconnect performance for distributed training.

4. Precision modes and numerical requirements.

5. Startup time and regional availability.

6. Cost per useful training step or inference request.

7. Operational complexity and portability.

4. Scheduling, Autoscaling, and Reliability

A scheduler assigns work to resources. It may consider requested CPU, memory, GPU count, node labels, data locality, fault domains, and workload priority. Scheduling is especially important for AI because accelerator nodes are expensive and often scarce.

Autoscaling changes resource quantity in response to demand. Horizontal scaling adds instances or replicas; vertical scaling changes the resources assigned to an instance; cluster or node autoscaling adds or removes worker machines. Horizontal scaling is commonly effective for stateless inference services. Training jobs are more difficult because changing the worker count may require coordinated distributed execution.

A simple capacity estimate for a stateless service is:

\[

R=\left\lceil\frac{\lambda}{c}\right\rceil,

\]

where \(R\) is the minimum number of replicas, \(\lambda\) is the incoming request rate, and \(c\) is the sustainable request rate per replica. This assumes homogeneous replicas, stable request cost, and no explicit reserve for failures or bursts. In practice, a safety margin is required and latency targets should be measured, not inferred solely from request rate.

Autoscaling signals may include CPU utilization, GPU utilization, queue length, requests per second, inference latency, or custom business metrics. CPU utilization alone is often a poor signal for GPU inference: a service can have low CPU use while its GPU queue is saturated.

Reliability requires health checks, multiple failure domains, restart policies, graceful termination, checkpointing, and observability. Training jobs should save checkpoints to durable storage so that a failed VM does not require restarting from the beginning.

5. Virtual Networking for AI Workloads

A virtual private cloud or equivalent virtual network provides an isolated logical network. It commonly contains subnets, route tables, gateways, network interfaces, and security policies.

Regions, Zones, and Subnets

A region is a geographic area containing cloud infrastructure. Availability zones are separated failure domains within a region. Deploying replicas across zones reduces the impact of a localized failure, although it may increase cost and cross-zone data-transfer charges.

A subnet is a range of IP addresses within a virtual network, usually associated with a zone or routing domain. Subnets are often categorized as public or private according to their route to the public internet. A public subnet may have a route through an internet gateway, but a resource is not necessarily publicly reachable unless its interface, firewall rules, and address configuration permit it.

CIDR notation describes an IP address block. In IPv4, a `/24` network contains 256 total addresses, though cloud platforms commonly reserve some addresses for network services. Subnet planning should allow for current resources, future growth, load balancers, managed services, and separate environments.

Routing and Connectivity

A route table maps destination prefixes to next hops such as a local network, internet gateway, NAT gateway, VPN endpoint, or private service endpoint. Longest-prefix matching generally selects the most specific applicable route.

Private subnets are commonly used for databases, training workers, and internal inference services. Such resources may access approved external services through a network address translation gateway, while unsolicited inbound connections remain blocked. Private endpoints can provide access to object storage or managed APIs without traversing the public internet.

Site-to-site VPNs encrypt traffic between an organization and a cloud network over shared infrastructure. Dedicated connections provide more predictable performance and may support regulatory or latency requirements, but they cost more and require physical or provider coordination. Encryption, routing, redundancy, and key management must all be considered.

Security Controls

Network security groups or equivalent stateful firewalls commonly filter traffic at the interface or instance level. Network access control lists may operate at subnet boundaries. Least privilege means allowing only the protocols, ports, source ranges, and destinations required by a service.

Network isolation should be combined with identity-based authorization. A private address does not prove that a caller is trustworthy. Mutual TLS, service identities, API authentication, encryption in transit, and audit logging are appropriate for sensitive AI services and data.

6. Load Balancing, DNS, and Content Delivery

A load balancer distributes requests among healthy service instances. Layer 4 load balancers operate mainly using transport information such as TCP ports. Layer 7 load balancers understand application protocols such as HTTP and can route based on hostnames, paths, headers, or cookies.

Health checks should test whether a service can actually serve traffic, not merely whether its process is running. Poor health checks can send requests to overloaded or partially initialized instances.

DNS maps domain names to records such as A, AAAA, CNAME, and TXT records. Recursive resolvers cache responses according to time to live (TTL). A lower TTL may allow faster changes but increases query traffic; a higher TTL reduces query load but delays propagation of changes. DNS-based routing can support geographic, weighted, latency-based, or failover policies, although DNS is not a substitute for application-level health and capacity management.

A content delivery network (CDN) caches content at geographically distributed edge locations. It is useful for model web interfaces, documentation, static assets, and sometimes downloadable model artifacts. A CDN can reduce origin load and user latency. Large model files require appropriate cache-control policies, integrity validation, access control, and consideration of cache invalidation. Dynamic inference requests usually still require a controlled origin service, although some edge platforms support specialized inference.

7. Applied Case: Illustrative Cloud Vision Service

Background

Consider an illustrative company, VisionFlow, which provides image classification to retail customers. Its model is trained weekly and serves online predictions through an API. Images may contain commercially sensitive information, and customers expect low latency during business hours.

Problem

The initial design uses one GPU VM with a public IP address. Training competes with inference for resources. A VM failure causes downtime, model files are copied manually, and customer traffic is sent directly to the application process. The company needs improved reliability, controlled network access, and predictable scaling.

Analysis

Training and inference have different resource requirements. Training needs a GPU cluster or scheduled GPU job, durable datasets, and checkpoint storage. Inference needs multiple replicas, a stable endpoint, health checks, and capacity that can change with demand. The public address on the VM increases exposure, while manual deployment creates configuration drift.

Approach and Solution

VisionFlow builds a versioned container image containing the inference server and model-loading logic. The image is stored in a private registry and scanned before release. A managed container cluster runs inference replicas across at least two availability zones. GPU nodes are separated from CPU nodes using scheduling labels and resource requests.

The public entry point is a Layer 7 load balancer. DNS maps the customer domain to the load balancer, and health checks remove unhealthy replicas. The workers are placed in private subnets. A controlled egress gateway permits access to approved services, while a private endpoint provides access to object storage. Customer traffic uses TLS, and service identities authorize access to model artifacts.

An autoscaler uses request rate and queue length, with a minimum number of replicas for availability and a maximum to control cost. Training runs as a separate scheduled job, writes checkpoints to durable storage, and publishes an immutable model version after validation. A deployment process can perform a gradual rollout and return to the previous model if error rate or latency increases.

Outcome

The design separates training from serving, reduces the impact of a single VM failure, improves auditability, and limits direct network exposure. It also introduces costs for multiple replicas, GPU capacity, load balancing, monitoring, and inter-zone traffic. The company therefore measures cost per prediction and training run rather than assuming that maximum hardware utilization is the only objective.

Learning Points

The case demonstrates that infrastructure architecture should follow workload behavior. Isolation, repeatability, and resilience are achieved through a combination of compute selection, containers, orchestration, network design, security controls, and operational measurement.

8. Challenges, Governance, and Best Practices

Cloud AI infrastructure creates several challenges. Accelerator availability may vary by region. Drivers and framework versions can be incompatible. Distributed training can be limited by network bandwidth or synchronization overhead. Autoscaling may be slow when GPU nodes take several minutes to start. Egress and cross-zone charges can materially change total cost.

Governance issues include protection of personal or confidential data, regional data-residency requirements, access logging, retention policies, software licensing, and incident response. A model endpoint should not expose training data through logs or debugging traces. Images and dependencies require vulnerability management, and access to production models should be auditable.

Recommended practices include:

• Define resource requests and limits explicitly.
• Use immutable, versioned images and model artifacts.
• Separate development, testing, and production networks and accounts.
• Place databases and workers in private subnets where appropriate.
• Use multiple zones for critical services.
• Monitor latency, queue depth, utilization, failures, cost, and data-transfer volume.
• Checkpoint long-running training jobs.
• Test autoscaling and failure recovery rather than assuming they work.
• Apply least privilege to identities and network rules.
• Benchmark end-to-end performance, including data loading and serialization.
• Select accelerators according to measured workload economics, not theoretical peak performance alone.

Contemporary Developments

Current cloud platforms increasingly offer confidential computing, specialized inference accelerators, high-speed accelerator interconnects, and managed model-serving systems. Infrastructure-as-code allows networks, clusters, policies, and services to be defined as reviewable configurations. Service meshes can provide traffic policy, telemetry, and mutual TLS, although they add operational complexity. Edge inference reduces latency and data transfer in some applications, but introduces device-management and model-distribution challenges.

The broader trend is toward heterogeneous infrastructure: CPU, GPU, TPU, and other accelerators are combined according to workload needs. AI engineers therefore need both model knowledge and systems knowledge. Efficient models are valuable, but their real-world performance depends on packaging, scheduling, memory movement, networking, and reliability engineering.

Chapter Summary

Cloud AI workloads rely on coordinated compute and networking abstractions. Virtual machines provide operating-system control and isolation through hypervisors, while containers package applications efficiently and orchestration platforms manage replicas, scheduling, updates, and recovery. CPUs suit general-purpose and preprocessing tasks; GPUs and TPUs accelerate parallel tensor operations, but their value depends on memory, software compatibility, communication, and measured workload behavior.

Autoscaling must use suitable signals and distinguish stateless inference from stateful or distributed training. Virtual networks organize resources through regions, zones, subnets, route tables, gateways, and security policies. Load balancers provide stable service endpoints, DNS provides name resolution and traffic policies, and CDNs reduce latency for cacheable content. Private connectivity, encryption, identity controls, and observability are essential for secure and governable AI systems.

The central design principle is workload alignment: infrastructure should be selected by analyzing performance, availability, security, portability, and total cost together.

Key Terms

Virtual machine: A software-defined computer running on shared physical infrastructure.

Hypervisor: Software that creates and manages virtual machines.

Container: An isolated application process and its dependencies sharing a host kernel.

Container image: An immutable package used to create containers.

Container registry: A service that stores and distributes container images.

Orchestration: Automated scheduling, deployment, scaling, and recovery of workloads.

Scheduler: A component that assigns workloads to suitable compute nodes.

CPU: A general-purpose processor designed for varied sequential and parallel tasks.

GPU: A massively parallel processor suited to many tensor and matrix operations.

TPU: A specialized tensor-processing accelerator.

Autoscaling: Automatically changing resource capacity in response to demand or policy.

Availability zone: A separated cloud failure domain within a region.

Subnet: A logical IP address range within a virtual network.

Route table: A set of rules mapping destinations to network next hops.

Load balancer: A system that distributes traffic among service instances.

DNS: The distributed naming system that maps domain names to network records.

CDN: A geographically distributed caching and delivery system.

Private endpoint: A private network interface or route to a managed service.

Checkpoint: A saved state from which a training job can resume.

Least privilege: Granting only the access required to perform an authorized task.

Review Questions

Short Answer Questions

1. What are the principal differences between a virtual machine and a container?

2. What role does a hypervisor play in cloud computing?

3. Why might GPU utilization be a better autoscaling signal than CPU utilization for inference?

4. What is the purpose of a private subnet?

5. How does a load balancer improve service availability?

Long Answer Questions

1. Compare VMs, containers, and serverless functions for deploying an AI inference service.

2. Explain the factors that should influence CPU, GPU, or TPU selection for model training.

3. Describe how an orchestrator schedules, updates, and recovers containerized workloads.

4. Explain how regions, availability zones, subnets, route tables, and gateways cooperate in a cloud network.

5. Discuss the security and governance practices required for a production AI platform.

Analytical / Application Questions

1. A workload is 90% parallelizable. Using Amdahl’s law, calculate its ideal speedup on 10 processors and explain why the result is less than 10.

2. Design a network and deployment architecture for an online GPU-based inference service that must not expose its worker nodes directly to the internet.

3. An inference service experiences low CPU utilization but rising request latency and GPU queue length. Propose an autoscaling policy and justify the signals you would use.

Multiple Choice Questions with Answers

Question 1

Which component allocates physical resources among virtual machines?

A. Hypervisor

B. DNS resolver

C. Content delivery network

D. Load balancer

Correct Answer: A. Hypervisor

Question 2

Which workload is generally most suitable for a GPU?

A. A small sequential shell script

B. Large parallel matrix operations in deep learning

C. A DNS zone transfer

D. User identity management

Correct Answer: B. Large parallel matrix operations in deep learning

Question 3

In Amdahl’s law, the variable p represents what?

A. The number of processors

B. The serial fraction of a workload

C. The parallelizable fraction of a workload

D. The network packet size

Correct Answer: C. The parallelizable fraction of a workload

Question 4

What is a primary advantage of containers over virtual machines?

A. They always provide stronger isolation than VMs

B. They usually start faster and use fewer resources

C. They eliminate the need for security controls

D. They include a separate guest kernel by default

Correct Answer: B. They usually start faster and use fewer resources

Question 5

Which component distributes requests among healthy service instances?

A. Load balancer

B. Hypervisor

C. Container image

D. Checkpoint

Correct Answer: A. Load balancer

Question 6

What is the principal function of DNS?

A. Encrypting GPU memory

B. Mapping domain names to network records

C. Scheduling containers on nodes

D. Creating VM snapshots

Correct Answer: B. Mapping domain names to network records

Question 7

Why are training checkpoints valuable?

A. They allow failed training jobs to resume from saved state

B. They replace network security groups

C. They guarantee linear speedup

D. They convert CPUs into GPUs

Correct Answer: A. They allow failed training jobs to resume from saved state

Question 8

Which design best limits direct internet exposure of inference workers?

A. Place every worker in a public subnet with a public IP

B. Place workers in private subnets behind a load balancer

C. Disable health checks and use direct addressing

D. Store credentials in the container image

Correct Answer: B. Place workers in private subnets behind a load balancer

Question 9

A service receives 100 requests per second and one replica sustainably handles 25 requests per second. Ignoring safety margin, what is the minimum replica count using R = ceil(λ/c)?

A. 2

B. 3

C. 4

D. 5

Correct Answer: C. 4

Question 10

Which practice improves container reproducibility?

A. Using unbounded latest tags for all dependencies

B. Pinning image and dependency versions

C. Embedding production secrets in image layers

D. Disabling image scanning

Correct Answer: B. Pinning image and dependency versions