2. Processor, Memory and Interconnect Architectures

Introduction: Why Architecture Determines Performance

High-performance computing (HPC) is often associated with large numbers of processors, but processor count alone does not determine performance. A computing system may contain many cores and still perform poorly if its memory system cannot supply data quickly, if communication between nodes is slow, or if software exposes insufficient parallelism. Effective performance therefore emerges from the interaction of computation, memory access, data movement, and communication.

This chapter examines the architectural foundations that govern those interactions. It begins with the CPU and the mechanisms used to exploit instruction-level and data-level parallelism. It then considers multicore processors, cache hierarchies, memory organization, NUMA, storage, and interconnects. The central theme is that an HPC application should be understood as a flow of instructions and data through a hierarchy of resources, each with different capacities, latencies, and bandwidths.

For data science professionals, these issues are particularly important. Matrix operations, simulation, machine learning, graph analytics, and large-scale data processing all depend on how efficiently processors access and exchange data. Architectural understanding enables better algorithm selection, performance diagnosis, system procurement, and software optimization.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the organization and performance characteristics of modern CPUs and multicore systems.

2. Distinguish instruction-level parallelism, thread-level parallelism, and SIMD/vector parallelism.

3. Analyze cache hierarchies, memory locality, bandwidth, latency, and NUMA behavior.

4. Compare storage technologies and identify their influence on data-intensive workloads.

5. Explain network topology, interconnect latency, bandwidth, and communication overhead.

6. Apply architectural reasoning to performance diagnosis and system design.

7. Evaluate current developments such as accelerators, chiplets, high-bandwidth memory, and computational storage.

The Processor as a Parallel Execution Engine

A processor executes instructions by repeatedly fetching, decoding, and executing them. Early processors generally completed one instruction before beginning the next. Modern CPUs, by contrast, contain complex mechanisms that allow several instructions to be in different stages of execution simultaneously.

Instruction-level parallelism

Instruction-level parallelism (ILP) is the ability to execute multiple independent instructions at the same time within a single instruction stream. A processor may use pipelining, superscalar issue, out-of-order execution, register renaming, and speculative execution to expose ILP.

In a pipeline, instruction processing is divided into stages such as fetch, decode, execution, memory access, and write-back. Once the pipeline is full, several instructions may be processed during each clock cycle. A superscalar processor can issue multiple instructions per cycle if suitable execution units and independent instructions are available.

Out-of-order execution allows the processor to execute a later instruction before an earlier instruction has completed, provided that dependencies are respected. For example, an arithmetic instruction that does not depend on a delayed memory load may proceed while the load is being serviced. Register renaming reduces false dependencies caused by reuse of architectural registers.

ILP is limited by data dependencies, branches, cache misses, and the structure of the program. If instruction B requires the result of instruction A, the two cannot be fully parallelized. Similarly, unpredictable branches can force the processor to discard speculative work. Consequently, a high clock frequency does not guarantee high application performance.

Multicore and multithreaded processors

A multicore processor integrates multiple processing cores on one package. Each core may have private registers, execution units, and lower-level caches, while higher-level caches and memory controllers may be shared. Multicore designs exploit thread-level parallelism: different software threads execute concurrently on different cores.

Simultaneous multithreading (SMT) allows one physical core to maintain the architectural state of multiple hardware threads. When one thread is stalled, perhaps waiting for memory, another may use otherwise idle execution resources. SMT can improve throughput, although competing threads share core resources and may reduce the performance of an individual thread.

Parallel speedup is commonly discussed using Amdahl’s law. If a fraction p of a program can be parallelized and the parallel portion executes on N processors, the ideal speedup is:

S(N) = 1 / [(1 − p) + p/N]

The non-parallel fraction places an upper bound on speedup. In practice, synchronization, scheduling, load imbalance, memory contention, and communication make actual speedup lower than this ideal. A workload with 95 percent parallelizable computation has a theoretical maximum speedup of 20, regardless of how many processors are added.

SIMD and vector processing

Single Instruction, Multiple Data (SIMD) executes one instruction over several data elements. A vector register may contain multiple integers or floating-point values, allowing an addition, comparison, or multiplication to be performed on an entire group of values.

SIMD is highly effective for dense numerical operations, image processing, signal processing, and machine-learning kernels. It requires suitable data parallelism and generally benefits from contiguous, aligned, regularly accessed data. Conditional branches, irregular memory access, and dependencies between neighboring elements can reduce its effectiveness.

Vector length and data type influence the amount of work completed per instruction. For example, a register capable of holding eight 32-bit values can process eight single-precision elements in one vector operation, subject to implementation and execution-unit constraints. Compilers may automatically vectorize loops, but programmers often need to provide clear loop bounds, avoid aliasing, and organize data appropriately.

SIMD should not be confused with multicore parallelism. SIMD operates within a core, while multicore execution distributes work across cores. A well-optimized application may use both: several threads, each executing vectorized instructions.

Memory Hierarchy and Locality

Processor speed has increased more rapidly than the latency of main memory. Modern systems therefore use a hierarchy of storage locations. At the top are registers, followed by one or more cache levels, main memory, and secondary or external storage. Upper levels are faster and smaller; lower levels are larger and slower.

A cache stores recently or frequently accessed copies of data. When the processor finds requested data in a cache, a cache hit occurs. If the data is absent, a cache miss requires retrieval from a lower level. Cache performance is commonly described using hit rate, miss rate, miss penalty, capacity, associativity, and line size.

Caches exploit two forms of locality. Temporal locality means that recently accessed data is likely to be used again. Spatial locality means that data near a recently accessed location is likely to be used soon. Cache lines transfer a block of neighboring bytes rather than a single value, making spatial locality essential.

The usual hierarchy includes:

• Registers: smallest and fastest storage, directly used by instructions.
• L1 cache: very small and fast, often divided into instruction and data caches.
• L2 cache: larger and somewhat slower, commonly private to a core.
• L3 or last-level cache: larger, often shared among several cores.
• Main memory: substantially larger but with higher latency.
• Storage: persistent and much slower than semiconductor memory.

A useful approximation for average memory access time is:

AMAT = hit time + miss rate × miss penalty

For multiple cache levels, the penalty includes the probability and cost of misses at each subsequent level. A small reduction in miss rate can therefore produce a significant improvement when the lower-level penalty is large.

Locality in data science algorithms

Consider matrix multiplication. If matrices are traversed in an order that repeatedly reuses blocks already present in cache, the processor performs many arithmetic operations per memory transfer. This is called blocking or tiling. By contrast, a traversal that repeatedly jumps through memory may generate cache misses even though it performs the same mathematical computation.

The distinction between arithmetic intensity and memory intensity is important. Arithmetic intensity is the number of computational operations performed per byte transferred from memory. High-arithmetic-intensity workloads may be limited by processor throughput, whereas low-arithmetic-intensity workloads are often memory-bandwidth bound. The roofline model represents attainable performance as constrained by both peak computational throughput and memory bandwidth:

Performance ≤ min(peak compute performance, arithmetic intensity × memory bandwidth)

The model helps determine whether optimization should focus on vectorization and instruction scheduling or on reducing data movement and improving locality.

Main Memory, Bandwidth, and NUMA

Main memory is organized through memory channels connected to memory controllers. Multiple channels allow several transfers to occur concurrently, increasing bandwidth. However, bandwidth is not the same as latency. Bandwidth measures the volume of data transferred per unit time; latency measures the delay before a transfer begins or completes. A workload may have adequate bandwidth but still perform poorly if it issues many small, latency-sensitive requests.

Modern multiprocessor systems frequently use Non-Uniform Memory Access (NUMA). In a NUMA system, each processor or socket has local memory, but it can also access memory attached to another socket. Local access generally has lower latency and may provide greater effective bandwidth than remote access. The operating system and runtime environment must therefore consider both thread placement and data placement.

A NUMA-aware strategy places a thread near the memory it uses. First-touch allocation is a common policy: a page is physically allocated in the NUMA node of the processor that first writes to it. If initialization is performed by only one thread, all pages may be placed on one node, creating remote-access traffic for other threads. Parallel initialization can distribute pages more appropriately.

NUMA problems often appear as unexplained variability, poor scaling beyond one socket, or high remote-memory traffic. Remedies include processor affinity, NUMA-aware allocators, parallel initialization, data partitioning, and reducing unnecessary sharing.

False sharing is another performance problem. It occurs when independent variables used by different threads occupy the same cache line. Although the threads do not logically share the variables, writes cause the cache line to move repeatedly between cores. Padding or reorganizing data can prevent this coherence traffic.

Storage Architecture for Data-Intensive Computing

Storage influences how quickly data can be loaded, checkpointed, and exchanged with compute nodes. Hard disk drives provide high capacity at relatively low cost but have mechanical seek and rotation delays. Solid-state drives eliminate mechanical movement and provide much lower access latency. NVMe devices communicate through PCIe and offer high parallelism and queue depth compared with older storage interfaces.

A storage system must be evaluated using more than sequential bandwidth. Important characteristics include random I/O performance, input-output operations per second (IOPS), latency, queue depth, durability, consistency, and capacity. Large sequential reads may achieve high bandwidth, while small random requests may be limited by latency and IOPS.

HPC environments often use parallel file systems that distribute files across multiple storage servers or devices. This enables many compute nodes to access data concurrently. Local scratch storage may be used for temporary files, intermediate results, and checkpointing, while shared storage supports persistent datasets.

Data layout is as important as storage hardware. Many small files can create metadata bottlenecks. Contiguous, partitioned, or columnar formats may improve throughput depending on the workload. Compression reduces storage traffic but consumes computational resources; whether it helps depends on the relative cost of computation and I/O.

Interconnects: Moving Data Between Cores and Nodes

In a multicore or clustered system, data moves through several levels of interconnect. Within a processor, cores communicate through buses, rings, meshes, or other on-chip networks. Between sockets, coherent interconnects maintain a consistent view of memory. Between nodes, high-speed networks carry messages, synchronization, and storage traffic.

Two fundamental network measures are latency and bandwidth. Latency is the time required to initiate and complete a communication, while bandwidth is the rate at which data can be transferred once communication is underway. A small message is usually dominated by latency; a large message is more strongly influenced by bandwidth.

Communication time can be approximated by:

Tcommunication = α + nβ

where α is startup latency, n is the message size, and β is the transfer time per byte. For a sequence of messages, additional terms may be needed to represent contention, protocol overhead, and synchronization.

Network topology

Topology determines how nodes are connected and how many links a message must traverse. Common topologies include:

• Bus: simple but limited by shared bandwidth and contention.
• Ring: each node has two neighbors; communication may require several hops.
• Mesh: nodes connect in a grid, offering multiple routes and regular scaling.
• Torus: mesh edge nodes are connected, reducing boundary effects and path lengths.
• Fat-tree: higher-level links provide greater aggregate bandwidth, often used in clusters.
• Hypercube: nodes are connected according to binary dimensions, providing short paths for certain system sizes.

No topology is universally best. A topology should match communication patterns, scale, fault-tolerance requirements, routing capabilities, and cost. A tightly coupled simulation with frequent nearest-neighbor exchange may benefit from a mesh or torus. A workload with arbitrary collective communication may require a high-bisection-bandwidth design such as a well-engineered fat-tree.

Bisection bandwidth describes the bandwidth available when a network is divided into two approximately equal parts. It is a useful indicator of whether many simultaneous communications can occur without severe congestion.

Communication models

Distributed-memory applications commonly use message passing. The Message Passing Interface (MPI) provides point-to-point and collective operations such as broadcast, reduce, gather, and all-to-all. The choice of operation affects both algorithmic behavior and network load.

Collective operations can be optimized using tree, ring, recursive doubling, or topology-aware algorithms. For example, a reduction may combine partial results in a tree, reducing the number of sequential stages. However, the best method depends on message size, node count, network topology, and implementation.

Overlap is a central optimization principle. Computation and communication can proceed concurrently if the application uses nonblocking communication and has independent work available. Overlap is not automatic: insufficient buffering, synchronization, or resource contention may prevent meaningful concurrency.

A Performance Reasoning Framework

Architectural analysis should begin with measurement rather than assumption. Useful metrics include processor utilization, instructions per cycle (IPC), cache-miss rates, memory bandwidth, branch-misprediction rate, NUMA traffic, I/O throughput, network latency, and application scaling.

A practical diagnostic sequence is:

1. Establish a baseline using representative input sizes.

2. Determine whether the workload is compute-, memory-, I/O-, or communication-bound.

3. Inspect locality, vectorization, thread placement, and synchronization.

4. Change one architectural factor at a time.

5. Re-measure performance and energy consumption.

Roofline analysis, hardware performance counters, tracing tools, profilers, and microbenchmarks can support this process. A microbenchmark that measures memory bandwidth, for example, does not predict application performance unless the application has a similar access pattern. Measurements must therefore be interpreted in context.

Applied Case: Accelerating a Distributed Matrix Analytics Pipeline

Background

This illustrative case concerns a university research group processing large covariance matrices for a machine-learning pipeline. The computation uses several nodes, each with two CPU sockets and multiple cores. The original implementation uses distributed matrix multiplication followed by a global reduction.

Situation and problem

Adding nodes initially improves runtime, but performance levels off after one socket per node. Profiling shows moderate CPU utilization, high last-level cache misses, and increasing communication time during the reduction. The team assumes that more cores are required, but the evidence suggests a combined memory and interconnect problem.

Analysis

The matrix is stored in a layout that causes non-contiguous access in one multiplication loop. Consequently, cache lines are fetched but only a small portion of each line is used. The loop is also difficult for the compiler to vectorize. In addition, the matrix is initialized by a single thread, placing most pages on one NUMA node. Threads running on the second socket therefore perform remote memory accesses.

The reduction exchanges many medium-sized messages. The selected collective algorithm creates synchronization phases, so computation frequently waits for communication. The cluster network has sufficient peak bandwidth, but the application’s message pattern creates contention and exposes startup latency.

Approach and solution

The team applies four changes. First, it tiles the matrix multiplication so that blocks fit into appropriate cache levels. Second, it changes the data layout and loop order to improve contiguous access and SIMD vectorization. Third, it initializes matrix blocks in parallel and binds threads to cores near their data. Finally, it aggregates partial results locally before using a topology-aware reduction, reducing the number of messages and synchronization points.

The team validates each change with hardware counters and scaling experiments. It measures cache-miss rates, remote-memory traffic, vector instruction usage, and time spent in collective operations rather than relying only on total runtime.

Outcome

The revised implementation obtains better single-node performance and maintains useful scaling across additional nodes. The principal improvement does not come from increasing processor frequency. It comes from reducing data movement, improving locality, using SIMD, respecting NUMA placement, and matching communication patterns to the interconnect.

Learning points

The case illustrates four general principles. First, processor performance depends on data access. Second, a multisocket machine should be treated as a NUMA system rather than a uniform pool of cores. Third, peak network bandwidth is not sufficient when communication involves many synchronization-sensitive messages. Finally, optimization is most effective when guided by measurements that connect software behavior to hardware events.

Challenges, Risks, and Professional Considerations

Architectural optimization has several limitations. Hardware counters may be vendor-specific, multiplexed, or difficult to interpret. A change that improves one workload may harm another because of different data sizes or access patterns. Benchmark results can also be misleading when inputs fit in cache, when data is already buffered, or when the benchmark excludes I/O and communication costs.

Oversubscription is a common mistake: creating more software threads than available hardware resources can increase context switching and contention. Other mistakes include ignoring thread affinity, assuming all memory accesses have equal cost, using synchronization inside tight loops, and optimizing computation while overlooking data conversion or file-format overhead.

Energy and sustainability should be included in architectural decisions. A faster configuration may consume substantially more power, while reducing data movement can improve both runtime and energy efficiency. Procurement decisions should consider performance per watt, cooling, memory capacity, network capability, software compatibility, and total cost of ownership rather than processor peak performance alone.

Governance and professional responsibility also matter. Performance experiments should document hardware configuration, compiler version, libraries, input data, thread placement, and measurement methodology. Reproducible reporting prevents unsupported performance claims. In shared clusters, users must respect allocation policies and avoid experiments that create excessive network or storage load.

Current Developments and Best Practices

Recent systems increasingly combine general-purpose CPUs with GPUs and other accelerators. These devices provide high parallel throughput but require explicit data movement or unified-memory mechanisms. The performance benefit depends on whether the workload exposes sufficient parallelism and whether transfers between host and accelerator can be hidden or minimized.

Chiplet-based processors divide a package into interconnected functional dies. High-bandwidth memory places memory closer to computation and can provide very high data-transfer rates, although capacity and cost may be constraints. Computational storage and near-data processing attempt to perform filtering, compression, or analytics closer to where data resides, reducing movement through the host system.

Best practice is to select architecture according to workload behavior. For compute-intensive vectorizable algorithms, prioritize execution throughput and SIMD capability. For memory-bound workloads, prioritize memory bandwidth, capacity, and locality. For distributed applications, examine latency, bisection bandwidth, topology, and collective-operation performance. For data pipelines, evaluate storage concurrency, metadata handling, and the cost of serialization.

A sound optimization strategy usually follows these principles:

• Improve algorithms and reduce unnecessary data movement before tuning instructions.
• Use blocking, appropriate data layout, and contiguous access to exploit locality.
• Combine thread-level parallelism with SIMD where the workload permits.
• Bind threads and allocate data with NUMA locality in mind.
• Measure communication time and message patterns, not only network peak bandwidth.
• Validate improvements with realistic workloads and repeatable experiments.
• Consider performance, energy, reliability, maintainability, and cost together.

Chapter Summary

HPC performance is determined by the coordinated behavior of processors, memory, storage, and interconnects. Modern CPUs exploit instruction-level parallelism through pipelines, superscalar execution, out-of-order scheduling, and speculation. Multicore and SMT systems provide thread-level parallelism, while SIMD and vector instructions process multiple data elements per instruction.

The memory hierarchy reduces the gap between processor speed and main-memory latency. Cache performance depends on temporal and spatial locality, and algorithmic techniques such as blocking can substantially reduce data movement. NUMA systems require attention to thread and data placement because remote memory access is more costly than local access. Storage must be evaluated using latency, bandwidth, IOPS, concurrency, and data layout. In distributed systems, interconnect latency, bandwidth, topology, bisection capacity, and communication algorithms determine scalability.

Architectural analysis should classify workloads as compute-, memory-, I/O-, or communication-bound and should use profiling and hardware measurements to guide decisions. Emerging systems add accelerators, chiplets, high-bandwidth memory, and computational storage, but these technologies introduce new programming and data-movement considerations. The most transferable principle is that performance comes from reducing unnecessary movement of instructions and data while matching software behavior to the architecture.

Key Terms

Instruction-level parallelism: Concurrent execution of independent instructions within an instruction stream.

Thread-level parallelism: Concurrent execution of multiple software threads.

SIMD: A processing model in which one instruction operates on multiple data elements.

Superscalar execution: Issuing multiple instructions in a processor cycle using several execution units.

Out-of-order execution: Executing ready instructions before earlier stalled instructions while preserving correct results.

SMT: Simultaneous multithreading that allows multiple hardware threads to share one core.

Cache line: The fixed-size block of memory transferred between a cache and a lower memory level.

Cache hit: A requested item found in the searched cache level.

Cache miss: A requested item absent from the searched cache level.

Temporal locality: The tendency to reuse recently accessed data.

Spatial locality: The tendency to access addresses near recently accessed data.

Arithmetic intensity: Computational operations performed per byte transferred.

NUMA: A memory organization in which access time depends on the memory’s location relative to a processor.

False sharing: Coherence traffic caused when threads modify different variables in the same cache line.

Latency: The delay before a requested operation or transfer completes.

Bandwidth: The amount of data transferred per unit of time.

Bisection bandwidth: Aggregate network bandwidth available across a division of a network into two parts.

Roofline model: A performance model bounded by peak computation and memory bandwidth.

Thread affinity: Binding software threads to selected processor cores.

Parallel file system: A storage system that distributes data and access across multiple devices or servers.

Review Questions

Short Answer Questions

1. What is instruction-level parallelism, and what factors limit it?

2. Distinguish cache latency from cache bandwidth.

3. Why can false sharing reduce multicore performance?

4. What is the difference between local and remote memory access in a NUMA system?

5. Define network latency and explain when it is especially important.

Long Answer Questions

1. Explain how pipelining, superscalar execution, out-of-order execution, and speculative execution work together in a modern CPU.

2. Discuss the cache hierarchy and explain how temporal and spatial locality influence application performance.

3. Explain the causes of NUMA-related performance degradation and describe methods for reducing it.

4. Compare common interconnect topologies and discuss how topology affects communication scalability.

5. Explain the roofline model and show how it can guide optimization decisions for a data science workload.

Analytical / Application Questions

1. An application uses 100 cores but achieves only 15 percent CPU utilization. Its cache-miss rate and remote-memory traffic are high. What architectural causes might explain the result, and what measurements or changes would you recommend?

2. A distributed program exchanges many small messages and shows poor scaling despite a high-bandwidth network. Use the communication model to explain the problem and propose an improved strategy.

3. A matrix-processing program has adequate processor capacity but spends most of its time waiting for memory. Explain how data layout, tiling, SIMD, NUMA placement, and arithmetic intensity could be analyzed to improve performance.

22. Multiple Choice Questions with Answers

Question 1

Which technique allows a processor to execute a later independent instruction while an earlier instruction is waiting for memory?

A. Out-of-order execution

B. Disk striping

C. NUMA allocation

D. Message aggregation

Correct Answer: A. Out-of-order execution

Question 2

A workload repeatedly accesses neighboring elements in an array. Which cache property does this behavior primarily exploit?

A. Temporal locality

B. Spatial locality

C. Branch speculation

D. Remote memory placement

Correct Answer: B. Spatial locality

Question 3

Which change is most likely to improve performance for a matrix multiplication that has poor cache reuse?

A. Increasing the number of small messages

B. Using blocking or tiling

C. Disabling vector instructions

D. Allocating all pages on one NUMA node

Correct Answer: B. Using blocking or tiling

Question 4

In the communication model T = α + nβ, what does α represent?

A. Arithmetic intensity

B. Cache capacity

C. Communication startup latency

D. Processor clock frequency

Correct Answer: C. Communication startup latency

Question 5

A thread accesses memory attached to another processor socket in a NUMA system. Compared with local access, this access will generally have:

A. Lower latency and higher locality

B. Higher latency and possible lower effective bandwidth

C. No performance difference

D. No need for cache coherence

Correct Answer: B. Higher latency and possible lower effective bandwidth

Question 6

Which situation is most suitable for SIMD processing?

A. A loop applying the same arithmetic operation to a large regular array

B. A workload dominated by unpredictable branches and irregular pointers

C. A file-system metadata operation

D. A synchronization barrier with no data computation

Correct Answer: A. A loop applying the same arithmetic operation to a large regular array

Question 7

A program performs few arithmetic operations for every byte loaded from memory. According to the roofline model, it is most likely:

A. Memory-bandwidth bound

B. Branch-prediction bound only

C. Storage-capacity bound by definition

D. Always limited by clock frequency

Correct Answer: A. Memory-bandwidth bound

Question 8

Which practice best reduces false sharing?

A. Placing frequently modified independent variables in separate cache lines

B. Increasing the number of threads sharing a cache line

C. Forcing all threads onto one core

D. Replacing local memory with remote memory

Correct Answer: A. Placing frequently modified independent variables in separate cache lines

Question 9

A cluster application sends many small messages and spends substantial time synchronizing. Which improvement is most directly relevant?

A. Reducing message count through aggregation or improved collective algorithms

B. Increasing matrix element size without changing communication

C. Disabling thread affinity

D. Reducing cache associativity

Correct Answer: A. Reducing message count through aggregation or improved collective algorithms

Question 10

Which metric is most useful for assessing whether a network can sustain simultaneous traffic between two large groups of nodes?

A. Cache hit rate

B. Bisection bandwidth

C. SIMD width

D. Disk seek time

Correct Answer: B. Bisection bandwidth