Multicore, Multiprocessor, and Heterogeneous Architectures

Introduction and Context

Modern communication and signal-processing equipment rarely relies on one general-purpose processor. A 5G base station, software-defined radio, radar, camera, or industrial controller must perform many operations concurrently: sampling, filtering, transforms, coding, packet processing, control, and machine-learning inference. Performance therefore depends not only on clock frequency but also on how effectively a workload is divided among processing elements, supplied with data, synchronized, and connected.

This chapter examines parallel processor organizations from the programmer’s and architect’s viewpoints. It distinguishes SIMD, vector, MIMD, multicore, GPU, DSP, and heterogeneous systems, then studies cache coherence, memory consistency, synchronization, interconnects, networks-on-chip (NoCs), scalability, and workload mapping.

Learning Objectives

After studying this chapter, a student should be able to:

1. Distinguish instruction-stream and data-stream parallelism.
2. Compare multicore CPUs, GPUs, DSPs, vector processors, and heterogeneous systems.
3. Explain cache coherence and memory-consistency models.
4. Select appropriate synchronization mechanisms and identify their costs.
5. Analyze interconnect and NoC latency, bandwidth, and scalability.
6. Map communication and signal-processing workloads onto suitable processing elements.
7. Evaluate performance using speedup, efficiency, workload balance, and communication overhead.

1. Forms of Parallelism and Processor Classification

Flynn’s taxonomy classifies architectures by the number of instruction streams and data streams:

SIMD executes one instruction over a vector of lanes. For example, adding two arrays of 16-bit samples can process eight values per instruction in an eight-lane unit. SIMD is highly effective when operations are regular, data-parallel, and free from frequent branches.

A vector processor presents vector registers and vector instructions whose operations apply to a sequence of elements. Modern CPU SIMD extensions are short-vector implementations; classic vector processors often support long vectors and sophisticated vector load/store mechanisms. The distinction is architectural rather than absolute.

MIMD processors have separate instruction streams. Independent cores may run an operating system, a control loop, a filter, and a communications protocol simultaneously. MIMD offers flexibility but requires coordination, scheduling, and data sharing.

Data, task, and pipeline parallelism

Data parallelism applies the same operation to many samples, pixels, or subcarriers. Task parallelism assigns different functions to different cores. Pipeline parallelism divides a stream-processing chain into stages. A radio receiver might pipeline synchronization, FFT, channel estimation, equalization, decoding, and packet delivery.

These forms can be combined. Each pipeline stage may use SIMD internally, while different stages execute on separate MIMD cores.

2. Multicore and Multiprocessor Organization

A multicore processor integrates multiple computational cores on one chip. A multiprocessor system may contain several chips or sockets. Shared-memory multiprocessors expose a common address space, although access time may be non-uniform in NUMA (non-uniform memory access) systems.

Important design dimensions include:

• Core specialization: identical general-purpose cores or asymmetric cores.
• Memory organization: shared cache, private cache, distributed cache, or local memories.
• Interconnect: bus, crossbar, ring, mesh, torus, or packet-switched NoC.
• Coherence scope: whether caches maintain a consistent copy of shared data.
• Programming model: threads, message passing, task graphs, accelerators, or streaming.

A shared bus is simple but becomes a bottleneck as core count increases. A crossbar can provide several simultaneous paths but requires substantial area and wiring. Rings reduce wiring relative to crossbars, while meshes scale more naturally in physical layout; however, mesh traffic may traverse multiple routers.

Amdahl’s law and practical speedup

If fraction p of an application is perfectly parallelizable and N processors execute that part with no overhead, the ideal speedup is

Here, S(N) is speedup relative to one processor, p is the parallel fraction, and N is the number of processors. The assumptions are fixed workload, perfect load balance, no communication cost, and no serial overhead. Real systems have additional time for communication, synchronization, cache misses, and scheduling, so measured speedup is lower.

For example, if p = 0.90 and N = 8,

The remaining 10% limits speedup even before hardware overhead is considered. Parallel efficiency is

For this example, efficiency is approximately 4.71/8 = 0.589, or 58.9%.

Gustafson’s law is useful for scaled workloads in which a larger machine solves a proportionally larger problem. It should not be confused with Amdahl’s fixed-workload assumption.

3. CPUs, GPUs, DSPs, and Heterogeneous Processing

General-purpose multicore CPUs

CPUs provide sophisticated control flow, large caches, virtual memory, branch prediction, and strong software compatibility. They are suitable for irregular algorithms, operating-system functions, protocol stacks, and control-intensive code. Their limited number of powerful cores and relatively complex control hardware may make them less energy-efficient for large regular vector operations.

Graphics processing units

GPUs contain many arithmetic lanes organized into groups that execute a common instruction, often called warps or wavefronts. A GPU is effective for high-throughput, regular workloads with abundant parallelism and predictable memory access. Branch divergence causes lanes in one group to follow different paths serially or under masks. Host-device transfer, kernel-launch overhead, and limited support for fine-grained synchronization can reduce benefits for small tasks.

A GPU is not simply a faster CPU: its performance depends on occupancy, memory coalescing, arithmetic intensity, and the ability to expose many independent threads.

Digital signal processors

DSPs are designed for predictable, repetitive numerical operations. Typical features include multiply-accumulate units, circular addressing, bit-reversed addressing for transforms, saturating arithmetic, deterministic interrupt response, and specialized memory arrangements. DSPs are particularly useful for finite impulse response (FIR) filters, infinite impulse response (IIR) filters, FFTs, modulation, beamforming, and audio processing.

For an FIR filter with coefficients h[k] and input x[n],

where M is the number of taps. Each output requires M multiplications and approximately M−1 additions, although implementation can reuse data and exploit SIMD or multiply-accumulate instructions. Fixed-point DSP implementations must specify word length, scaling, rounding, and saturation; otherwise overflow can corrupt the signal.

Heterogeneous systems

A heterogeneous system combines unlike processors, such as CPU cores, GPU compute units, DSPs, FPGA fabric, and AI accelerators. The CPU may coordinate tasks and handle irregular control; a DSP may execute deterministic filters; a GPU may process large image or matrix batches; and an FPGA may provide low-latency streaming logic.

The benefit is matching each workload to the most suitable engine. The cost includes programming complexity, data movement, synchronization across address spaces, compiler limitations, and verification effort. A theoretically fast accelerator may reduce total performance if transferring data to it costs more than the computation saved.

4. Cache Coherence and Memory Consistency

Cache coherence

Cache coherence concerns multiple cached copies of the same memory location. A coherent system ensures that writes eventually become visible according to a defined protocol and that processors do not indefinitely read stale values. Coherence is commonly described using three properties:

• Write propagation: a write becomes observable by other relevant caches.
• Write serialization: writes to one location are observed in a consistent order.
• Read-after-write visibility: a processor does not indefinitely read an old value after its own write.

A write-invalidate protocol invalidates other cached copies when one core writes. A write-update protocol broadcasts new data to other caches. Write-invalidate usually generates less traffic when data are written repeatedly by one core.

Snooping protocols monitor a shared broadcast medium and are practical for small systems. Directory protocols record sharers and send targeted messages, reducing broadcast traffic and supporting larger systems at the cost of directory storage and management.

The MESI protocol labels a cache line Modified, Exclusive, Shared, or Invalid. MOESI adds an Owned state, allowing a dirty line to be shared by one cache while memory remains stale. The exact states and transitions depend on the implementation.

False sharing is a performance problem, not a coherence failure: two cores modify different words in the same cache line, causing repeated invalidations. Padding, data partitioning, and ownership-aware layouts can reduce it.

Memory consistency

Memory consistency specifies the order in which memory operations become visible to processors. Coherence is per-location; consistency governs relationships among operations to different locations. Sequential consistency requires that the result is equivalent to some interleaving that preserves each processor’s program order. It is easy to reason about but can restrict hardware optimization.

Relaxed models allow loads and stores to be reordered subject to synchronization rules. Acquire operations prevent later memory operations from moving before the acquire; release operations prevent earlier operations from moving after the release, in the abstract programming model. Language standards and processor architectures differ, so portable concurrent programs should use defined atomic operations and memory-order primitives rather than relying on intuition.

A data-race-free program using the synchronization guarantees of its language or platform is generally easier to reason about than one relying on accidental ordering.

5. Synchronization and Communication

Synchronization coordinates access to shared state and establishes ordering. Common mechanisms are:

• Mutexes: provide mutual exclusion for critical sections.
• Spinlocks: repeatedly test a lock; appropriate only for short waits on systems where sleeping is costly.
• Semaphores: represent permits or counted resources.
• Barriers: release participants after all have arrived.
• Atomics: perform indivisible read-modify-write operations such as compare-and-swap.
• Message queues and mailboxes: transfer ownership or values without unrestricted shared access.
• Lock-free structures: use atomics to permit system-wide progress, but require careful proof and memory-reclamation design.

Amdahl-style reasoning applies to synchronization: if every iteration waits at a barrier, the fastest core may spend substantial time idle. Fine-grained locks can improve concurrency but increase overhead and contention. Coarse-grained locks are simpler but serialize more work. Priority inversion occurs when a high-priority task waits for a lock held by a low-priority task; priority inheritance or priority ceiling protocols can mitigate it in real-time systems.

A producer-consumer queue must define ownership, capacity, full and empty conditions, memory ordering, and interrupt behavior. In a real-time DSP pipeline, bounded queues are preferable because they allow worst-case latency analysis.

6. Interconnects and Networks-on-Chip

An interconnect determines how processors, memories, caches, and accelerators exchange requests and data. Important metrics are bandwidth, latency, topology, arbitration delay, energy per transfer, fault tolerance, and scalability.

A packet-switched NoC contains routers, links, endpoint network interfaces, buffers, routing logic, and flow control. A mesh places a router at each mesh coordinate; each router connects to neighboring routers and usually to a local endpoint. Links are normally bidirectional, either as paired directed channels or as a bidirectional physical connection.

Figure: A bidirectional mesh NoC. Every mesh coordinate contains a router; heterogeneous endpoints attach through local interfaces.

For a packet traversing H router-to-router links, a useful first-order latency model is

L_total is end-to-end latency; L_in and L_out are injection and endpoint/ejection delays; L_router is per-router forwarding delay; L_link is per-link propagation and serialization delay; and L_queue is the accumulated queuing delay along the path. H is defined explicitly as the number of links traversed, so the packet visits H forwarding transitions in the usual store-and-forward abstraction. If a convention counts H as routers visited rather than links, the equation must be redefined accordingly. The equation is a model, not a universal timing law: wormhole flow control, packet length, virtual channels, contention, and pipelining can change the detailed result.

For an uncontended 3-link route with L_in = 2 ns, L_router = 1 ns, L_link = 0.5 ns, L_out = 1 ns, and L_queue = 0,

This example includes router delay at every hop. Under contention, L_queue is positive and may dominate.

In an n × n mesh, the Manhattan distance between coordinates (x₁,y₁) and (x₂,y₂) is

This counts links between routers, assuming endpoints attach to the stated routers. It is not automatically the number of routers or switches encountered. For example, from (0,0) to (2,1), H = 2 + 1 = 3 links; the route may involve four routers including source and destination routers, depending on endpoint convention.

Bisection bandwidth, average hop count, buffer depth, and traffic distribution help evaluate a NoC. Adaptive routing can avoid congested regions but needs additional logic and must prevent deadlock, often through virtual channels or restricted turn rules.

7. Mapping Communication and Signal-Processing Workloads

A disciplined mapping process is:

1. Express the application as a task graph: nodes are computations and edges are data dependencies.
2. Measure operation types, data volume, deadlines, and parallelism.
3. Classify each task as regular/irregular, compute-/memory-bound, latency-sensitive, or throughput-oriented.
4. Select an execution engine.
5. Place communicating tasks near one another when possible.
6. Allocate buffers and define ownership and synchronization.
7. Measure execution, transfer, contention, and energy costs.
8. Rebalance and verify worst-case behavior.

A filter bank with independent channels is naturally data-parallel and maps well to SIMD, DSP cores, or a GPU. A control loop with unpredictable branches belongs on a CPU. A streaming FFT followed by beamforming may map to a DSP or FPGA when deterministic latency matters. Packet parsing, which has irregular lengths and branches, is often CPU-oriented, although selected stages may use accelerators.

A useful performance decomposition is

The terms are not always strictly additive because hardware overlaps them; therefore this is an accounting model for identifying bottlenecks, not a claim that all times occur serially. Arithmetic intensity is the ratio of useful arithmetic operations to bytes transferred from a specified memory level. High arithmetic intensity favors compute accelerators, while low intensity makes bandwidth and locality decisive.

8. Applied Case: Illustrative Software-Defined Radio Receiver

Background

Consider an illustrative 4-channel software-defined radio receiving a continuous stream of complex baseband samples. The system must meet a fixed block deadline while minimizing energy.

Problem

The receiver performs digital down-conversion, FIR filtering, FFT analysis, channel estimation, forward-error-correction decoding, and control/packet handling. A single CPU misses deadlines during high traffic, but moving everything to a GPU produces excessive transfer latency for small blocks.

Analysis

Down-conversion and FIR filtering contain repeated multiply-accumulate operations and have predictable data access. FFT processing is regular but requires careful memory layout. FEC decoding may expose parallelism but can involve data-dependent iterations. Control and packet handling are branch-heavy.

The task graph shows high-volume sample edges between front-end stages and stricter latency for control edges. Placing communicating stages on distant NoC nodes would increase hop count and contention.

Approach / Solution

The CPU handles configuration, packet scheduling, and exception handling. A DSP performs down-conversion and FIR filtering using circular buffers and SIMD multiply-accumulate instructions. A GPU processes sufficiently large FFT batches when transfer and launch overhead are amortized. A dedicated accelerator or DSP handles FEC decoding when deterministic timing is required. Double-buffered DMA transfers overlap movement of one block with computation on another.

Buffers are partitioned by channel to reduce false sharing. Producer-consumer queues use bounded capacity. Release/acquire operations publish completed buffers, while a control-plane mutex protects configuration changes. The NoC placement keeps the DSP, accelerator, and sample memory close and reserves bandwidth for real-time traffic.

Outcome

The heterogeneous arrangement improves throughput without requiring every task to use the GPU. More importantly, it makes deadlines analyzable: predictable stages use fixed-size blocks and bounded queues, while non-real-time management is isolated from the sample path.

Learning Points

1. Peak accelerator throughput does not guarantee lower end-to-end latency.
2. Data movement and synchronization must be included in mapping decisions.
3. Regular signal-processing kernels benefit from SIMD, DSP, or GPU execution.
4. Control and exception paths need flexible processors.
5. Placement, buffering, and priority are part of architecture, not afterthoughts.

9. Challenges, Limitations, and Contemporary Directions

Parallel systems face power and thermal limits, memory-wall effects, synchronization overhead, load imbalance, nondeterminism, and difficult debugging. Heterogeneous programming also creates portability problems: an optimization for one GPU, DSP, or NoC may not transfer to another platform.

Verification must address data races, deadlock, livelock, numerical reproducibility, accelerator faults, and timing deadlines. Safety-critical communications equipment may require traceability, isolation, fault detection, and certified development processes. Shared data can also create security risks such as cache side channels; partitioning, constant-time techniques, and access controls may be necessary.

Contemporary systems increasingly combine chiplets, 3-D integration, coherent accelerators, near-memory processing, sparse AI engines, and programmable data paths. AI can assist compiler scheduling or workload prediction, but its use does not remove the need for deterministic bounds, validation, and human oversight in safety- or mission-critical systems. Open standards and portable programming models improve reuse, but they cannot eliminate differences in memory hierarchy and synchronization cost.

Best practice is to optimize only after measurement, document the memory model, specify ownership and synchronization, use roofline-style compute/bandwidth analysis where appropriate, test worst-case traffic, and preserve a clear reference implementation for verification.

Chapter Summary

Parallel architecture uses multiple instruction and data streams to increase throughput or reduce latency. SIMD and vector processing exploit regular data parallelism; MIMD supports independent tasks; CPUs provide flexible control; DSPs provide efficient deterministic numerical processing; GPUs provide massive throughput for regular workloads; and heterogeneous systems combine these strengths.

Performance is limited by serial work, data movement, memory bandwidth, synchronization, imbalance, and interconnect contention. Cache coherence manages multiple cached copies of a location, whereas memory consistency defines visibility ordering across operations. Correct concurrent software requires explicit atomics, locks, barriers, queues, and documented memory-order rules.

Interconnect topology strongly affects scalability. A mesh NoC uses routers at mesh coordinates and bidirectional links. A sound latency model includes per-hop router and link delay, endpoint terms, and queuing; hop definitions must be stated. Effective workload mapping treats computation, communication, placement, energy, and real-time constraints as one design problem.

Key Terms

SIMD: Single instruction, multiple data execution over parallel data lanes.

MIMD: Multiple independent instruction streams operating on multiple data streams.

Vector processor: Processor that applies vector instructions to sequences of elements.

Multicore: A chip containing multiple processing cores.

NUMA: Memory organization in which access time depends on the memory’s location relative to a core.

GPU: Throughput-oriented processor with many parallel arithmetic lanes.

DSP: Processor specialized for repetitive, numerical signal-processing operations.

Heterogeneous architecture: System combining different processor or accelerator types.

Cache coherence: Rules ensuring compatible visibility of copies of a memory location.

Memory consistency: Rules defining ordering and visibility of memory operations.

False sharing: Coherence traffic caused by independent variables sharing one cache line.

Atomic operation: Indivisible operation that cannot be observed halfway through.

Barrier: Synchronization point at which participants wait for one another.

NoC: Packet-switched network connecting components on a chip.

Router: NoC component that selects an output direction for a packet or flit.

Hop: One defined link traversal in a network path.

Bisection bandwidth: Capacity across a minimum cut dividing a network into two parts.

Arithmetic intensity: Useful arithmetic operations per byte transferred from a specified memory level.

False sharing: Unnecessary invalidation caused by distinct words occupying one cache line.

Workload mapping: Assigning computation and data to processing elements and memories.

Review Questions

Short Answer Questions

1. What distinguishes SIMD from MIMD execution?
2. Why can false sharing reduce multicore performance even when different words are being modified?
3. State the difference between cache coherence and memory consistency.
4. Name two advantages and two costs of heterogeneous processing.
5. What does a router do in a mesh NoC?

Long Answer Questions

1. Compare CPU, GPU, and DSP architectures for a real-time digital receiver.
2. Explain write-invalidate coherence and contrast snooping with directory protocols.
3. Discuss how Amdahl’s law influences the usefulness of adding processor cores.
4. Explain memory ordering, acquire/release synchronization, and the role of atomic operations.
5. Describe the design trade-offs among bus, ring, crossbar, and mesh interconnects.

Analytical / Application Questions

1. A workload is 85% parallelizable. Calculate ideal speedup and efficiency on 6 processors, then explain why measured results may be lower.
2. In a mesh, a packet travels from (1,0) to (3,2). Calculate its Manhattan hop count and identify the quantities required for an end-to-end latency estimate.
3. Design a processor mapping for a four-stage radio pipeline containing FIR filtering, FFT, packet parsing, and control. Justify processor selection, buffer ownership, synchronization, and NoC placement.

Multiple Choice Questions with Answers

Question 1

Which workload is most naturally suited to SIMD execution?

A. Independent FIR outputs using the same coefficients and regular sample access

B. A control task with unpredictable branches and one input value

C. A mutual-exclusion protocol for a shared lock

D. A device-reset routine with sequential dependencies

Correct Answer: A. Independent FIR outputs using the same coefficients and regular sample access

Question 2

For p = 0.90 and N = 8, what is the ideal Amdahl speedup?

A. 2.50

B. 4.71

C. 7.20

D. 8.90

Correct Answer: B. 4.71

Question 3

What is the main purpose of cache coherence?

A. To increase processor clock frequency

B. To ensure compatible visibility of cached copies of a memory location

C. To guarantee that all instructions execute in source order

D. To eliminate all memory-access latency

Correct Answer: B. To ensure compatible visibility of cached copies of a memory location

Question 4

Two cores repeatedly modify different words in the same cache line. What problem may result?

A. Dead code elimination

B. False sharing

C. Instruction starvation

D. Branch fusion

Correct Answer: B. False sharing

Question 5

In the chapter’s NoC latency model, H is defined as:

A. The number of bytes in a packet

B. The number of router-to-router links traversed

C. The number of processors in the system

D. The number of cache states

Correct Answer: B. The number of router-to-router links traversed

Question 6

A packet crosses 3 links with L_in = 2 ns, L_router = 1 ns per hop, L_link = 0.5 ns per hop, L_out = 1 ns, and no queuing. What is its modeled latency?

A. 4.5 ns

B. 6.0 ns

C. 7.5 ns

D. 9.0 ns

Correct Answer: C. 7.5 ns

Question 7

Which processor is generally best suited to branch-heavy protocol control?

A. A general-purpose CPU

B. A fixed-function FIR accelerator only

C. A GPU requiring thousands of identical threads

D. A vector unit without scalar control support

Correct Answer: A. A general-purpose CPU

Question 8

Which statement correctly describes memory consistency?

A. It defines visibility and ordering relationships among memory operations

B. It means every cache line is stored in every cache

C. It counts the number of routers in a NoC

D. It is identical to arithmetic intensity

Correct Answer: A. It defines visibility and ordering relationships among memory operations

Question 9

What is a likely benefit of a directory coherence protocol over broadcast snooping in a larger system?

A. It removes the need for memory

B. It can target coherence messages to recorded sharers

C. It guarantees zero cache misses

D. It requires only one processor

Correct Answer: B. It can target coherence messages to recorded sharers

Question 10

For mesh coordinates (0,0) and (2,1), what is the Manhattan hop count?

A. 1

B. 2

C. 3

D. 4

Correct Answer: C. 3