2. Quantitative Design and Performance EvaluationIntroduction and ContextComputer design is ultimately concerned with producing useful results within acceptable limits of time, energy, cost, reliability, and complexity. Architectural ideas such as pipelining, caching, parallel execution, vector processing, and specialization are valuable only when their effects can be measured and related to a workload. Quantitative design provides the discipline for making those comparisons. Performance evaluation is not simply the act of reporting a processor’s clock frequency or a benchmark score. It involves defining the task, identifying the relevant performance metric, constructing an appropriate model, measuring execution under controlled conditions, and interpreting the result in relation to cost and constraints. A design that is fastest for one workload may be inefficient for another; a design with the highest throughput may have unacceptable response time; and an accelerator that improves performance may consume too much energy to be suitable for deployment. This chapter develops the principal quantitative tools used in computer design. It begins with execution-time equations and the relationship among instruction count, cycles per instruction, and clock rate. It then examines latency, throughput, speedup, Amdahl’s law, Gustafson’s law, benchmarking, workload characterization, roofline analysis, and performance-per-watt and performance-per-cost measures. The emphasis is on interpretation: the objective is not merely to calculate a number, but to use that number to support a defensible design decision. Learning ObjectivesAfter studying this chapter, a student should be able to: 1. Derive and apply execution-time equations using instruction count, CPI, and clock frequency. 2. Distinguish latency, throughput, response time, and utilization. 3. Calculate and interpret speedup, efficiency, and scalability. 4. Apply Amdahl’s law and Gustafson’s law to parallel and specialized designs. 5. Design meaningful benchmark experiments and identify sources of measurement error. 6. Characterize workloads using instruction mix, locality, parallelism, and arithmetic intensity. 7. Use roofline analysis to identify compute-bound and memory-bound behavior. 8. Compare systems using performance-per-watt, performance-per-cost, and total cost considerations. 9. Evaluate quantitative evidence critically when making architectural decisions. Performance as a Quantitative Design ObjectiveThe most direct definition of performance is the amount of useful work completed per unit time. For a single request, performance is often represented as the reciprocal of execution time: \[ Performance = \frac{1}{Execution\ Time} \] Thus, lower execution time means higher performance. This definition is especially appropriate for batch programs, where the principal concern is how long one job takes. In interactive and real-time systems, however, other measures may be more important. Response time is the delay experienced by an individual request, while throughput is the number of completed requests per unit time. A web server may be judged by requests per second and tail latency, whereas a scientific application may be judged by the time required to complete one simulation. Latency and throughput should not be confused. A pipeline can have a latency of several cycles for one operation while accepting a new operation every cycle. Increasing the number of functional units may improve throughput without reducing the latency of an individual operation. Conversely, reducing the depth of a pipeline may reduce latency but limit the maximum operating frequency. A sound evaluation begins by stating the performance question. “Which system is faster?” is incomplete unless the task, input size, software configuration, and metric are specified. More precise questions include: Which processor minimizes the time to factor a matrix of a given size? Which accelerator processes the greatest number of images per joule? Which server satisfies a 99th-percentile response-time constraint at the lowest capital cost? The Execution-Time ModelFor a program executed on a processor, the fundamental equation is \[ CPU\ Time = CPU\ Clock\ Cycles \times Clock\ Cycle\ Time \] Since clock cycle time is the reciprocal of clock frequency, \[ CPU\ Time = \frac{CPU\ Clock\ Cycles}{Clock\ Rate} \] If the program executes IC instructions and the average number of clock cycles per instruction is CPI, then \[ CPU\ Clock\ Cycles = IC \times CPI \] and therefore \[ CPU\ Time = \frac{IC \times CPI}{Clock\ Rate} \] This equation is central because it exposes three design levers: reducing instruction count, reducing average CPI, and increasing clock rate. These levers are not independent. A compiler optimization may increase instruction count but reduce CPI; a deeper pipeline may increase clock rate but introduce branch penalties; and a complex instruction may reduce instruction count while requiring multiple internal cycles. The reciprocal form is also useful: \[ CPI = \frac{CPU\ Clock\ Cycles}{Instruction\ Count} \] Average CPI hides the fact that instructions have different costs. If instruction class i represents a fraction fᵢ of all instructions and requires CPIᵢ cycles, then \[ CPI_{average} = \sum_i f_i CPI_i \] where the instruction fractions sum to one. For example, if 50% of instructions have CPI 1, 30% have CPI 2, and 20% have CPI 4, the average CPI is 0.5(1) + 0.3(2) + 0.2(4) = 1.9. This weighted formulation helps identify whether performance is dominated by arithmetic operations, memory references, branches, synchronization, or another class. Worked Example: Separating the Design LeversSuppose a program executes 2.4 × 10⁹ instructions on a 3 GHz processor with an average CPI of 1.5. Its execution time is \[ T = \frac{2.4 \times 10^9 \times 1.5}{3 \times 10^9} = 1.2\ seconds \] Assume a compiler transformation reduces instruction count by 15%, but increases CPI by 8%. The new time is proportional to \[ 0.85 \times 1.08 = 0.918 \] of the original time. The execution time becomes approximately 1.102 seconds, giving a speedup of about 1.089, or 8.9%. The transformation is beneficial even though CPI increased, because the reduction in instruction count is larger in relative terms. This example illustrates why isolated metrics can mislead. A lower CPI is not automatically better if it requires many more instructions, and a higher clock rate is not necessarily useful if it causes substantial stalls. Speedup, Efficiency, and ScalingSpeedup compares a baseline system with an improved system: \[ Speedup = \frac{T_{old}}{T_{new}} \] A speedup of 2 means that the new system completes the task in half the time. When comparing performance values instead of times, the ratio is reversed because performance is the reciprocal of time. For a parallel system with p processing elements, parallel efficiency is commonly defined as \[ Efficiency = \frac{Speedup}{p} \] An efficiency of 0.75 means that the system achieves 75% of ideal linear scaling. Efficiency below one results from serial work, communication, synchronization, load imbalance, memory contention, and resource overheads. Efficiency is meaningful only when the workload and the number of processing elements are clearly specified. Strong scaling measures how execution time changes as more processors are applied to a fixed problem size. Weak scaling measures whether execution time remains approximately constant when both the problem size and processor count increase proportionally. A system can show good weak scaling while showing poor strong scaling. This distinction is important in high-performance computing, cloud services, and distributed analytics. Amdahl’s LawAmdahl’s law estimates the maximum speedup obtained when only a portion of a workload is improved. If fraction f of the original execution time benefits from a speedup s, then \[ Speedup = \frac{1}{(1-f) + \frac{f}{s}} \] The part that is not improved remains a limiting factor. If 80% of a program can be accelerated by a factor of 10, the overall speedup is \[ \frac{1}{0.2 + 0.8/10} = 3.57 \] not 10. Even an infinitely fast implementation of the improved part would provide at most 1/(1 − 0.8) = 5 times speedup. Amdahl’s law is valuable for prioritizing engineering effort. Before investing in a specialized accelerator, a designer should estimate the fraction of total execution time that the accelerator can affect. A spectacular accelerator may have little system-level effect if data preparation, I/O, synchronization, or serial control dominates. Gustafson’s LawAmdahl’s law assumes a fixed problem size. Gustafson’s law considers scaled workloads in which additional processing capacity is used to solve a larger problem in approximately the same elapsed time. If s is the fraction of parallel execution time spent in the parallel portion and p processors are used, a common form is \[ Speedup_G = p - (1-s)(p-1) \] Gustafson’s perspective is appropriate when users want higher resolution, larger datasets, more detailed simulations, or more simultaneous requests as resources increase. The two laws are not contradictory. Amdahl’s law highlights the limit for a fixed workload; Gustafson’s law describes the benefit when the workload scales with the machine. Benchmarking as an Experimental MethodA benchmark is a repeatable workload and measurement procedure used to compare systems. Good benchmarking requires more than running a program once and recording elapsed time. The experiment should specify hardware, operating-system version, compiler and flags, input data, thread count, power state, warm-up policy, measurement interval, and statistical treatment. Benchmarks can be classified as: - Microbenchmarks, which isolate operations such as memory latency, branch prediction, or floating-point throughput.
- Kernel benchmarks, which represent computationally important loops or functions.
- Synthetic benchmarks, which generate controlled instruction or access patterns.
- Application benchmarks, which execute realistic programs or representative sections of them.
- End-to-end benchmarks, which measure the complete user-visible workflow, including I/O and orchestration.
Each category answers a different question. A microbenchmark may reveal a hardware capability but fail to predict application performance. An end-to-end benchmark is realistic but may make it difficult to identify the source of a bottleneck. A reliable measurement generally follows this sequence: define the performance question, select a representative workload, configure systems equivalently, warm up the software when necessary, collect repeated observations, examine variability, and report both central tendency and relevant tail behavior. The arithmetic mean is not always appropriate. For execution times across different inputs, the geometric mean is often useful for aggregating normalized ratios, while percentile values are essential for latency-sensitive services. Benchmark results can be distorted by compiler optimization, cache warmth, background activity, turbo frequency changes, thermal throttling, data placement, virtualization, and inadequate sample size. Reporting only the best run can produce an optimistic and irreproducible result. Reproducibility also requires publishing the workload and measurement conditions rather than presenting an isolated score. Workload CharacterizationWorkload characterization connects a program’s behavior with architectural resources. Important dimensions include instruction mix, working-set size, memory locality, branch behavior, parallelism, synchronization, input variability, and arithmetic intensity. Instruction mix identifies the proportion of arithmetic, load/store, control-flow, and system instructions. Memory behavior includes cache-hit rates, miss penalties, bandwidth demand, and access regularity. Temporal locality means that recently accessed data is likely to be reused; spatial locality means that nearby addresses are likely to be accessed. These properties influence cache effectiveness and memory traffic. Parallelism may exist at the instruction, data, task, or request level. A vectorizable numerical loop exposes data-level parallelism, while independent user requests expose request-level parallelism. A workload with abundant parallelism may still scale poorly if synchronization or memory bandwidth becomes dominant. Arithmetic intensity is the amount of computation performed per byte transferred to or from a limiting memory level: \[ Arithmetic\ Intensity = \frac{Operations}{Bytes\ Transferred} \] A matrix multiplication with substantial reuse may have high arithmetic intensity, whereas streaming vector addition typically performs few operations per byte. Arithmetic intensity is central to roofline analysis. Roofline AnalysisThe roofline model provides a visual and analytical method for estimating attainable performance. It compares a processor’s peak computational rate with its memory-bandwidth limit. If peak compute performance is Pₘₐₓ operations per second, sustainable memory bandwidth is B bytes per second, and arithmetic intensity is I operations per byte, then the attainable performance is bounded by \[ Performance \leq min(P_{max}, B \times I) \] The intersection of the memory roof and compute roof occurs at \[ I^* = \frac{P_{max}}{B} \] If a workload has intensity below I, it is likely memory-bound; increasing arithmetic-unit capacity alone will provide little benefit. If its intensity exceeds I, it may be compute-bound, so improving arithmetic throughput or instruction efficiency may help. For example, suppose a processor has a peak rate of 2 tera-operations per second and a sustainable memory bandwidth of 500 gigabytes per second. The ridge point is 2,000/500 = 4 operations per byte. A workload with intensity 1 operation per byte has a bandwidth roof of 500 giga-operations per second and is memory-bound. A workload with intensity 8 operations per byte is limited by the 2 tera-operation compute roof, assuming other overheads are negligible. The roofline model is a bound, not a guarantee. Real performance may fall below the roof because of instruction dependencies, imperfect vectorization, cache conflicts, branch divergence, synchronization, limited occupancy, or inefficient memory access. Multi-level roofline models can represent different cache and memory bandwidth ceilings. The model is especially useful because it suggests targeted optimizations: increase reuse and reduce data movement for a memory-bound kernel, or improve vectorization and instruction scheduling for a compute-bound kernel. Energy, Power, and CostPerformance alone is insufficient for many contemporary systems. Power is the instantaneous rate of energy consumption, measured in watts, while energy is power integrated over time. For a fixed task, \[ Energy = Power \times Execution\ Time \] when power is approximately constant. A system with higher power can still use less energy if it finishes substantially sooner, although peak power and cooling requirements may remain important. Performance per watt is commonly used to compare energy efficiency: \[ Performance/Watt = \frac{Useful\ Work\ per\ Unit\ Time}{Power} \] Alternatively, energy per operation or energy per completed request may be more directly interpretable. Measurements should distinguish dynamic power, static or leakage power, memory power, storage power, and system overhead. Accelerator-only measurements can be misleading if host processors, data movement, and cooling are excluded. Cost evaluation may include acquisition cost, software licensing, maintenance, electricity, facilities, staffing, and replacement. A useful metric is performance per unit cost, but the relevant cost period must be stated. Total cost of ownership is often more meaningful than purchase price. A slower system may be preferable if it is substantially cheaper, more reliable, or easier to maintain; conversely, a more expensive system may be justified when reduced execution time directly increases revenue or scientific productivity. Multi-objective decisions can be represented using a constraint or Pareto analysis. A design is dominated if another design is at least as good in every relevant dimension and strictly better in one. When no design dominates, the choice depends on priorities such as latency, throughput, energy, capital cost, and reliability. An Illustrative Case: Selecting an Accelerator for Image InferenceBackgroundConsider an illustrative organization deploying a convolutional neural-network inference service. The service must process a large stream of images while keeping 99th-percentile response time below a specified operational target. Three candidate platforms are available: a general-purpose CPU, a GPU, and a specialized inference accelerator. Situation and ProblemA preliminary test reports that the GPU has the highest peak operations per second. However, the complete service includes image decoding, preprocessing, transfer to the accelerator, inference, and postprocessing. Management must decide whether the GPU should be deployed or whether the accelerator’s lower peak rate may provide better system value. AnalysisThe team first measures the end-to-end pipeline rather than inference alone. Profiling shows that inference represents 70% of baseline execution time, preprocessing represents 15%, data transfer 10%, and postprocessing 5%. Even if inference becomes infinitely fast, the non-inference components impose a theoretical maximum speedup of 1/0.30 = 3.33 under a fixed workload. This is an Amdahl constraint. The team then characterizes the inference kernels. Some convolution layers have high arithmetic intensity and are compute-bound, while other layers have low intensity and are limited by memory traffic. Roofline analysis indicates that improving peak arithmetic throughput will help only the compute-bound layers. The accelerator, although having lower peak throughput than the GPU, provides better data reuse and lower transfer overhead for the model’s common tensor shapes. Repeated end-to-end tests are conducted after warm-up, using identical input batches and software versions. Median latency, 99th-percentile latency, images per second, power, and acquisition cost are recorded. The GPU provides the highest throughput at large batch sizes, but its latency becomes unacceptable at small batches and its power is higher. The specialized accelerator provides slightly lower peak throughput but better tail latency and higher images per joule. The CPU has the lowest acquisition cost but cannot satisfy the throughput requirement without deploying multiple nodes. Approach and DecisionThe organization chooses the specialized accelerator for the latency-sensitive service and retains GPUs for an offline batch-processing system. This is not a universal conclusion that specialized accelerators are superior. It follows from matching each platform to a workload and metric: tail latency and energy efficiency for online inference, and maximum throughput for large offline batches. Outcome and Learning PointsThe case demonstrates five principles. First, peak specifications do not equal application performance. Second, end-to-end measurement is necessary when data movement and preprocessing matter. Third, Amdahl’s law limits the benefit of optimizing only one phase. Fourth, roofline analysis helps explain why different kernels respond differently to hardware improvements. Finally, performance-per-watt and latency constraints can change the preferred design even when raw throughput points to another platform. Common Errors and Limits of Quantitative EvaluationSeveral mistakes recur in performance studies. One is confusing clock rate with performance; clock frequency is only one term in the execution-time equation. Another is comparing averages when tail latency determines user experience. A third is using a nonrepresentative benchmark or optimizing specifically for a benchmark that does not reflect production behavior. Other errors include double-counting speedups, ignoring initialization and data-transfer costs, comparing different compiler settings, failing to report input size, and treating a theoretical peak as an attainable sustained rate. A benchmark may also exhibit a statistical selection bias if only favorable runs are reported. Measurement tools themselves can perturb timing through instrumentation overhead. Models such as Amdahl’s law and roofline analysis simplify reality. They are valuable for reasoning, but they do not replace measurement. Results may also change with software versions, workload distributions, thermal conditions, hardware aging, and security mitigations. Therefore, quantitative conclusions should state assumptions and uncertainty. Professional and Ethical ConsiderationsPerformance claims are technical communication and should be reproducible, transparent, and appropriately qualified. Organizations should disclose whether measurements are peak, sustained, kernel-only, or end-to-end. Benchmark manipulation, selective reporting, undisclosed hardware differences, and inappropriate extrapolation can lead to costly engineering decisions and undermine trust. Energy and cost claims also have governance implications. A design that minimizes operational energy may require more embodied resources or create electronic waste through frequent replacement. Data-center decisions should consider cooling, grid conditions, resilience, and service-level obligations. In safety-critical or regulated systems, a faster implementation must not compromise correctness, determinism, security, or auditability. Emerging DirectionsPerformance evaluation is increasingly heterogeneous. CPUs, GPUs, tensor accelerators, reconfigurable logic, near-memory devices, and distributed systems may cooperate in one application. Consequently, evaluation must include data movement, orchestration overhead, and software portability rather than focusing on a single processor. Automated performance analysis uses hardware counters, tracing, statistical profiling, and machine-learning models to identify bottlenecks. AI-assisted compiler optimization can search schedules, tilings, and mappings, but its recommendations still require validation on representative workloads. Carbon-aware scheduling and energy-proportional computing are also expanding the objective from performance per watt to performance subject to energy and environmental constraints. Reproducible benchmarking is becoming more important as systems grow complex. Containerized environments, workload suites, open measurement scripts, confidence intervals, and artifact publication help researchers and engineers distinguish genuine architectural improvements from experimental noise. Best Practices for Design DecisionsA disciplined evaluation should: 1. Define the user-visible objective and constraints before selecting a metric. 2. Use the execution-time equation to identify plausible optimization levers. 3. Separate latency, throughput, utilization, energy, and cost rather than treating them as interchangeable. 4. Characterize the workload before choosing hardware. 5. Measure end-to-end behavior as well as important kernels. 6. Use Amdahl’s law to estimate the value of optimizing a program fraction. 7. Use Gustafson’s law when problem size or service demand scales with resources. 8. Apply roofline analysis to distinguish computation limits from data-movement limits. 9. Repeat experiments, report variability, and document configurations. 10. Present trade-offs and assumptions clearly instead of reporting a single attractive number. Chapter SummaryQuantitative design links architectural choices to measurable outcomes. Program execution time is determined by instruction count, average CPI, and clock rate, but these variables interact and must be considered together. Speedup measures improvement, while efficiency describes how effectively additional processing resources are used. Amdahl’s law limits the benefit of optimizing a fixed fraction of execution; Gustafson’s law explains why larger scalable workloads can continue to benefit from parallel resources. Benchmarking is an experimental activity requiring representative workloads, controlled conditions, repeated measurements, and transparent reporting. Workload characterization examines instruction mix, locality, parallelism, memory behavior, and arithmetic intensity. Roofline analysis relates arithmetic intensity to compute and bandwidth ceilings, guiding optimization toward either computation or data movement. Finally, performance-per-watt and performance-per-cost extend evaluation beyond speed. Sound design decisions therefore combine models, measurements, constraints, and professional judgment. Key TermsExecution time: The elapsed time required to complete a specified computation. Clock rate: The number of processor clock cycles generated per second. CPI: Average clock cycles required to execute one instruction. Instruction count: The number of instructions executed by a program for a specified input and implementation. Latency: Time between initiating an operation and receiving its result. Throughput: Amount of useful work or number of requests completed per unit time. Speedup: Ratio of baseline execution time to improved execution time. Parallel efficiency: Speedup divided by the number of processing elements used. Strong scaling: Performance behavior as processor count increases for a fixed problem size. Weak scaling: Performance behavior as problem size increases with processor count. Amdahl’s law: A model showing how an unimproved portion limits total speedup. Gustafson’s law: A model estimating speedup for workloads that scale with available processors. Benchmark: A repeatable workload and procedure used to measure or compare performance. Workload characterization: Analysis of the computational, memory, control-flow, and parallel properties of a workload. Arithmetic intensity: Operations performed per byte transferred. Roofline model: A performance-bound model based on compute capacity and memory bandwidth. Performance per watt: Useful performance delivered for each watt of power consumed. Total cost of ownership: Acquisition and ongoing costs incurred throughout a system’s useful life. Tail latency: High-percentile response time, such as the 99th-percentile latency. Review QuestionsShort Answer Questions1. State the CPU execution-time equation in terms of instruction count, CPI, and clock rate. 2. Distinguish latency from throughput using a suitable example. 3. What does average CPI conceal, and how can instruction mix reveal it? 4. Define arithmetic intensity and explain its relevance to memory behavior. 5. Why is peak processor throughput not necessarily equal to application performance? Long Answer Questions1. Explain the execution-time model and analyze the trade-offs among instruction count, CPI, and clock rate. 2. Derive Amdahl’s law and discuss its implications for accelerator design. 3. Compare Amdahl’s law and Gustafson’s law, including the situations in which each is most useful. 4. Describe a rigorous benchmarking methodology for comparing two heterogeneous computing platforms. 5. Explain roofline analysis and show how it guides optimization of compute-bound and memory-bound workloads. Analytical / Application Questions1. A program spends 65% of its time in a kernel that is accelerated by a factor of 12. Calculate the overall speedup and identify the limiting factor. 2. A processor has a peak performance of 3 tera-operations per second and sustainable memory bandwidth of 600 GB/s. Determine the ridge-point arithmetic intensity and classify a workload with intensity 2 operations per byte. 3. Two systems have equal throughput, but System A uses 250 W and costs twice as much as System B, which uses 400 W. Explain which system is preferable under energy-efficiency, acquisition-cost, and total-cost-of-ownership objectives, and identify what additional measurements are required. 22. Multiple Choice Questions with AnswersQuestion 1Which expression correctly gives CPU execution time when instruction count, average CPI, and clock rate are known? A. Instruction count × clock rate ÷ CPI B. Instruction count × CPI ÷ clock rate C. CPI × clock rate ÷ instruction count D. Clock rate ÷ (instruction count × CPI) Correct Answer: B. Instruction count × CPI ÷ clock rate Question 2A program has 80% of its execution time accelerated by a factor of 10. What is the maximum overall speedup according to Amdahl’s law? A. 10.0 B. 8.0 C. 5.0 D. 3.57 Correct Answer: D. 3.57 Question 3Which situation is most appropriately analyzed using Gustafson’s law? A. A fixed-size task must finish faster on a new processor B. A larger problem is solved as additional processors become available C. A single instruction’s latency is measured D. A cache hit rate is calculated Correct Answer: B. A larger problem is solved as additional processors become available Question 4A workload performs 600 operations while transferring 200 bytes. What is its arithmetic intensity? A. 0.33 operations per byte B. 3 operations per byte C. 200 operations per byte D. 800 operations per byte Correct Answer: B. 3 operations per byte Question 5A system has peak performance of 2 TOPS and memory bandwidth of 500 GB/s. At what arithmetic intensity do the compute and bandwidth ceilings intersect? A. 0.25 operations per byte B. 2 operations per byte C. 4 operations per byte D. 250 operations per byte Correct Answer: C. 4 operations per byte Question 6Which metric is most appropriate when the main requirement is the number of completed requests per second? A. Latency B. Throughput C. CPI D. Clock cycle time Correct Answer: B. Throughput Question 7Why can a lower-CPI processor still be slower than another processor? A. CPI is unrelated to execution time B. The lower-CPI processor may execute many more instructions or have a lower clock rate C. A lower CPI always increases memory bandwidth D. Clock rate cannot affect performance Correct Answer: B. The lower-CPI processor may execute many more instructions or have a lower clock rate Question 8In roofline analysis, a workload with arithmetic intensity below the ridge point is generally limited by which resource? A. Memory bandwidth B. Peak arithmetic throughput C. Instruction count only D. Acquisition cost Correct Answer: A. Memory bandwidth Question 9Which practice most improves the credibility of a benchmark result? A. Reporting only the fastest run B. Changing compiler options for each system without disclosure C. Using repeated measurements and documenting workload and configuration D. Using only peak hardware specifications Correct Answer: C. Using repeated measurements and documenting workload and configuration Question 10Two systems complete the same useful work, but one completes it in less time while consuming more power. Which measure directly determines the energy used for that task? A. Power divided by clock rate B. Execution time multiplied by power C. CPI multiplied by instruction count D. Throughput divided by latency Correct Answer: B. Execution time multiplied by power
|