Concurrency, Memory, and Performance

Introduction

Modern software rarely performs one task at a time. A web server may receive thousands of requests while reading files, querying databases, and communicating with external services. A mobile application must respond to user input while rendering its interface. Data-processing systems may divide a large computation across multiple processor cores. These requirements make memory management, concurrency, and performance central concerns in professional software development.

This chapter examines how Java and Python represent objects in memory, reclaim unused objects, share state, and execute concurrent work. It also explains race conditions, synchronization, processes, threads, futures, asynchronous programming, profiling, and safe optimization. The central principle is that performance improvements must be based on evidence: an optimization that is not measured may add complexity without improving the system.

Learning Objectives

By the end of this chapter, you should be able to:

• Explain object references, reachability, mutability, and garbage collection.
• Distinguish processes, threads, parallelism, concurrency, and asynchronous execution.
• Identify race conditions and explain how synchronization prevents unsafe access.
• Use suitable approaches involving Java threads, Python threads, processes, futures, and asynchronous tasks.
• Interpret basic time, memory, throughput, latency, and scalability measurements.
• Profile a system and select optimizations that preserve correctness.
• Evaluate performance, safety, maintainability, and resource trade-offs.

1. Memory as a Runtime Resource

A program uses several logically different memory areas. The stack normally stores method-call frames, local variables, and execution state. Each thread has its own stack. The heap stores dynamically allocated objects and is commonly shared by threads within a process. A process also contains executable code, static data, operating-system resources, and libraries.

The exact layout is implementation-dependent. Java programs execute inside the Java Virtual Machine (JVM), while Python programs execute within an interpreter such as CPython. Therefore, source-level statements should not be assumed to correspond directly to a particular physical memory operation.

References and objects

A variable that refers to an object does not generally contain the complete object itself. It contains a reference, or handle, through which the object can be accessed. Assignment may copy a reference rather than duplicate the object.

first = [10, 20]
second = first
second.append(30)
print(first)       # [10, 20, 30]

Both variables refer to the same list. In Java, a similar effect occurs with object references:

List<Integer> first = new ArrayList<>();
first.add(10);
List<Integer> second = first;
second.add(20);    // first also observes 20

Primitive values such as Java int and Python immutable integers have different language-level semantics from mutable objects, although implementations may optimize their storage.

Mutability and immutability

A mutable object can change its state after creation. Lists, dictionaries, Java ArrayList objects, and many application-domain entities are mutable. An immutable object cannot be changed after creation; an apparent modification creates another value. Strings are immutable in both Java and Python.

Immutability is especially valuable in concurrent programs because immutable data can be shared without synchronization for state changes. It also reduces accidental aliasing. However, an immutable reference does not necessarily imply an immutable object. In Java, final prevents reassignment of a reference but does not automatically prevent mutation of the referenced object.

final List<String> names = new ArrayList<>();
names.add("Asha");       // allowed
// names = new ArrayList<>();  // not allowed

A reliable design distinguishes between ownership, permitted mutation, and visibility to other components. Defensive copying, read-only interfaces, and immutable value objects can make these boundaries explicit.

2. Garbage Collection and Object Lifetime

Garbage collection (GC) automatically reclaims objects that can no longer be reached by a live program. An object is normally eligible for collection when no chain of references from a root leads to it. Roots may include active stack variables, static fields, thread-local data, and runtime-managed references.

A simplified tracing model has three stages:

1. Start from roots and mark reachable objects.
2. Treat unmarked objects as unreachable.
3. Reclaim or reorganize their storage.

Real collectors use generational collection, concurrent marking, compaction, and other techniques. Young objects often die quickly, so many collectors inspect recently allocated regions more frequently than long-lived regions. Collection may pause application threads, run concurrently with them, or use a hybrid strategy.

Java provides automatic memory management through the JVM. Java developers must still release non-memory resources such as files, sockets, database connections, and locks. A garbage collector does not guarantee timely release of these resources. Use try-with-resources in Java and context managers such as Python's with statement where appropriate.

CPython primarily uses reference counting and supplements it with a cyclic garbage collector. Reference counting can reclaim an object immediately when its count reaches zero, but cycles require additional detection. Other Python implementations may use different memory-management strategies. Consequently, code should not depend on the exact time at which an object is destroyed.

Memory leaks in garbage-collected languages

Garbage collection prevents leaks caused by failing to call free, but applications can still retain objects unnecessarily. Common causes include:

• Unbounded caches.
• Global collections that are never cleared.
• Event listeners that remain registered.
• Thread-local values with excessive lifetime.
• Queues that grow faster than consumers remove items.
• Closures or callbacks retaining large object graphs.

A memory leak in a managed runtime is usually a reachability problem: the object is still reachable, although the application no longer needs it.

3. Concurrency, Parallelism, and Execution Units

Concurrency is the composition of multiple tasks whose execution periods overlap. Parallelism is the simultaneous execution of tasks, usually on different processor cores. A program may be concurrent without being parallel—for example, one processor can interleave tasks. It may also use parallelism within a concurrent design.

A process is an operating-system execution environment with its own address space and resources. Processes are relatively isolated. Communication requires mechanisms such as pipes, sockets, shared memory, or message queues.

A thread is an execution path inside a process. Threads in one process normally share its heap and open resources, but each has its own stack and execution state. Shared memory makes communication efficient but creates risks involving inconsistent updates and visibility.

Java and Python execution considerations

Java threads can execute Java code concurrently, subject to JVM implementation and hardware. Java provides strong library support for executors, locks, atomic variables, concurrent collections, and futures.

In CPython, the Global Interpreter Lock (GIL) traditionally restricts simultaneous execution of Python bytecode by multiple threads within one interpreter. Threads remain useful for I/O-bound work because a thread can wait while another runs, but CPU-bound Python code often benefits more from processes or optimized native libraries. Python implementations and versions may differ, so the execution environment must be verified rather than assumed.

4. Race Conditions and the Memory Model

A race condition occurs when program behavior depends on the timing or interleaving of concurrent operations. Consider a shared counter. The operation counter = counter + 1 is conceptually a read, an addition, and a write. If two threads read the same old value before either writes, one update can be lost.

Suppose the initial value is 0:

Two increments occurred, but the final value is 1. The operation is not atomic merely because it appears on one source-code line.

A concurrent program also requires visibility: one thread must eventually observe another thread's updates. It requires ordering: operations must be observed in an order consistent with the synchronization rules. Language memory models define these guarantees. Correctness should not depend on accidental hardware behavior or a particular compiler optimization.

Synchronization mechanisms

Common mechanisms include:

• Mutual exclusion locks, which permit only one thread at a time to enter a critical section.
• Read-write locks, which allow multiple readers but exclude writers.
• Semaphores, which limit access to a fixed number of permits.
• Atomic variables, which provide indivisible operations for selected state changes.
• Concurrent collections, which implement carefully designed coordination internally.
• Message passing, which transfers data rather than exposing shared mutable state.
• Barriers and latches, which coordinate progress between tasks.

A critical section should be small but complete: protecting only one statement may be insufficient if the surrounding invariant requires several operations to occur together.

#### Java example

private final Object lock = new Object();
private int balance = 0;

public void deposit(int amount) {
    if (amount < 0) throw new IllegalArgumentException();
    synchronized (lock) {
        balance += amount;
    }
}

The lock protects the update. In production, a ReentrantLock, an AtomicInteger, or a higher-level concurrent design may be more appropriate depending on the invariant and contention.

#### Python example

from threading import Lock

lock = Lock()
count = 0

def increment():
    global count
    with lock:
        count += 1

The with statement ensures that the lock is released even when an exception occurs.

Deadlock and livelock

A deadlock occurs when tasks wait indefinitely for one another. Four conditions commonly characterize deadlock: mutual exclusion, hold and wait, no pre-emption, and circular wait. Practical prevention includes acquiring multiple locks in a consistent global order, using timeouts, reducing lock scope, and avoiding unnecessary nested locks.

A livelock occurs when tasks remain active but repeatedly react to one another without making progress. Starvation occurs when a task waits indefinitely because other tasks continually receive the resource first. Fair scheduling, bounded waiting, and appropriate queue design can reduce these problems.

5. Futures and Asynchronous Programming

A future is a handle representing a result that may be available later. It separates task submission from result retrieval and can represent success, failure, cancellation, or timeout.

Java example:

ExecutorService pool = Executors.newFixedThreadPool(4);
Future<String> future = pool.submit(() -> loadUserName());

try {
    String name = future.get(2, TimeUnit.SECONDS);
} catch (TimeoutException e) {
    future.cancel(true);
} finally {
    pool.shutdown();
}

Python provides similar facilities through concurrent.futures:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as pool:
    future = pool.submit(load_user_name)
    name = future.result(timeout=2)

A future is not automatically parallel. Its execution depends on the executor, available workers, and workload. Creating too many workers can increase context switching, memory use, contention, and external-service pressure.

Asynchronous programming and event loops

Asynchronous programming models operations that can suspend while waiting. An event loop monitors tasks and resumes those that are ready. This is efficient for workloads containing many independent waits, such as network requests.

Python example:

import asyncio

async def fetch(url):
    ...

async def main(urls):
    results = await asyncio.gather(*(fetch(url) for url in urls))
    return results

await yields control while the operation is waiting. An asynchronous function that performs blocking work directly can freeze the event loop. Blocking database drivers, file operations, or CPU-heavy calculations should be replaced with asynchronous alternatives or moved to an executor when justified.

Asynchronous programming is not a universal replacement for threads. It requires compatible libraries, careful cancellation handling, bounded concurrency, and clear error propagation. It is most useful when tasks spend substantial time waiting and can cooperate with the event loop.

6. Performance Models and Measurements

Performance must be described with measurable quantities:

• Latency: time required for one operation, often measured in milliseconds.
• Throughput: completed operations per unit time, such as requests per second.
• CPU utilization: proportion of available processing capacity in use.
• Memory footprint: memory occupied by the process or selected components.
• Scalability: how performance changes as workload or resources increase.
• Tail latency: high-percentile latency, such as the 95th or 99th percentile.

Average latency can conceal unacceptable delays. A service with a 20 ms average may still have a 900 ms 99th-percentile latency.

For a simple sequential workload, total time can be approximated as:

Here, T_CPU is computation time, T_wait is time waiting for I/O, locks, or external services, and T_overhead includes scheduling, allocation, synchronization, and framework costs. This is a diagnostic model, not a universal exact equation; the components may overlap in concurrent systems.

Amdahl's law estimates an upper bound on speedup when only part of a workload can be parallelized:

S(N) is speedup with N parallel workers, P is the fraction of execution that can be parallelized, and 1 − P is the inherently serial fraction. The model assumes fixed workload, ideal balancing, negligible communication cost, and no additional contention. Real systems generally achieve less speedup.

For example, if P = 0.8 and N = 4, then:

Thus, four workers cannot produce four times the performance because 20% remains serial.

7. Profiling and Safe Optimization

Profiling observes where time and memory are actually used. Useful techniques include:

• CPU sampling, which periodically records active call stacks.
• Instrumentation, which records selected method-entry and method-exit events.
• Allocation profiling, which identifies allocation volume and retaining objects.
• Lock and thread analysis, which reveals contention and blocked states.
• Database and network tracing, which identifies external waiting.
• Load testing, which evaluates behavior under realistic concurrency.

A disciplined optimization process is:

1. Define a measurable performance requirement.
2. Establish a representative baseline.
3. Profile under realistic data and workload.
4. Identify the dominant bottleneck.
5. Change one important factor.
6. Re-measure performance and resource use.
7. Run correctness, concurrency, and regression tests.
8. Document the reason for retaining the optimization.

Common safe improvements include choosing suitable algorithms and data structures, reducing unnecessary allocations, batching I/O, caching stable results with bounded size, indexing database queries, and limiting concurrency to the capacity of downstream systems.

Unsafe or premature optimizations include removing synchronization without proving exclusive ownership, relying on undocumented garbage-collector behavior, increasing thread counts indefinitely, replacing clear code with obscure micro-optimizations, and caching data without an invalidation strategy. Correctness, security, observability, and maintainability are performance constraints, not obstacles to be ignored.

8. Illustrative Case: A Slow Order-Service API

Background

An illustrative online retailer has an order API that becomes slow during promotional events. Average response time is acceptable, but the 99th percentile exceeds the service-level objective. CPU usage is moderate, while database connections are frequently exhausted.

Problem analysis

Profiling shows that each request performs three independent remote lookups sequentially: customer data, inventory data, and shipping estimates. Each request also creates an unbounded number of worker tasks during traffic bursts. A shared in-memory cache is modified by several threads without a clear ownership or synchronization policy.

The system therefore has three problems: avoidable waiting, uncontrolled concurrency, and unsafe shared mutation.

Approach

The team first creates a baseline using a fixed workload and records median and tail latency, throughput, CPU use, memory, and database-pool utilization. It then:

1. Uses bounded asynchronous requests for the three independent lookups.
2. Applies a semaphore so that concurrent inventory requests remain within database capacity.
3. Replaces the shared mutable cache with a thread-safe cache abstraction and an explicit expiration policy.
4. Adds timeouts and cancellation propagation for remote services.
5. Profiles allocations and removes an unnecessary copy of the order representation.
6. Repeats load and failure testing, including partial service outages.

Outcome

The redesign reduces avoidable sequential waiting and prevents request bursts from overwhelming the database. More importantly, it makes capacity limits explicit. The team does not claim success merely from a lower average latency; it verifies tail latency, error rate, memory stability, and correctness under concurrent access.

Learning points

Concurrency should be introduced in response to a measured bottleneck. Independent I/O can often overlap, but downstream capacity must bound that overlap. Thread-safe components do not automatically make a complete multi-step business operation atomic. Finally, performance work must include failure behavior, cancellation, and operational observability.

9. Governance, Security, and Contemporary Practice

Concurrency errors can become security vulnerabilities. A race condition in authorization, payment, inventory, or file handling may permit duplicate actions or inconsistent security decisions. Shared mutable data should therefore be minimized around trust boundaries, and critical operations should be made atomic at the appropriate database or service layer.

Resource exhaustion is also a governance concern. Unbounded task creation, queues, caches, or retries can become denial-of-service conditions. Systems should use limits, back-pressure, timeouts, circuit breakers, and audit logs.

Modern runtimes increasingly provide structured concurrency, virtual threads, task groups, improved garbage collectors, and asynchronous database and networking libraries. These tools can simplify scalable design, but they do not eliminate fundamental issues such as ownership, cancellation, data races, external capacity, and observability. AI-assisted programming tools may suggest concurrent code, but generated code must be reviewed for lock ordering, exception handling, cancellation, data ownership, and benchmark validity.

Best-Practice Checklist

• Prefer immutable values or clear ownership for shared data.
• Protect invariants, not merely individual variables.
• Use the highest-level concurrency abstraction that expresses the requirement.
• Bound workers, queues, retries, caches, and external requests.
• Establish lock-order rules and release resources reliably.
• Treat cancellation, timeouts, and exceptions as normal execution paths.
• Profile representative workloads rather than guessing.
• Measure tail latency and memory stability, not only averages.
• Verify every optimization with correctness and regression tests.
• Document assumptions about runtime, libraries, hardware, and workload.

Chapter Summary

Memory management depends on references, object reachability, mutability, and runtime behavior. Garbage collection reclaims unreachable objects, but it does not release external resources or prevent logical memory leaks. Processes provide isolation, threads share process memory, and asynchronous tasks efficiently coordinate waiting operations. Concurrency introduces risks involving atomicity, visibility, ordering, races, deadlock, starvation, and resource exhaustion. Locks, atomic variables, concurrent collections, message passing, futures, and event loops provide different solutions and trade-offs.

Performance engineering is an empirical discipline. Latency, throughput, utilization, memory footprint, tail behavior, and scalability should be measured under representative workloads. Amdahl's law explains why serial work limits parallel speedup, while profiling identifies the actual bottleneck. Safe optimization preserves correctness, bounds resource use, and is validated through testing and measurement.

Key Terms

Reference: A value used to access an object.

Mutability: The ability of an object to change state after creation.

Immutability: A property whereby an object cannot change after creation.

Heap: Runtime storage commonly used for dynamically allocated objects.

Stack: Per-thread storage for call frames and execution state.

Garbage collection: Automatic reclamation of unreachable objects.

Reachability: The condition that an object can be accessed from a live root.

Process: An isolated operating-system execution environment.

Thread: An execution path within a process.

Concurrency: Overlapping progress of multiple tasks.

Parallelism: Simultaneous execution of tasks.

Race condition: Timing-dependent behavior caused by unsafe concurrent access.

Critical section: Code that accesses shared state and requires coordination.

Mutex: A mutual-exclusion synchronization mechanism.

Deadlock: Indefinite waiting caused by circular resource dependence.

Future: A handle for a result that may become available later.

Event loop: A scheduler that monitors and resumes asynchronous tasks.

Latency: Time required to complete an operation.

Throughput: Completed work per unit of time.

Profiling: Measuring runtime behavior to locate resource costs.

Review Questions

Short Answer Questions

1. Why can assigning one object reference to another create aliasing?
2. Why does garbage collection not eliminate the need to close files and database connections?
3. Distinguish concurrency from parallelism.
4. What is a race condition, and why is a seemingly simple increment potentially unsafe?
5. Why should performance optimization begin with profiling?

Long Answer Questions

1. Compare processes, threads, and asynchronous tasks in terms of memory, communication, isolation, and suitable workloads.
2. Explain reachability-based garbage collection and discuss at least four causes of logical memory leaks.
3. Describe how locks, atomic variables, concurrent collections, and message passing address shared-state problems.
4. Explain futures and event loops, including their treatment of waiting, failure, cancellation, and resource limits.
5. Use Amdahl's law to explain why increasing the number of workers does not normally produce proportional speedup.

Analytical / Application Questions

1. A service performs three independent network calls sequentially. Design a bounded concurrent approach and identify the risks that must be controlled.
2. Two threads update a shared inventory count while a third reads it for authorization. Analyze possible failures and propose a correctness-preserving design.
3. A benchmark shows improved average latency but worse 99th-percentile latency and increased memory use after adding workers. Decide whether to retain the change and justify the measurements and further tests required.

Multiple Choice Questions with Answers

Question 1

Which statement best describes an object reference?

A. It always contains a complete independent copy of the object

B. It provides access to an object and may be copied without copying the object

C. It can only refer to immutable objects

D. It is stored only in the operating-system kernel

Correct Answer: B. It provides access to an object and may be copied without copying the object

Question 2

When is an object generally eligible for garbage collection?

A. Immediately after its constructor returns

B. When its class is no longer imported

C. When it is unreachable from live runtime roots

D. Whenever a method returns

Correct Answer: C. When it is unreachable from live runtime roots

Question 3

What is the principal difference between concurrency and parallelism?

A. Concurrency requires multiple computers, while parallelism requires one

B. Concurrency concerns overlapping progress, while parallelism concerns simultaneous execution

C. Concurrency applies only to I/O, while parallelism applies only to memory

D. They are identical terms in all contexts

Correct Answer: B. Concurrency concerns overlapping progress, while parallelism concerns simultaneous execution

Question 4

Which situation illustrates a race condition?

A. A thread reads an immutable string

B. Two threads update a shared counter through an unsynchronized read-modify-write sequence

C. A process exits after closing its files

D. An event loop awaits a network response

Correct Answer: B. Two threads update a shared counter through an unsynchronized read-modify-write sequence

Question 5

Which technique most directly prevents two threads from entering a critical section at the same time?

A. Mutual-exclusion locking

B. Increasing heap size

C. Object serialization to text

D. Garbage collection

Correct Answer: A. Mutual-exclusion locking

Question 6

For CPU-bound Python code running in traditional CPython, which approach can provide genuine process-level parallelism?

A. Using multiple processes

B. Adding more local variable names

C. Using an immutable string

D. Calling garbage collection more often

Correct Answer: A. Using multiple processes

Question 7

What does a future represent?

A. A guaranteed successful result available immediately

B. A handle for a result that may become available later

C. A permanent operating-system process

D. A lock that prevents all other tasks from running

Correct Answer: B. A handle for a result that may become available later

Question 8

Given P = 0.8 and N = 4, what speedup does the idealized Amdahl model predict?

A. 1.25

B. 2.00

C. 2.50

D. 4.00

Correct Answer: C. 2.50

Question 9

Which practice is most appropriate before applying a performance optimization?

A. Increase all thread pools first

B. Remove synchronization to reduce overhead

C. Measure a representative baseline and profile the workload

D. Assume average latency represents all users

Correct Answer: C. Measure a representative baseline and profile the workload

Question 10

Why should asynchronous programs avoid blocking operations on the event-loop thread?

A. Blocking operations can prevent other ready tasks from progressing

B. Blocking operations always corrupt the heap

C. They automatically disable garbage collection permanently

D. They convert all objects into immutable values

Correct Answer: A. Blocking operations can prevent other ready tasks from progressing