Concurrency and Asynchronous Programming

Introduction and Context

Modern software rarely performs one activity at a time. A web server may receive many requests, a graphical application must remain responsive while loading data, and a scientific program may calculate independent results simultaneously. Concurrency is the ability to manage multiple activities during overlapping periods. Parallelism is the simultaneous execution of multiple activities, usually on multiple processor cores. A program can be concurrent without being parallel: a single processor can rapidly switch between tasks, creating the appearance of simultaneous progress.

Concurrency improves responsiveness, throughput, and resource utilisation, but it also introduces risks. Multiple execution paths may access shared data in unpredictable orders. Without suitable controls, programs can produce incorrect results, freeze permanently, or behave differently each time they run. This chapter presents the foundations needed to design safe concurrent and asynchronous programs in C++, Java, and Python.

Learning Objectives

After studying this chapter, you should be able to:

• distinguish processes, threads, concurrency, parallelism, and asynchronous execution;
• describe thread life cycles and common scheduling behaviour;
• identify race conditions and explain why they occur;
• use locks, mutexes, synchronized sections, semaphores, and atomic operations appropriately;
• explain deadlock, starvation, livelock, and related hazards;
• use futures, tasks, thread pools, and asynchronous operations;
• implement basic responsive or parallel programs safely;
• select a suitable concurrency strategy and evaluate its limitations.

1. Processes, Threads, and Execution

1.1 Processes

A process is a running program with its own virtual address space, operating-system resources, and execution state. Processes are strongly isolated: one process normally cannot directly read or modify another process’s memory. Communication therefore requires mechanisms such as pipes, files, sockets, shared memory, or message queues.

Process isolation improves reliability and security. If one process crashes, other processes may continue running. However, creating processes and communicating between them generally costs more than creating threads within one process.

1.2 Threads

A thread is an independent path of execution within a process. Threads in the same process share code, heap memory, and many operating-system resources, but each thread has its own stack, program counter, and register state. Shared memory makes communication efficient, but it also creates the possibility of data races.

A process may contain one thread or many threads. For example, a desktop application might use:

• a user-interface thread to process input and redraw windows;
• a worker thread to read a large file;
• a network thread to receive messages;
• a logging thread to write diagnostic information.

1.3 Concurrency and parallelism

Concurrency concerns the structure of a program: several tasks can make progress independently. Parallelism concerns execution: several tasks actually run at the same time. Parallel execution requires appropriate hardware or processor support, although operating systems can provide concurrency on a single core through context switching.

A useful performance model is Amdahl’s law. If a fraction p of a program can be parallelised and the program uses N equally effective processors, the ideal speedup is:

Here, S(N) is speedup, p is the parallel fraction, and N is the number of processors. The formula assumes no communication, scheduling, synchronisation, or memory overhead. It therefore represents an upper bound rather than a guaranteed result. If p = 0.8 and N = 4, then S(4) = 1/(0.2 + 0.8/4) = 2.5. Even four processors cannot produce a fourfold speedup because 20% of the work remains serial.

1.4 Scheduling and nondeterminism

The operating system scheduler selects runnable threads for execution. Scheduling may depend on processor load, priorities, interrupts, and timing. Consequently, the order in which threads execute is usually nondeterministic. A correct concurrent program must remain correct for every permitted ordering, not merely for the ordering observed during testing.

A thread commonly passes through the following states:

1. New: the thread has been created but has not started.
2. Runnable: it is eligible to execute.
3. Running: instructions are currently executing.
4. Blocked or waiting: it is waiting for a lock, input, notification, or time interval.
5. Terminated: execution has completed.

2. Asynchronous Programming and Tasks

Synchronous code waits for an operation to finish before continuing. Asynchronous code starts an operation and allows other work to proceed while the operation is pending. The result can later be obtained through a callback, event, promise, or future.

Asynchronous programming is particularly useful for input/output (I/O), such as network requests and file operations. I/O often spends more time waiting for external devices than using the CPU. A program can therefore remain responsive while the operation is in progress.

A task is a unit of work that may be scheduled independently. A future is an object representing a result that may become available later. A future normally provides operations to:

• test whether the result is ready;
• wait for completion;
• retrieve the result or an exception;
• sometimes cancel the operation.

Asynchronous execution does not automatically mean parallel execution. An event loop can manage many waiting operations on one thread. Conversely, a task submitted to a thread pool may execute in parallel.

Language examples

In C++, std::async can run a callable and return a std::future:

#include <future>

int square(int x) { return x * x; }

int main() {
    std::future<int> result =
        std::async(std::launch::async, square, 12);
    int value = result.get();       // waits if necessary
}

std::launch::async requests asynchronous execution. Calling get() waits for completion and retrieves the value. If the task threw an exception, get() rethrows it in the calling thread.

In Java, ExecutorService and Future provide a common task model:

import java.util.concurrent.*;

ExecutorService pool = Executors.newFixedThreadPool(2);
Future<Integer> result = pool.submit(() -> 12 * 12);

try {
    int value = result.get();       // waits for completion
} finally {
    pool.shutdown();
}

A Java CompletableFuture supports non-blocking composition:

CompletableFuture
    .supplyAsync(() -> loadData())
    .thenApply(data -> transform(data))
    .thenAccept(data -> display(data));

In Python, concurrent.futures supports thread and process pools:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=2) as pool:
    future = pool.submit(lambda: 12 * 12)
    value = future.result()

Python threads are useful for I/O-bound work. In standard CPython implementations, the Global Interpreter Lock (GIL) limits simultaneous execution of Python bytecode in many CPU-bound threaded programs. CPU-intensive work may benefit more from ProcessPoolExecutor, although process creation and data transfer add overhead.

3. Shared State and Race Conditions

A shared resource is data or an external object accessed by more than one execution path. Examples include a bank-account balance, a file, a queue, or a graphical component. Shared state is safe only when all relevant operations are properly coordinated.

A race condition occurs when the result depends on the timing or ordering of concurrent operations. A particularly serious form is a data race, in which two threads access the same memory location concurrently, at least one access is a write, and there is no appropriate synchronisation. In C++, a data race causes undefined behaviour. In Java, it can produce stale or surprising values. In Python, individual implementation-level operations may appear atomic, but compound operations are not generally safe without coordination.

Consider the operation counter = counter + 1. It appears to be one statement, but it usually contains:

1. read the current value;
2. add one;
3. write the new value.

If two threads read 10 before either writes, both may write 11. The expected result after two increments is 12, but the observed result may be 11. This is a lost update.

Critical sections

A critical section is a region that accesses shared state and must not be executed by conflicting threads simultaneously. A mutual-exclusion lock, commonly called a mutex, ensures that at most one thread enters the protected region at a time.

C++:

#include <mutex>

class Counter {
    int value = 0;
    std::mutex lock;
public:
    void increment() {
        std::lock_guard<std::mutex> guard(lock);
        ++value;
    }
    int get() {
        std::lock_guard<std::mutex> guard(lock);
        return value;
    }
};

std::lock_guard follows the acquire-and-release lifetime pattern: it locks on construction and unlocks automatically when leaving scope, including through an exception.

Java:

class Counter {
    private int value = 0;

    public synchronized void increment() {
        value++;
    }

    public synchronized int get() {
        return value;
    }
}

The synchronized keyword provides mutual exclusion and memory-visibility guarantees for the object’s monitor.

Python:

import threading

class Counter:
    def __init__(self):
        self.value = 0
        self.lock = threading.Lock()

    def increment(self):
        with self.lock:
            self.value += 1

The with statement ensures that the lock is released when the block ends.

Atomic operations

An atomic operation appears indivisible to other threads. Atomic variables are appropriate for simple state transitions, counters, flags, and references when the required operation is supported by the language memory model. They do not automatically make a larger algorithm safe.

For example, an atomic counter can safely support increment operations, but the compound rule “if the counter is below a limit, then increment it” may require a lock or a compare-and-exchange loop. Atomicity and visibility are related but distinct concerns: a thread must not only update a value safely, but other threads must also observe updates according to the language’s memory model.

Immutability and message passing

The simplest shared state is state that is not shared. Immutable objects cannot change after construction and can generally be safely read by multiple threads. Message passing avoids direct shared mutation: one worker sends data to another through a thread-safe queue or channel. These approaches often reduce the number of locking decisions and make programs easier to reason about.

4. Synchronisation Mechanisms

4.1 Mutexes and monitors

A mutex protects a critical section. A monitor combines shared data, the operations that access it, and a lock controlling those operations. Java’s synchronized objects are monitor-like. Locks should be held for the shortest practical period, because long critical sections reduce concurrency and increase waiting.

4.2 Read-write locks

A read-write lock permits multiple concurrent readers but normally allows only one writer. It is useful when reads are frequent, writes are infrequent, and the protected data structure supports safe concurrent reading. It may be counterproductive when writes are common or when lock-management overhead is high.

4.3 Semaphores

A semaphore maintains a count representing available permits. A binary semaphore has two logical states, while a counting semaphore can control access to a limited number of resources, such as database connections. Unlike a mutex, a semaphore is primarily a permit-counting mechanism and may not require the releasing thread to be the acquiring thread, depending on the library.

4.4 Condition variables

A condition variable allows a thread to sleep until a condition involving shared state becomes true. The condition must always be checked in a loop because a notification does not itself prove that the condition is true.

Conceptually:

lock mutex
while queue is empty:
    wait on condition variable
remove item from queue
unlock mutex

Waiting atomically releases the associated lock and reacquires it before returning. The loop handles spurious wake-ups and competing consumers.

4.5 Barriers and latches

A barrier makes a group of threads wait until all members reach a defined point. A latch is usually a one-way countdown: threads wait until a count reaches zero. These tools are useful in staged computations, testing, and coordinated startup.

5. Deadlock and Other Liveness Problems

A deadlock occurs when a set of threads waits indefinitely because each thread holds a resource needed by another. Four conditions are traditionally associated with deadlock:

• mutual exclusion: a resource cannot be used by multiple owners simultaneously;
• hold and wait: a thread holds one resource while waiting for another;
• no pre-emption: resources cannot be forcibly taken away;
• circular wait: a cycle of dependencies exists.

For example, Thread A holds lock 1 and waits for lock 2, while Thread B holds lock 2 and waits for lock 1.

A robust prevention strategy is to impose a global lock order. If every thread acquires lock 1 before lock 2, circular wait cannot arise from those locks. Other strategies include acquiring multiple locks atomically, reducing lock scope, avoiding nested locks, using timeouts, and preferring immutable data or message passing.

A livelock occurs when threads remain active but repeatedly respond to one another without making progress. Starvation occurs when a thread waits indefinitely because other threads repeatedly receive the resource or scheduling opportunity. Fair locks, bounded waiting, suitable scheduling, and careful queue design can reduce starvation.

Deadlock detection can represent threads and resources as a wait-for graph. A cycle indicates a potential deadlock. Prevention is usually preferable because detection may occur only after the program has stopped progressing.

6. Memory Visibility and Ordering

Concurrency correctness involves more than mutual exclusion. A thread may temporarily retain values in processor registers or caches. A language memory model defines when a write by one thread becomes visible to another and which instruction reorderings are permitted.

A happens-before relationship establishes that one action is guaranteed to be observed before another action. Lock release followed by a later acquisition of the same lock commonly establishes such an ordering. Thread creation and joining also establish important visibility relationships. Volatile or atomic facilities may provide visibility, but their precise guarantees differ by language.

A join operation waits for a thread to finish. It is useful both for completion control and for ensuring that the joining thread observes the completed thread’s results according to the language rules. Programs should use documented concurrency primitives rather than relying on timing, sleep calls, or accidental machine behaviour.

7. Thread Pools and Task Decomposition

Creating a new thread for every small task can be expensive and may exhaust system resources. A thread pool creates a bounded set of reusable worker threads. Tasks are placed in a queue and workers take tasks as they become available.

Thread pools provide:

• bounded concurrency;
• lower thread-creation overhead;
• centralised shutdown and monitoring;
• a convenient separation between task submission and task execution.

However, a pool can become saturated. A task that waits for another task submitted to the same small pool can cause a pool deadlock. Blocking operations should be limited, and pool sizes should reflect whether tasks are CPU-bound or I/O-bound.

For CPU-bound work, a rough starting point is a number of workers near the number of available processor cores, followed by measurement. For I/O-bound work, more workers may be useful because many workers spend time waiting, but excessive workers increase memory use and context switching. These are heuristics, not universal formulas.

Good task decomposition follows several principles:

1. make tasks sufficiently large to justify scheduling overhead;
2. avoid unnecessary sharing between tasks;
3. define ownership of each result clearly;
4. propagate exceptions and cancellation explicitly;
5. combine partial results only through safe operations;
6. measure rather than assume that parallelism improves performance.

8. Worked Example: A Thread-Safe Work Queue

A producer-consumer queue allows producer threads to add work and consumer threads to remove it. The queue, mutex, and condition variable form one synchronised abstraction. A producer waits when the queue is full; a consumer waits when it is empty.

Illustrative C++ design:

#include <condition_variable>
#include <mutex>
#include <queue>

class WorkQueue {
    std::queue<int> items;
    std::mutex m;
    std::condition_variable notEmpty;
    bool closed = false;

public:
    void push(int item) {
        {
            std::lock_guard<std::mutex> g(m);
            if (closed) throw std::runtime_error("queue closed");
            items.push(item);
        }
        notEmpty.notify_one();
    }

    bool pop(int& item) {
        std::unique_lock<std::mutex> lock(m);
        notEmpty.wait(lock, [this] {
            return closed || !items.empty();
        });
        if (items.empty()) return false;
        item = items.front();
        items.pop();
        return true;
    }

    void close() {
        {
            std::lock_guard<std::mutex> g(m);
            closed = true;
        }
        notEmpty.notify_all();
    }
};

The queue’s invariant is that only code holding m may inspect or modify items and closed. The consumer waits in a predicate-based loop. Closing the queue wakes all consumers so they can terminate rather than wait forever. This design demonstrates mutual exclusion, condition variables, explicit shutdown, and ownership of shared state.

9. Applied Case: Responsive Document Application

Background

Consider an illustrative desktop document editor that opens large files, checks spelling, saves documents, and displays progress. In an initial implementation, all operations run on the user-interface thread.

Problem

Opening a large document freezes the interface. Users cannot scroll or cancel the operation, and the application appears to have crashed. Moving all work to background threads is not sufficient: concurrent updates to the document and user-interface widgets could corrupt state or violate GUI-toolkit rules.

Analysis

The user interface requires short, predictable operations. File reading and spell checking are I/O-bound or CPU-intensive and may take substantial time. The document model is shared, while most GUI frameworks require widgets to be accessed only from the interface thread.

Approach and solution

A suitable design uses:

1. an interface thread that receives input and renders widgets;
2. a bounded worker pool for file and spell-check tasks;
3. immutable snapshots of document text sent to workers;
4. futures or callbacks for completion and failure reporting;
5. cancellation tokens checked periodically by long operations;
6. result messages placed in a thread-safe queue;
7. interface-thread handlers that apply validated results.

The worker never directly modifies GUI objects. It processes a snapshot and returns a result containing a version number. When the result arrives, the interface applies it only if its version is still current. This prevents an older, slower task from overwriting newer user edits.

Outcome

The interface remains responsive, background work is bounded, and stale results are rejected. Exceptions are delivered through the future rather than silently lost. Shutdown first stops accepting work, requests cancellation, waits for workers, and then releases resources.

Learning points

Responsiveness requires architectural separation, not merely additional threads. Shared mutable objects should be minimised. Results need ownership, versioning, and error-handling policies. Cancellation is cooperative in many task frameworks; a program must design operations to observe cancellation rather than assume that a running task can always be forcibly stopped.

10. Testing, Debugging, and Professional Practice

Concurrent failures are often intermittent. A program may pass thousands of tests and fail under a different processor schedule. Professional testing should therefore combine ordinary functional tests with stress, load, and timing-variation tests.

Useful practices include:

• use race detectors, thread sanitizers, static analysis, and lock-analysis tools;
• record thread identities, task identifiers, lock events, and timestamps;
• test shutdown, cancellation, timeouts, and exception paths;
• avoid logging that changes timing so substantially that it hides the defect;
• create deterministic tests with barriers or controlled executors where possible;
• verify invariants, not merely final output;
• document which thread owns each resource and which locks protect each field.

Concurrency can also create security vulnerabilities. Incorrect synchronisation may bypass access controls, expose partially updated credentials, corrupt audit records, or permit denial of service through unbounded task submission. Systems handling personal, financial, or safety-critical data should apply least privilege, bounded queues, authenticated communication, secure logging, and formal review of shared-state assumptions.

11. Contemporary Developments and Limitations

Modern platforms increasingly support structured concurrency, where child tasks have clear lifetimes and failures propagate to a supervising scope. This reduces “orphan” background tasks and makes shutdown more predictable. Actor systems, reactive streams, asynchronous generators, and coroutine frameworks provide alternatives to manually managing threads.

Coroutines are lightweight units that suspend at defined points and resume later. They are particularly effective for asynchronous I/O, but they do not automatically make CPU-heavy work parallel. A coroutine scheduler may still use one thread unless tasks are explicitly distributed across workers.

Artificial-intelligence systems also rely heavily on concurrency: data pipelines load batches while accelerators compute, inference servers handle many requests, and distributed workers exchange results. The same principles remain applicable: define ownership, bound queues, propagate failures, protect shared state, and measure end-to-end behaviour. AI libraries may hide threads or processes, so developers must understand their execution and resource-management policies.

Concurrency has unavoidable costs. Locks cause contention; communication causes overhead; parallel tasks may require duplicated data; debugging is difficult; and more workers can reduce performance through context switching. A sequential program that is correct and fast enough is often preferable to a concurrent program whose complexity is not justified.

Best-Practice Checklist

• Prefer immutable data, local state, and message passing where practical.
• Protect every shared mutable invariant with a clearly documented mechanism.
• Keep critical sections short and avoid calling unknown code while holding a lock.
• Establish and document a global lock-acquisition order.
• Use thread pools rather than unbounded thread creation.
• Treat futures as carriers of both results and exceptions.
• Design cancellation and shutdown before implementing background work.
• Never update thread-confined user-interface objects from worker threads.
• Do not use arbitrary sleeps as synchronisation.
• Test under load and use tools that detect races and deadlocks.
• Measure performance and verify that parallelism provides a real benefit.

Chapter Summary

Concurrency enables several tasks to make progress during overlapping periods, while parallelism executes tasks simultaneously. Processes provide isolation; threads provide efficient shared-memory execution but require careful synchronisation. Asynchronous programming separates the start of an operation from the later delivery of its result, commonly through tasks, futures, callbacks, or event loops.

Race conditions arise when correctness depends on unpredictable timing. Mutexes, synchronized sections, atomic variables, semaphores, condition variables, immutable data, and message passing are tools for controlling shared state. Deadlock results from circular resource waiting and can be reduced through lock ordering, reduced lock scope, and avoidance of nested locks. Starvation and livelock are additional liveness problems.

Thread pools and structured task decomposition provide practical control over resource use. Safe programs also require visibility guarantees, explicit ownership, exception propagation, cancellation, bounded queues, and disciplined shutdown. Concurrency should be introduced when responsiveness, throughput, or parallel computation justifies its complexity, and its behaviour should be validated with testing, measurement, and specialised tools.

Key Terms

Process: An executing program with its own address space and operating-system resources.

Thread: An execution path within a process.

Concurrency: The management of multiple potentially overlapping activities.

Parallelism: Simultaneous execution of multiple activities.

Asynchronous execution: Starting work without requiring the caller to wait immediately for completion.

Task: A schedulable unit of work.

Future: An object representing a result that may become available later.

Race condition: A defect or behaviour whose result depends on execution timing.

Data race: Unsynchronised concurrent access to the same memory location when at least one access writes.

Critical section: Code that accesses shared state and must be protected.

Mutex: A mutual-exclusion lock allowing one owner at a time.

Atomic operation: An operation that appears indivisible to other threads.

Semaphore: A synchronisation object that controls a count of permits.

Condition variable: A mechanism for waiting until a shared-state condition may be true.

Deadlock: Indefinite waiting caused by cyclic resource dependencies.

Starvation: Indefinite denial of a thread’s opportunity to proceed.

Livelock: Repeated activity without meaningful progress.

Thread pool: A reusable, bounded collection of worker threads.

Memory model: Rules governing visibility, ordering, and permitted behaviour of concurrent memory operations.

Structured concurrency: A model in which concurrent task lifetimes and failure propagation follow explicit scopes.

Review Questions

Short Answer Questions

1. Distinguish a process from a thread.
2. Why is asynchronous execution not necessarily parallel execution?
3. What is a race condition? Give one example.
4. State two ways to reduce the risk of deadlock.
5. What purpose does a future serve?

Long Answer Questions

1. Explain the differences among concurrency, parallelism, and asynchronous programming, including appropriate practical examples.
2. Describe how mutexes, atomic variables, semaphores, and condition variables solve different synchronisation problems.
3. Explain the four conditions associated with deadlock and discuss prevention techniques.
4. Compare thread pools with creating a new thread for every task.
5. Design a safe producer-consumer system and explain its queue, shutdown, and exception-handling policies.

Analytical / Application Questions

1. Two threads update a shared account balance using a read-modify-write operation. Explain how a lost update can occur and propose two correct designs.
2. A desktop application freezes while downloading a file. Design an asynchronous solution that preserves interface responsiveness, handles failure, and supports cancellation.
3. A program uses two locks, and some threads acquire them in the order A–B while others use B–A. Analyse the liveness risk and propose a corrected design.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes concurrency from parallelism?

A. Concurrency requires multiple processors, while parallelism requires one processor.

B. Concurrency manages overlapping activities, while parallelism executes activities simultaneously.

C. Concurrency applies only to processes, while parallelism applies only to threads.

D. Concurrency is always faster than sequential execution, while parallelism is not.

Correct Answer: B. Concurrency manages overlapping activities, while parallelism executes activities simultaneously.

Question 2

What is the primary purpose of a mutex?

A. To create a new process

B. To guarantee that only one thread enters a protected critical section at a time

C. To increase the number of processor cores

D. To convert synchronous code into network code

Correct Answer: B. To guarantee that only one thread enters a protected critical section at a time

Question 3

Why can two increments of a shared counter produce only one net increment?

A. The processor permanently deletes one instruction.

B. Both threads may read the same old value and later overwrite one another’s result.

C. A mutex automatically reverses one increment.

D. A future can contain only one integer.

Correct Answer: B. Both threads may read the same old value and later overwrite one another’s result.

Question 4

Which condition is associated with deadlock?

A. All resources are immutable.

B. Every thread releases locks before acquiring any lock.

C. A circular wait exists among threads holding resources.

D. Tasks are executed sequentially.

Correct Answer: C. A circular wait exists among threads holding resources.

Question 5

What does a future normally represent?

A. A result that may become available later

B. A permanently terminated process

C. A lock that can never be released

D. A compiler warning

Correct Answer: A. A result that may become available later

Question 6

Which approach commonly reduces shared-state race conditions?

A. Using arbitrary sleep calls

B. Using immutable data or message passing

C. Increasing the number of unbounded threads

D. Ignoring exceptions from worker tasks

Correct Answer: B. Using immutable data or message passing

Question 7

If 80% of a program is parallelisable and four processors are used, the ideal Amdahl speedup is approximately:

A. 1.25

B. 2.00

C. 2.50

D. 4.80

Correct Answer: C. 2.50

Question 8

Why should a condition variable usually be waited on in a loop?

A. To ensure the thread creates another process

B. To handle spurious wake-ups and recheck the shared condition

C. To prevent all other threads from running permanently

D. To make a mutex unnecessary

Correct Answer: B. To handle spurious wake-ups and recheck the shared condition

Question 9

What is a major advantage of a bounded thread pool?

A. It guarantees that every task is parallelised.

B. It prevents all possible deadlocks.

C. It limits concurrency and avoids unbounded thread creation.

D. It removes the need for exception handling.

Correct Answer: C. It limits concurrency and avoids unbounded thread creation.

Question 10

In a responsive graphical application, why should a worker usually avoid directly modifying interface widgets?

A. Widgets are always stored on disk.

B. Many GUI frameworks confine widget access to the interface thread.

C. Worker threads cannot calculate results.

D. Interface widgets cannot display text.

Correct Answer: B. Many GUI frameworks confine widget access to the interface thread.