Distributed Systems and Trust Models

Introduction

Blockchain systems are best understood not as isolated databases but as a special class of distributed systems. A distributed system consists of multiple computers that communicate over a network and cooperate to provide a service. These computers may be operated by one organisation, by several known organisations, or by participants who do not fully trust one another. Blockchain technology builds on distributed-systems ideas such as networking, replication, consensus, fault tolerance and cryptographic authentication, while adding a distinctive method for maintaining a shared history without relying entirely on a central administrator.

The central problem is deceptively simple: how can independent computers agree on the order and validity of records when communication is imperfect and some participants may fail or behave dishonestly? In a conventional database, a trusted administrator resolves conflicts. In a blockchain, that responsibility is distributed among network participants according to a protocol and a trust model.

This chapter develops the foundations needed to understand that problem. It examines peer-to-peer networks, data replication, failure models, Byzantine failures, consensus, trust assumptions and the blockchain trilemma. The goal is not to claim that blockchains eliminate trust. Rather, blockchains redistribute trust among software rules, cryptographic mechanisms, economic incentives, hardware, network assumptions and human governance.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the structure and operation of peer-to-peer networks.
2. Distinguish centralised, distributed and decentralised architectures.
3. Describe replication and explain why it improves availability and resilience.
4. Classify crash, omission, network-partition and Byzantine failures.
5. Explain how consensus protocols address disagreement among nodes.
6. Compare permissioned and permissionless trust models.
7. Interpret the blockchain trilemma involving decentralisation, security and scalability.
8. Evaluate blockchain designs in practical organisational and technical contexts.

1. Distributed Systems as the Foundation of Blockchain

In a centralised system, a primary server or authority processes requests and maintains the authoritative database. In a distributed system, processing and data are shared across multiple computers, commonly called nodes. Nodes communicate by sending messages, but there is no guarantee that every message will arrive immediately, arrive in order or arrive at all.

A distributed system must therefore deal with several basic conditions:

• Concurrency: multiple nodes may act at the same time.
• Partial failure: one node may fail while others continue operating.
• Uncertain timing: the network may delay messages unpredictably.
• Inconsistent local views: nodes may temporarily hold different information.
• Coordination costs: agreement requires communication and computational resources.

A distributed system is not automatically decentralised. A company may operate a distributed database across many data centres while retaining complete administrative control. Decentralisation concerns the distribution of authority and control, not merely the number of machines.

Centralisation, distribution and decentralisation

These concepts should be separated carefully:

A system can be distributed but centralised, decentralised but relatively small, or both distributed and decentralised. This distinction is important when evaluating claims about blockchain.

2. Peer-to-Peer Networks

A peer-to-peer (P2P) network allows participating nodes to communicate directly or indirectly without requiring every transaction to pass through a single permanent server. Each peer may provide services to other peers, such as forwarding transactions, storing ledger data or proposing blocks.

In practice, many P2P systems are not perfectly symmetrical. Some nodes have greater bandwidth, storage or processing capacity. Full nodes may maintain complete ledger histories, while lightweight clients request proofs or selected data from other nodes. Nevertheless, the network’s basic communication model is peer-based rather than dependent on one central communication hub.

Network formation and discovery

A new node must discover other nodes. It may use configured addresses, a directory service, DNS seeds, bootstrap peers or previously learned addresses. After joining, it establishes connections and exchanges information about transactions, blocks and network status.

A blockchain node commonly performs the following actions:

1. Connects to one or more peers.
2. Exchanges protocol and software-version information.
3. Requests missing blocks or state data.
4. Validates received transactions and blocks.
5. Relays valid information to selected peers.
6. Rejects invalid, malformed or conflicting messages.

This process creates an information-dissemination pattern known as gossip propagation. A node forwards information to several peers, those peers forward it further, and the information gradually spreads through the network. Gossip is resilient and simple, but it does not guarantee instantaneous or universal delivery.

Network topology

A topology describes how nodes are connected. A star topology has a central hub and is vulnerable to hub failure. A mesh topology provides many alternative paths but requires greater communication and management. Blockchain P2P networks commonly use partially connected mesh-like structures rather than connecting every node to every other node.

The number of connections affects both resilience and cost. Too few connections can isolate a node or make it easier for an attacker to control its view of the network. Too many connections increase bandwidth, memory and processing overhead.

Figure: A partially connected peer-to-peer network in which nodes relay information across multiple paths.

Sybil resistance

In an open P2P network, an attacker may create many false identities. This is called a Sybil attack. Since identity creation may be inexpensive, counting identities is not sufficient to measure influence. Blockchain protocols therefore commonly use a resource-based mechanism, such as computational work, staked economic value or permissioned membership, to limit the influence of false identities.

The principle is important: decentralised consensus generally counts an appropriate scarce resource or authorised voting power, not simply the number of network addresses.

3. Replication and Shared State

Replication is the maintenance of copies of data or state on multiple nodes. Replication supports availability, fault tolerance and local access. If one copy becomes unavailable, another may continue serving users. In blockchain networks, many nodes independently store and validate a common sequence of transactions or a state derived from that sequence.

Replication creates a fundamental trade-off. More copies can improve resilience, but every copy requires storage, bandwidth, verification and operational maintenance. Replicated data also creates a consistency problem: copies must eventually or immediately agree about the accepted state.

Types of consistency

• Strong consistency: after a successful update, later reads observe the update according to a specified ordering guarantee.
• Eventual consistency: if updates stop and communication continues, replicas eventually converge, although temporary differences are permitted.
• Causal consistency: causally related operations are observed in a consistent order, while unrelated operations may be seen differently.
• Blockchain finality: a protocol-specific guarantee that a transaction or block will not be reversed, either probabilistically or deterministically.

Not all blockchains provide the same finality. In a chain using probabilistic finality, a transaction becomes increasingly difficult to reverse as additional blocks are added. In a protocol with deterministic finality, a valid decision becomes final once the protocol reaches a specified commitment, subject to its fault assumptions.

Replication models

In primary-backup replication, one primary node coordinates updates and backups follow it. This is operationally straightforward but creates dependence on the primary. In multi-primary replication, several nodes accept updates, requiring conflict resolution. In state-machine replication, nodes execute the same ordered commands from the same initial state and should therefore reach the same result.

Blockchains are closely related to state-machine replication. Nodes receive transactions, agree on an order or block structure, execute validation rules and derive a common state. The blockchain structure additionally provides an auditable history linked through cryptographic hashes.

4. Failure Models and Fault Tolerance

A failure model states what can go wrong and what the protocol is expected to tolerate. A design cannot be assessed without identifying its assumptions.

Common failure types

A crashed node is usually easier to handle than a malicious node. A crashed node sends no conflicting messages. A Byzantine node may send one statement to one participant and a contradictory statement to another, deliberately attempt to confuse the network or follow the protocol selectively.

Availability and redundancy

If independent components each have availability probability p and a service remains available when at least one of n replicated components is available, the service availability under a simplified independence assumption is:

Here, A_service is the probability that the replicated service is available, p is the availability of each component over the chosen period and n is the number of replicas. The formula assumes identical component availability and statistical independence. Real systems often violate independence because replicas may share software defects, power supplies, networks or administrators.

For example, if three independent replicas each have p = 0.99 availability, then:

A_service = 1 − (0.01)^3 = 1 − 0.000001 = 0.999999.

The apparent improvement is large, but it would be misleading if all replicas could fail because of one common software bug. Redundancy improves resilience only when failures are sufficiently independent.

The CAP perspective

The CAP theorem is often summarised as a choice among consistency, availability and partition tolerance. More precisely, when a network partition occurs, a distributed data service cannot simultaneously guarantee both strong consistency and availability for all requests. Partition tolerance is generally necessary in geographically distributed systems because communication failures can occur; the practical choice is how the system behaves during the partition.

CAP does not state that a system permanently chooses only two of the three properties, nor does it directly classify every blockchain. It is a framework for reasoning about guarantees during partitions. Blockchain protocols may preserve safety, meaning that conflicting states are not accepted, by temporarily reducing liveness, meaning that progress may stop until communication or quorum conditions improve.

5. Byzantine Faults and Consensus

The Byzantine Generals Problem is a model of agreement in which some participants may be traitors and send inconsistent information. The metaphor illustrates the difficulty of reaching a reliable decision when messages may be false and participants cannot assume that every other participant is honest.

A consensus protocol usually seeks properties such as:

• Agreement or safety: honest nodes do not decide conflicting outcomes.
• Validity: an accepted outcome satisfies protocol rules and reflects appropriate proposals.
• Termination or liveness: honest nodes eventually make progress under stated network conditions.
• Fault tolerance: the properties continue to hold despite a specified number or proportion of faulty participants.

These properties can conflict operationally. A protocol that refuses to decide during uncertainty may preserve safety but lose liveness. A protocol that decides quickly with weak verification may preserve liveness while risking inconsistent outcomes.

Byzantine fault tolerance

In classical practical Byzantine fault-tolerant replication, a commonly cited resilience condition is:

Here, n is the total number of known voting replicas and f is the maximum number of Byzantine replicas the protocol is designed to tolerate. This condition applies to particular authenticated, partially synchronous Byzantine agreement models and should not be treated as a universal law for every blockchain protocol.

For example, tolerating f = 1 Byzantine replica requires at least n = 4 replicas under this model. The condition arises from the need to obtain overlapping quorums despite conflicting messages. Other assumptions, such as signatures, synchrony, membership control and the consensus design, can change the precise requirements.

Public blockchains often use different approaches. Proof-of-work links influence to computational expenditure; proof-of-stake links influence to locked economic value and protocol rules. These mechanisms address open membership and Sybil resistance, but they do not make Byzantine behaviour impossible. They make certain attacks costly, detectable or punishable under specified assumptions.

Consensus versus validation

Validation checks whether a transaction or block follows rules: for example, whether a digital signature is valid, whether an account has sufficient balance or whether a block obeys size and format limits. Consensus determines which valid transactions and blocks become part of the authoritative history when multiple valid alternatives exist.

A transaction may therefore be valid in isolation but not included in the final ledger. Conversely, consensus cannot safely make an invalid transaction valid if all honest nodes enforce the same validation rules.

6. Trust Models

Trust is not binary. A system may require trust in different entities and mechanisms to different degrees. A useful analysis asks: Who is authorised to participate? Who can change the rules? What prevents dishonest behaviour? What happens when assumptions fail?

Centralised trust

In a centralised model, users trust an institution to maintain records, authenticate participants, reverse errors and comply with law. This can be efficient and accountable when the institution is competent and regulated. Its risks include concentration of power, single points of failure, censorship and institutional misuse.

Federated or consortium trust

A consortium ledger distributes operation among identified organisations. Membership, governance and liability can be defined contractually. This model is often appropriate where participants know one another but do not want one firm to control the shared record. It can provide higher throughput and privacy than an open public chain, but it still depends on the consortium’s governance and membership processes.

Permissionless economic trust

A permissionless blockchain allows broad participation. Since participants may be pseudonymous, the system relies on cryptographic verification, consensus rules, economic incentives and transparent data. Users trust that the protocol’s assumptions hold and that no coalition can acquire sufficient influence to rewrite history or censor activity.

This is not trustlessness. Users may still trust wallet software, network connectivity, exchanges, developers, hardware, oracles and legal institutions. The more accurate phrase is minimised or redistributed trust.

Trust in oracles and external data

A blockchain can verify data already recorded on its network, but it cannot independently determine whether an external event occurred. An oracle may report temperature, shipment status, exchange rates or legal events. If the oracle is wrong or compromised, the blockchain may immutably preserve incorrect information. Therefore, oracle governance, multiple data sources, dispute procedures and accountability are essential.

7. The Blockchain Trilemma

The blockchain trilemma describes a recurring design tension among decentralisation, security and scalability. It is a heuristic rather than a formal theorem. Different architectures make different trade-offs, and improvements in protocol engineering can shift practical boundaries, but no design obtains unlimited performance, unlimited participation and unlimited adversarial security simultaneously.

• Decentralisation: control and validation are distributed across many independent participants; no small group can easily dominate the system.
• Security: the system resists forgery, censorship, double spending, manipulation and other attacks under its threat model.
• Scalability: the system can process increasing transaction volume with acceptable latency and cost.

Increasing the number of validators may improve decentralisation but increase communication overhead. Requiring every validator to process every transaction may strengthen independent verification but limit throughput. Increasing block size may improve capacity but raise hardware and bandwidth requirements, potentially reducing the number of people able to run full nodes.

A system should therefore specify the relevant dimensions. Scalability may mean transactions per second, confirmation latency, cost per transaction, storage growth or the number of users supported. Security may concern safety, liveness, censorship resistance, privacy or resistance to particular economic attacks.

Layered approaches

Many contemporary systems use layers. A base layer provides settlement and security, while payment channels, rollups, sidechains or application-specific networks process activity elsewhere. Layering can improve throughput, but it introduces new assumptions about bridges, operators, data availability, fraud proofs, validity proofs or withdrawal procedures. Moving activity away from the base layer does not eliminate trust; it changes where trust is placed.

8. Applied Case: A Consortium Food-Traceability Network

The following is an illustrative case, not a report of a particular real-world deployment.

Background

A food distributor works with farms, processors, logistics providers, retailers and regulators. Participants maintain separate databases, and product information is frequently reconciled by email and spreadsheets. When contamination is suspected, identifying affected batches takes several days.

Problem

The participants need a shared record of batch creation, transformation, shipment and receipt. No single organisation is willing to give another complete control over the record. At the same time, product data may include commercially sensitive information, and the system must comply with data-protection obligations.

Analysis

A permissionless public blockchain would provide broad transparency but could expose sensitive business information, impose unnecessary transaction costs and complicate governance. A consortium network is more suitable because participating organisations can be identified and held accountable.

The system might use replicated ledgers operated by a selected set of organisations. Digital signatures authenticate submissions. Consensus among authorised validators orders events. Sensitive documents remain off-chain, while the ledger stores hashes, identifiers and access-controlled references. This design preserves evidence of whether a document has changed without publishing the document itself.

The trust model is not merely technical. Participants must trust the membership authority to admit legitimate organisations, trust validators to follow procedures, and trust data-entry processes to record real events. A blockchain cannot prove that a worker entered the correct batch number or that a sensor was properly calibrated.

Approach and solution

The consortium establishes:

1. A membership and certificate authority.
2. A governance committee defining validation and dispute rules.
3. Replicated validator nodes operated by independent organisations.
4. Digital signatures for every custody event.
5. Access controls separating public, partner and regulator views.
6. An audit process for correcting erroneous entries through compensating records rather than silently rewriting history.
7. A privacy policy governing retention, deletion and disclosure.

Outcome

A retailer can trace a product batch through the recorded chain of custody more rapidly. Regulators can obtain an auditable history, and participants gain a shared reference point. However, the system does not guarantee that every recorded event is factually true. Its benefits arise from improved coordination, tamper evidence, common visibility and accountable governance.

Learning points

• A permissioned blockchain may be more appropriate than a public chain when participants are identifiable.
• Replication improves resilience but creates storage and consistency costs.
• Immutability does not equal truthfulness.
• Governance and data-entry controls are as important as consensus.
• Privacy requirements may favour storing proofs or references rather than raw personal or commercial data.

9. Professional, Ethical and Governance Considerations

Blockchain professionals must evaluate systems beyond technical novelty. A decentralised design can distribute responsibility without clearly assigning accountability. If a protocol causes financial loss, users need to know which operators, developers, custodians or legal entities have duties.

Privacy is also significant. Replicated ledgers may expose transaction patterns, identities or commercially sensitive relationships. Personal data stored on an immutable ledger can conflict with data-minimisation and deletion requirements. Designers should consider pseudonymisation, encryption, off-chain storage, selective disclosure and retention controls, while recognising that encryption keys may be lost or compromised.

Energy use depends on the consensus mechanism and implementation. Proof-of-work can consume substantial electricity because security is tied to computational expenditure. Proof-of-stake generally uses less direct computation but introduces questions about wealth concentration, validator governance, penalties and complex incentive design.

Smart contracts and automated rules may produce irreversible outcomes. Code review, formal testing, access controls, emergency procedures and transparent upgrade governance are essential. Artificial intelligence may assist anomaly detection, node monitoring and fraud analysis, but an AI system does not replace consensus or establish the truth of external data. Its outputs must be governed, auditable and tested for bias and manipulation.

10. Best Practices for Evaluating a Blockchain System

A practical evaluation should begin with the problem rather than the technology. Ask whether a shared database, replicated service or legally governed platform would solve the problem more simply.

For a proposed blockchain, document:

• the assets and events being recorded;
• the participants and their incentives;
• the required privacy and regulatory controls;
• the failure and adversary model;
• the consensus and finality mechanism;
• the meaning of a valid transaction;
• the performance and storage requirements;
• the governance and upgrade process;
• the recovery and dispute procedures;
• the assumptions concerning oracles, wallets, bridges and network access.

Test both normal operation and failure scenarios. Simulate node crashes, delayed messages, malicious validators, key compromise, partition recovery and incorrect external data. Measure not only throughput but also confirmation time, verification cost, storage growth, decentralisation of operators and the consequences of losing a participant.

Chapter Summary

Distributed systems coordinate multiple computers despite concurrency, communication delays and partial failures. Blockchain networks commonly use peer-to-peer communication, gossip propagation, cryptographic authentication, replicated state and consensus protocols to maintain a shared history.

Replication increases availability and resilience but introduces consistency and resource costs. Fault models distinguish crashes and omissions from Byzantine behaviour, in which participants may act arbitrarily or maliciously. Consensus protocols seek agreement, validity, progress and fault tolerance under explicit assumptions. Classical Byzantine replication may require n ≥ 3f + 1 known replicas to tolerate f Byzantine replicas in a particular model, while permissionless systems use mechanisms such as proof-of-work or proof-of-stake to resist Sybil attacks.

Trust is redistributed rather than eliminated. Public, consortium and centralised systems make different assumptions about identity, governance, incentives and accountability. The blockchain trilemma highlights tensions among decentralisation, security and scalability. Practical designs therefore require explicit trade-offs, layered architectures and careful governance. A blockchain can provide shared coordination and tamper evidence, but it cannot guarantee that externally supplied information is true.

Key Terms

Peer-to-peer network: A network in which participating nodes communicate and provide services without relying on one permanent central hub.

Node: A computer or software process participating in a distributed network.

Distributed system: A system whose components operate across multiple networked computers.

Decentralisation: The distribution of authority and control among independent participants.

Replication: Maintaining multiple copies of data or state.

Consistency: The degree to which replicas provide compatible views of data and operations.

Finality: A guarantee that an accepted transaction or state will not be reversed under stated assumptions.

Gossip propagation: Relay of information from node to node across a network.

Sybil attack: An attack using many false identities to gain disproportionate influence.

Fault model: A specification of the failures or adversarial behaviours a system must tolerate.

Byzantine failure: Arbitrary, potentially malicious behaviour by a system participant.

Consensus: A protocol process through which nodes agree on an ordered or accepted result.

State-machine replication: Running the same ordered commands on replicated nodes to derive a common state.

Permissioned blockchain: A blockchain in which participation is restricted or authenticated.

Permissionless blockchain: A blockchain designed for broad participation without pre-approved membership.

Oracle: A mechanism that supplies external information to a blockchain application.

Safety: The property that conflicting or invalid outcomes are not accepted.

Liveness: The property that the system continues to make progress under stated conditions.

Blockchain trilemma: The design tension among decentralisation, security and scalability.

Governance: The rules and institutions through which a system is managed, changed and held accountable.

Review Questions

Short Answer Questions

1. What distinguishes a distributed system from a decentralised system?
2. Why is replication used in blockchain networks?
3. Define a Byzantine failure and contrast it with a crash failure.
4. What is the purpose of a consensus protocol?
5. Why does a blockchain still require trust even when it has no central administrator?

Long Answer Questions

1. Explain how a peer-to-peer blockchain network discovers nodes and propagates transactions and blocks. Discuss the benefits and limitations of gossip propagation.
2. Compare centralised, consortium and permissionless trust models, including their assumptions, strengths and weaknesses.
3. Explain the relationship between replication, consistency, availability and finality in a distributed ledger.
4. Discuss the Byzantine Generals Problem and explain the meaning and limitations of the condition n ≥ 3f + 1.
5. Analyse the blockchain trilemma and explain why increasing block capacity or validator requirements can affect decentralisation and security.

Analytical / Application Questions

1. A consortium of hospitals wants to share audit records but must protect patient confidentiality. Recommend an architecture and trust model, and justify what should be stored on-chain and off-chain.
2. A public blockchain has high transaction demand, rising storage requirements and a small number of operators controlling most validation power. Analyse the trade-offs involved and propose two possible remedies.
3. A supply-chain blockchain records a sensor-reported temperature, but the sensor was later found to be defective. Explain why consensus cannot correct this problem automatically and design an oracle-governance process to reduce recurrence.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes decentralisation from distribution?

A. Distribution concerns multiple computers, while decentralisation concerns how authority is shared.

B. Distribution requires public access, while decentralisation requires private access.

C. Distribution eliminates replication, while decentralisation requires replication.

D. They are identical terms with no meaningful distinction.

Correct Answer: A. Distribution concerns multiple computers, while decentralisation concerns how authority is shared.

Question 2

What is the primary purpose of replication in a distributed system?

A. To guarantee that every external data item is truthful.

B. To improve availability and resilience by maintaining multiple copies.

C. To eliminate all communication between nodes.

D. To ensure that the system has only one authoritative server.

Correct Answer: B. To improve availability and resilience by maintaining multiple copies.

Question 3

Which failure involves a participant sending contradictory or deliberately misleading messages?

A. Crash failure

B. Omission failure

C. Byzantine failure

D. Power-saving failure

Correct Answer: C. Byzantine failure

Question 4

In the simplified availability formula A_service = 1 − (1 − p)^n, what does n represent?

A. The number of replicas

B. The network latency in seconds

C. The number of transactions

D. The probability of one component failing

Correct Answer: A. The number of replicas

Question 5

Under the stated classical Byzantine replication model, how many replicas are required to tolerate f = 2 Byzantine replicas?

A. 3

B. 5

C. 6

D. 7

Correct Answer: D. 7

Question 6

What is the main purpose of a Sybil-resistance mechanism?

A. To compress all ledger data into one file.

B. To prevent false identities from obtaining disproportionate influence.

C. To guarantee privacy of every transaction.

D. To replace transaction validation.

Correct Answer: B. To prevent false identities from obtaining disproportionate influence.

Question 7

Which property means that honest nodes do not accept conflicting outcomes?

A. Liveness

B. Availability

C. Safety

D. Scalability

Correct Answer: C. Safety

Question 8

Why can a blockchain not automatically guarantee that an oracle’s report is correct?

A. Consensus validates protocol messages but cannot independently observe every external event.

B. Blockchains cannot use digital signatures.

C. Oracles are always operated by miners.

D. Replication prevents external data from being recorded.

Correct Answer: A. Consensus validates protocol messages but cannot independently observe every external event.

Question 9

Which option is one dimension of the blockchain trilemma?

A. Anonymity only

B. Scalability

C. Encryption key length

D. Database normalisation

Correct Answer: B. Scalability

Question 10

A permissioned blockchain is generally characterised by which feature?

A. No participant can be identified under any circumstances.

B. Participation or validation is restricted through an authorised membership process.

C. It cannot replicate data.

D. It must use proof-of-work.

Correct Answer: B. Participation or validation is restricted through an authorised membership process.