11. Dependability, Security, Privacy and Safety Engineering

Introduction and Context

Software-intensive systems increasingly control financial transactions, medical devices, transportation, energy infrastructure, industrial production, public services and personal data. In such environments, functional correctness alone is insufficient. A system may provide its specified functions and still be unacceptable if it is frequently unavailable, vulnerable to attack, unsafe in operation, or careless with personal information.

This chapter examines four closely related concerns: dependability, security, privacy and safety. Dependability is the ability of a system to deliver trusted service. Security protects systems and information against unauthorized access, misuse, alteration and disruption. Privacy concerns the appropriate collection, use, disclosure and retention of information about identifiable individuals. Safety is the prevention of unacceptable harm to people, property or the environment.

These properties overlap but are not interchangeable. A system can be secure but unsafe, such as a locked industrial controller that issues an unsafe command. It can be safe but lack privacy, as when a monitoring system prevents physical injury while collecting excessive personal data. Engineering them requires more than testing at the end of development. It requires risk analysis, disciplined architecture, secure development practices, assurance evidence, operational monitoring and governance throughout the system life cycle.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish dependability, availability, reliability, resilience, security, privacy and safety.

2. Apply threat modeling and risk-based engineering to software systems.

3. Explain secure software development life-cycle practices and security controls.

4. Design fault-tolerant and fail-safe mechanisms.

5. Use privacy-by-design principles and data-protection reasoning.

6. Construct and evaluate assurance arguments and evidence.

7. Interpret the role of standards, regulations and organizational governance.

8. Analyze trade-offs through realistic system scenarios.

Dependability as a System Property

Dependability is commonly understood as the ability of a system to provide services that can justifiably be trusted. It is a composite property rather than a single measurable feature. The principal attributes are:

• Availability: readiness of the system for correct service when required.
• Reliability: continuity of correct service over a specified period and environment.
• Maintainability: ability to be repaired, modified or restored efficiently.
• Safety: absence of unacceptable risk of physical or environmental harm.
• Security: resistance to unauthorized access, use, disclosure, alteration or destruction.
• Resilience: ability to withstand, adapt to and recover from disruptive events.

Reliability and availability are related but distinct. A system that fails rarely may still have poor availability if each failure requires a long restoration period. A system can also be highly available while producing incorrect results; availability therefore does not imply correctness. Dependability assessment must state the service being considered, the operating environment, the time period and the consequences of failure.

A useful conceptual model views failures through a chain of events. A fault is a defect or abnormal condition, such as corrupted memory or a design error. A fault may produce an error, an incorrect internal state. If the error reaches the system boundary and affects service, it becomes a failure. Fault avoidance attempts to prevent faults; fault removal finds and eliminates them; fault tolerance limits their effects; and failure forecasting estimates their likelihood and consequences.

Quantitative measures support engineering decisions. If a system operates correctly for mean time between failures (MTBF) and requires mean time to repair (MTTR), a simplified steady-state availability estimate is:

`Availability = MTBF / (MTBF + MTTR)`

This formula is useful for reasoning but does not capture correlated failures, maintenance logistics, degraded service or cyberattacks. Operational definitions and evidence are essential.

Security, Privacy and Safety: Different Questions

Security asks: Who may do what, under which conditions, and how can unauthorized behavior be prevented or detected? Its familiar objectives are confidentiality, integrity and availability, often called the CIA triad. Authentication establishes identity; authorization determines permitted actions; accountability and non-repudiation support traceability.

Privacy asks: What personal or sensitive information is collected, why, how is it used, with whom is it shared, and how long is it retained? Privacy is not merely confidentiality. Information may be securely stored yet collected without a legitimate purpose or used beyond the expectations of the data subject. Privacy engineering therefore addresses purpose limitation, data minimization, transparency, user control, accuracy, retention and rights of access or deletion, subject to applicable law.

Safety asks: Could the system contribute to unacceptable harm, and what controls prevent or limit that harm? Safety analysis considers hazards, hazardous events, severity, likelihood, exposure and controllability. A safety requirement may demand that a system enter a controlled state when sensor values become unreliable. Security controls may be needed because an attacker could deliberately create the same condition. This intersection is especially important in cyber-physical systems, medical devices, vehicles and industrial automation.

Risk-Based Engineering and Assurance

Risk is generally evaluated through the combination of the likelihood of an adverse event and its consequence. In practice, qualitative categories such as low, medium and high are often used, while safety-critical and security-critical systems may require more formal quantitative analysis. Risk assessment should not be confused with risk acceptance: identifying a risk does not justify leaving it untreated.

A practical risk process includes:

1. Define system boundaries, assets, stakeholders and assumptions.

2. Identify hazards, threats, vulnerabilities and possible failure modes.

3. Estimate likelihood, impact and uncertainty.

4. Select treatments: avoid, reduce, transfer, monitor or accept the risk.

5. Assign owners and verification criteria.

6. Monitor residual risk and reassess after changes or incidents.

Security threat modeling provides a structured form of this process. The team identifies assets and trust boundaries, maps data flows, considers attacker capabilities and records threats. STRIDE is one useful mnemonic: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege. It is not a complete method, but it helps teams ask systematic questions. Attack trees represent how an attacker might achieve a goal; misuse cases describe malicious or undesirable interactions; abuse cases extend ordinary use-case analysis.

Safety analysis uses complementary methods. Failure Mode and Effects Analysis (FMEA) examines component failure modes and their effects. Fault-tree analysis starts with a hazardous top event and reasons backward using logical combinations of causes. Hazard and Operability Study (HAZOP) applies guide words such as “more,” “less,” “none” and “reverse” to discover deviations from intended operation. Preliminary hazard analysis is useful early, while detailed analyses become more precise as architecture and interfaces are defined.

Assurance is the reasoned confidence that a system satisfies its claims. An assurance case normally contains a top-level claim, subclaims, context, assumptions, evidence and a logical argument linking evidence to claims. For example, a claim that an infusion pump prevents overdose may be supported by requirements traceability, hazard analysis, design inspection, test results, formal verification of dosage limits and operational procedures. Testing alone rarely establishes a convincing assurance case because tests sample behavior while hazards may concern rare combinations of conditions.

Secure Development Across the Life Cycle

Security is most economical and effective when integrated from requirements through retirement. A secure development life cycle typically includes the following activities.

Security requirements. Requirements identify assets, trust boundaries, authentication needs, authorization rules, cryptographic expectations, logging, recovery objectives and regulatory obligations. They should be testable. “The system shall be secure” is not a useful requirement; “administrative actions shall require multifactor authentication and be recorded with synchronized timestamps” is more actionable.

Architecture and design. Designers apply least privilege, separation of duties, secure defaults, defense in depth, complete mediation and fail-safe behavior. Threat modeling is performed before implementation and revisited after major architectural changes. Interfaces, dependencies and privileged components receive particular scrutiny.

Implementation. Secure coding practices address input validation, output encoding, memory safety, error handling, secrets management, dependency control and safe concurrency. Static analysis, software composition analysis and peer review can identify defects early. Cryptographic algorithms should be selected from established, reviewed standards rather than invented for a project.

Verification. Security testing includes unit and integration tests, fuzzing, abuse-case tests, penetration testing, configuration review and, where justified, formal methods. Tests should verify both prevention and detection. For example, a system should reject unauthorized access and generate an actionable audit event.

Deployment and operation. Secure configuration, patch management, vulnerability disclosure, backups, incident response and monitoring are operational responsibilities. A secure design can be undermined by default passwords, excessive privileges or unpatched components.

Retirement. Decommissioning must address data deletion, credential revocation, remaining interfaces, software dependencies and contractual obligations. Retirement is part of the security and privacy life cycle, not an administrative afterthought.

Fault Tolerance, Resilience and Safe Failure

Fault tolerance uses redundancy and controlled behavior to preserve or restore service when faults occur. Redundancy may be spatial, such as replicated servers, or temporal, such as retrying an operation. It may also involve diverse implementations to reduce common-mode failures. Replication without independence is misleading: three identical instances can fail together because of the same software defect or cloud-region outage.

Common mechanisms include:

• Checks and monitoring: detect invalid states and abnormal behavior.
• Timeouts and circuit breakers: prevent indefinite waiting and cascading failure.
• Retries with backoff: recover from transient faults without creating overload.
• Bulkheads: isolate resource pools so one failure does not exhaust the entire system.
• Graceful degradation: provide reduced but safe functionality.
• Checkpointing and recovery: restore a consistent state after interruption.
• Fail-safe design: move to a state that minimizes harm when control is lost.
• Fail-secure design: deny unauthorized access when security decisions cannot be made.

These mechanisms involve trade-offs. Retrying a payment request can cause duplicate charges if operations are not idempotent. Automatic failover can spread corrupted data. A fail-safe state for a railway signal may differ from a fail-safe state for an online service. Safety and availability objectives must therefore be analyzed together rather than optimized independently.

Resilience extends beyond component reliability. It includes preparation, resistance, response, recovery and learning. Disaster recovery plans, dependency mapping, recovery-time objectives, recovery-point objectives and exercises convert resilience from an aspiration into an operational capability. Chaos experiments can reveal hidden dependencies, but they must be authorized, bounded and stopped when safety conditions are threatened.

Privacy by Design and Data Governance

Privacy by design embeds privacy considerations into architecture rather than adding a notice after implementation. A design team should first determine whether personal data is necessary. If it is, the team should minimize collection, separate identifiers from operational data, restrict access, protect data in transit and at rest, define retention periods and document processing purposes.

Privacy-enhancing techniques include pseudonymization, anonymization where re-identification risk is acceptably low, aggregation, access logging, differential privacy in statistical release, federated analysis and selective disclosure. No technique is universally adequate. Pseudonymized data may remain personal data if re-identification is reasonably possible, and anonymization claims must consider auxiliary datasets and inference attacks.

A privacy impact assessment examines affected individuals, data flows, purposes, risks, controls and residual impacts. It should include vulnerable groups and secondary uses, not merely the original business objective. Consent is not a universal solution: it must be meaningful, informed and revocable where applicable, and some processing requires another lawful basis under the governing legal framework.

Organizations must establish accountability through policies, records of processing, role-based access, breach response, supplier oversight, retention schedules and staff training. Legal requirements vary by jurisdiction; therefore, engineers should consult applicable frameworks such as data-protection laws, sectoral regulations and contractual obligations rather than assuming that one global rule applies everywhere.

Standards, Regulations and Professional Governance

Standards provide shared vocabulary, practices and evidence expectations. ISO/IEC 27001 addresses information-security management systems; ISO/IEC 27002 provides security-control guidance; ISO/IEC 27005 supports information-security risk management. ISO/IEC 25010 offers a quality model including reliability, security and other software quality characteristics. IEC 61508 provides a general functional-safety framework for electrical, electronic and programmable electronic systems, while domain standards such as ISO 26262 address road-vehicle functional safety. NIST frameworks and guidance are widely used for cybersecurity risk management and secure software development.

Compliance is not identical to security or safety. Certification or an audit may show that specified processes and controls exist, but it does not guarantee that every vulnerability or hazard has been eliminated. Professional judgment remains necessary. Engineers have duties to report material risks, preserve evidence, protect confidential information, avoid deceptive assurance claims and prioritize human welfare over schedule pressure.

Governance assigns responsibility. A risk register without owners is ineffective; a security team without authority cannot enforce controls; and a safety assessor who is not sufficiently independent may overlook organizational pressure. Important decisions should be documented, reviewable and traceable from stakeholder need to requirement, design, verification and operational evidence.

Illustrative Case Study: Connected Infusion Pump Service

Background

A hospital deploys network-connected infusion pumps managed through a central service. Clinicians configure dosage limits, the pump records administration events, and maintenance personnel receive diagnostic alerts. The system processes patient identifiers and medication information while controlling a safety-critical device.

Situation and Problem

The hospital wants remote monitoring and automatic software updates. However, an attacker might alter dosage settings, a network outage might interrupt communication, and excessive data collection could expose patients. A software update that improves security could also introduce a defect affecting dose calculation.

Analysis

The assets include patient safety, dosage integrity, pump availability, credentials, audit records and patient privacy. Trust boundaries exist between the pump, hospital network, cloud service, clinician workstation and maintenance channel. Threat modeling identifies spoofed clinician accounts, tampered commands, denial of service and compromised update packages. Safety analysis identifies hazards such as over-infusion, under-infusion and loss of alarm functionality. Privacy analysis examines identifiers, retention and access by vendors.

The team distinguishes local safety functions from network convenience functions. A pump must enforce dosage limits locally and continue safe operation if the cloud service is unavailable. Network commands require strong authentication, authorization based on clinical role, integrity protection and an auditable record. Updates require signed packages, verification before installation, rollback capability and staged deployment.

Approach and Solution

The architecture uses least privilege, mutually authenticated channels, hardware-backed key storage and segmented hospital networks. Clinicians receive only the permissions required for their wards and duties. The pump rejects commands outside locally stored safety limits. Communication failure produces a defined local mode rather than an uncontrolled stop. Duplicate command protection and idempotent protocols prevent repeated administration instructions.

FMEA and fault-tree analysis are performed for overdose and loss-of-alarm hazards. Security threat models are reviewed whenever interfaces change. Patient identifiers are minimized in operational telemetry, and access to detailed records is logged. Updates are tested on representative devices, deployed to a pilot group, monitored and capable of rapid rollback. An assurance case links safety claims to hazard controls, verification results, operational procedures and staff training.

Outcome

The resulting design does not eliminate all risk, but it reduces the likelihood and impact of foreseeable failures and attacks. The hospital can continue essential local operation during network disruption, detect anomalous administration commands, investigate access events and control software changes. Residual risks are recorded with owners and reviewed by clinical, engineering, security and privacy representatives.

Learning Points

The case illustrates that security, privacy, safety and availability must be engineered together. Network connectivity is not automatically beneficial; it changes the threat and hazard landscape. Local enforcement of critical safety constraints, controlled updates, independent assurance evidence and explicit operational responsibilities are more valuable than a general claim that the system is “secure.”

Challenges and Common Mistakes

Several recurring errors weaken dependability engineering. Teams may treat compliance checklists as substitutes for analysis, assume that encryption solves privacy, or rely on penetration testing to discover design flaws. They may define availability targets without identifying the service users actually require, or deploy redundancy without examining common dependencies.

Other mistakes include over-collecting telemetry, retaining logs indefinitely, granting broad administrative privileges, failing to test recovery, ignoring insider threats, and allowing emergency changes to bypass review permanently. Safety cases can become paperwork exercises when evidence is disconnected from claims. Security controls can also reduce availability or usability if introduced without operational analysis.

Uncertainty deserves explicit treatment. Threats evolve, components are replaced, assumptions become invalid and users behave differently from the design model. Continuous monitoring, incident learning, independent review and periodic reassessment are therefore essential. Automation should support expert judgment, not conceal unresolved trade-offs.

Current Developments

Cloud-native systems, software supply chains, edge computing and artificial intelligence are changing assurance practice. Infrastructure is increasingly ephemeral, so identity, policy and observability must replace assumptions based on fixed network location. Software bills of materials help organizations identify vulnerable dependencies. Signed provenance and reproducible builds improve confidence in what is deployed.

AI-enabled systems introduce concerns about data leakage, model manipulation, bias, unpredictable behavior and inadequate explanations. Their assurance requires evaluation data, monitoring for drift, human oversight and clearly bounded use. Digital twins and automated testing can improve safety analysis, but the model itself must be validated. Autonomous systems require explicit handling of uncertainty, degraded modes and human intervention.

Best-Practice Checklist

A mature engineering organization should:

• define dependability, security, privacy and safety objectives in measurable terms;
• maintain current asset inventories, data-flow diagrams and dependency maps;
• perform threat, hazard and privacy analyses early and after major changes;
• trace requirements to architecture, implementation, tests and assurance evidence;
• apply least privilege, secure defaults, defense in depth and fail-safe behavior;
• design for recovery, graceful degradation and operational observability;
• test backups, incident response, failover and update rollback in realistic conditions;
• minimize personal data and establish purpose, retention and access controls;
• manage third-party and open-source dependencies throughout their life cycle;
• assign accountable owners and independently review high-consequence decisions.

Chapter Summary

Dependability engineering seeks trusted service through reliability, availability, maintainability, resilience, security and safety. Security protects systems and information from unauthorized action; privacy governs responsible processing of personal information; safety limits unacceptable harm. These concerns intersect but require distinct analyses and controls.

Risk-based engineering combines threat modeling, hazard analysis, privacy impact assessment and assurance arguments. Secure development integrates security requirements, architecture, coding, verification, deployment, operations and retirement. Fault tolerance uses detection, redundancy, recovery and controlled degradation, while safe and secure failure states depend on context. Standards and regulations provide useful structures, but compliance does not replace professional judgment or evidence. Effective engineering is continuous, multidisciplinary and accountable.

Key Terms

Dependability: The ability of a system to deliver trusted service.

Availability: Readiness to provide correct service when required.

Reliability: Continuity of correct service over a specified time and environment.

Maintainability: Ease and speed with which a system can be repaired or modified.

Resilience: Ability to withstand, adapt to and recover from disruption.

Fault: A defect or abnormal condition that can produce an incorrect state.

Error: An incorrect internal state caused by a fault.

Failure: Inability of a system to provide its specified service.

Threat modeling: Structured identification and analysis of potential malicious actions.

Vulnerability: A weakness that can be exploited or contribute to harm.

Risk: Combination of the likelihood and consequence of an adverse event.

Fault tolerance: Limitation of fault effects so that service continues or recovers.

Fail-safe: Behavior that moves toward a state minimizing harm when control is lost.

Privacy by design: Incorporation of privacy protection throughout system design.

Pseudonymization: Processing that separates direct identifiers while retaining possible linkage.

Assurance case: Structured claims, arguments, evidence and assumptions supporting confidence in a system.

FMEA: Analysis of failure modes and their effects.

Fault-tree analysis: Top-down analysis of causes leading to an undesired event.

Defense in depth: Use of multiple independent or layered protective controls.

Least privilege: Granting only the access necessary for an authorized task.

Review Questions

Short Answer Questions

1. Distinguish reliability from availability.

2. What is the difference between a fault, an error and a failure?

3. Why is privacy broader than confidentiality?

4. State four objectives of a secure development life cycle.

5. What is the purpose of an assurance case?

Long Answer Questions

1. Explain the relationship and differences among dependability, security, privacy and safety.

2. Describe a risk-based threat and hazard analysis process for a software-intensive system.

3. Discuss fault tolerance mechanisms and the trade-offs associated with redundancy and failover.

4. Explain how privacy-by-design principles influence architecture, data management and operations.

5. Evaluate the role and limitations of standards and regulations in assuring dependable systems.

Analytical / Application Questions

1. A banking service is highly available but occasionally authorizes duplicate payments during retries. Analyze the dependability and security issues and propose controls.

2. Design a preliminary threat model and safety strategy for an autonomous warehouse vehicle that operates near workers.

3. A hospital wants to use detailed patient telemetry to train an AI model. Analyze the privacy, security, safety and governance questions that should be resolved before deployment.

22. Multiple Choice Questions with Answers

Question 1

Which statement most accurately distinguishes reliability from availability?

A. Reliability concerns readiness at a particular instant, while availability concerns service continuity over time.

B. Reliability concerns continuity of correct service, while availability also depends on restoration time.

C. Reliability applies only to hardware, while availability applies only to software.

D. Reliability and availability are identical measures with different names.

Correct Answer: B. Reliability concerns continuity of correct service, while availability also depends on restoration time.

Question 2

A replicated service has three identical instances, all dependent on the same vulnerable library. What major dependability concern remains?

A. The system has excessive maintainability.

B. The replicas create a common-mode failure.

C. The system cannot use monitoring.

D. Redundancy automatically eliminates the vulnerability.

Correct Answer: B. The replicas create a common-mode failure.

Question 3

Which activity is most directly associated with STRIDE threat modeling?

A. Estimating mean time to repair

B. Identifying spoofing, tampering and elevation-of-privilege threats

C. Calculating software maintainability

D. Selecting a disaster-recovery location

Correct Answer: B. Identifying spoofing, tampering and elevation-of-privilege threats

Question 4

Why does encryption alone not guarantee privacy?

A. Encryption cannot protect data in transit.

B. Encrypted data can never be accessed by authorized users.

C. Privacy also concerns purpose, collection, retention, use and disclosure.

D. Encryption applies only to safety-critical systems.

Correct Answer: C. Privacy also concerns purpose, collection, retention, use and disclosure.

Question 5

A payment request is automatically retried after a timeout and results in two charges. Which design property would most directly reduce this risk?

A. Idempotency

B. Pseudonymization

C. Data minimization

D. Fault-tree analysis

Correct Answer: A. Idempotency

Question 6

In an assurance case, what connects evidence such as test results to a top-level safety claim?

A. A logical argument with supporting subclaims and assumptions

B. A list of unprioritized vulnerabilities

C. A database backup policy

D. A user-interface prototype

Correct Answer: A. A logical argument with supporting subclaims and assumptions

Question 7

Which design is most consistent with fail-safe behavior in a safety-critical controller?

A. Continue issuing commands using stale sensor data indefinitely.

B. Disable all logging during abnormal conditions.

C. Move to a controlled state that minimizes foreseeable harm when control is lost.

D. Grant emergency access to every user.

Correct Answer: C. Move to a controlled state that minimizes foreseeable harm when control is lost.

Question 8

What is a principal purpose of a privacy impact assessment?

A. To maximize the amount of personal data collected

B. To examine data processing, affected individuals, risks and controls

C. To replace all security testing

D. To prove that a system has no residual risk

Correct Answer: B. To examine data processing, affected individuals, risks and controls

Question 9

Which statement best describes compliance with a security standard?

A. It guarantees that no vulnerability exists.

B. It makes threat modeling unnecessary.

C. It can provide structured controls and evidence but does not replace engineering judgment.

D. It applies only after a system is retired.

Correct Answer: C. It can provide structured controls and evidence but does not replace engineering judgment.

Question 10

In the infusion-pump case, why should critical dosage limits be enforced locally?

A. To ensure safe constraints remain effective during network disruption or cloud compromise

B. To eliminate the need for authentication.

C. To allow unrestricted remote commands.

D. To increase the collection of patient identifiers.

Correct Answer: A. To ensure safe constraints remain effective during network disruption or cloud compromise