Emerging Architectures and Capstone Design

Introduction

Big data engineering is moving beyond the traditional separation between data collection, storage, processing, analytics, and governance. Modern systems increasingly place computation near data sources, combine batch and streaming semantics, expose governed data products across organizational boundaries, protect workloads in use, and automate operational decisions. These developments are not isolated technologies. Together, they represent a shift from pipeline construction toward the design of adaptive, governed, economically measurable data platforms.

This chapter examines seven closely related developments: edge analytics, real-time lakehouses, unified batch-stream processing, data sharing, confidential computing, autonomous operations, and AI-assisted engineering. Each can create significant value, but each also introduces trade-offs involving consistency, latency, cost, security, explainability, skills, and operational complexity.

The chapter concludes with a capstone design method. The objective is not to select the newest technology, but to defend an end-to-end architecture against explicit requirements and measurable outcomes.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose, operating model, and limitations of emerging big data architectures.
2. Compare edge, cloud, lakehouse, streaming, sharing, and confidential-computing patterns.
3. Design a unified batch-stream architecture with appropriate correctness and latency guarantees.
4. Evaluate governance, security, ethical, regulatory, and economic consequences.
5. Use measurable service-level objectives and value metrics to defend architectural decisions.
6. Critically assess AI-generated engineering artefacts and autonomous operational actions.
7. Produce and defend an end-to-end capstone architecture.

1. From Data Pipelines to Data Systems

A conventional data platform often consists of operational databases, extract-transform-load jobs, a warehouse, dashboards, and machine-learning services. This model remains useful, but it becomes inadequate when data must be processed in milliseconds, when devices are intermittently connected, when data cannot be moved freely across jurisdictions, or when analytical and operational workloads must use the same facts.

An emerging architecture treats data as a continuously changing, governed product. Its principal concerns are:

• Location: where data is generated, stored, and processed;
• Time: how quickly data must be observed and acted upon;
• Semantics: whether events, corrections, duplicates, and late records are handled consistently;
• Trust: whether users can verify provenance, quality, access, and transformation history;
• Economics: whether the value of a decision exceeds the total cost of producing it;
• Adaptation: whether the platform can detect and respond to changing workloads and failures.

A useful conceptual distinction is between data plane and control plane. The data plane transports, stores, and computes over data. The control plane defines policies, schemas, identities, quality rules, deployment processes, observability, and permitted actions. Emerging architectures are successful only when both planes are designed together.

2. Edge Analytics and Distributed Intelligence

2.1 Definition and motivation

Edge analytics performs some data processing near the point at which data is generated: a sensor, vehicle, factory, mobile device, retail store, or telecommunications site. The edge may filter, aggregate, classify, compress, or act on data before transmitting selected information to a central platform.

The principal motivations are low latency, reduced network use, resilience during disconnection, privacy, and local autonomy. For example, a manufacturing controller may stop a machine immediately when vibration indicates a dangerous condition. Sending the raw vibration stream to a distant cloud service may be too slow or unreliable for that decision.

Edge processing should not be confused with simply installing a small server at a remote location. It is an architectural allocation of computation and authority. Designers must decide which decisions are local, which are centralized, and how local decisions are reconciled with central policy.

2.2 Processing patterns

Common edge patterns include:

• Filtering: transmit only records satisfying a threshold or rule.
• Windowed aggregation: send counts, averages, percentiles, or extrema for a time interval.
• Feature extraction: transform raw signals into features used by a model.
• Local inference: run a trained model without sending raw data elsewhere.
• Store-and-forward: retain data locally during outages and transmit it later.
• Hierarchical analytics: process at device, site, regional, and cloud levels.

A key danger is irreversible reduction. If an edge device sends only an average, later analysts cannot recover an unusual peak. Retention policies should therefore distinguish between data required for immediate control and data required for audit, diagnosis, or future model development.

2.3 Trade-offs

Edge analytics reduces network traffic and response time, but increases fleet-management complexity. Devices may have limited CPU, memory, storage, and power. Software versions can diverge, clocks can drift, and physical compromise may be possible. Model updates require signing, staged rollout, rollback, and monitoring for local data drift.

The latency budget can be represented as:

Here, L_total is end-to-end latency, while the component terms represent capture, queueing, computation, network transfer, and actuation latency. The terms must use the same unit, normally milliseconds. This additive model is an approximation: parallel operations, batching, and overlapping communication can change the exact calculation. It is most useful for identifying where a latency target is being consumed.

An edge design is justified when the value or safety consequence of faster local action exceeds the additional cost and operational risk. It is not justified merely because edge computing is fashionable.

3. Real-Time Lakehouses

3.1 Concept

A lakehouse combines the flexible, economical storage model associated with data lakes with the transactional management, schema control, performance, and governance expected of warehouses. A real-time lakehouse extends this idea to continuously arriving data and low-latency queries.

Its logical layers commonly include:

1. Ingestion: events, files, database changes, and device messages.
2. Raw or bronze layer: minimally altered, replayable data.
3. Validated or silver layer: standardized, deduplicated, and quality-checked data.
4. Curated or gold layer: business entities, aggregates, features, and data products.
5. Serving layer: dashboards, APIs, operational applications, and machine-learning systems.
6. Governance layer: identity, cataloguing, lineage, quality, retention, and policy enforcement.

The layers are conceptual rather than mandatory physical databases. A well-designed platform avoids unnecessary copying and defines which layer is authoritative for each use case.

3.2 Transactionality and freshness

Real-time lakehouses must reconcile frequent writes, concurrent readers, schema evolution, updates, deletions, and replay. Table formats and storage engines may provide atomic commits, snapshot isolation, versioning, and partition or clustering mechanisms. These capabilities support reproducibility: an analyst can identify the table version used for a result rather than querying an unstable collection of files.

Freshness is not identical to latency. Latency measures the time from event generation to availability or action. Freshness describes how current a dataset is relative to its source. A table can have low query latency but poor freshness if ingestion is delayed; conversely, a fresh table may be expensive to query.

A useful freshness measure for an observation at time t is:

where t_event is the event time represented by the newest accepted data. The units must be consistent, such as seconds. For a service objective, designers may specify a percentile rather than an average, for example, “99% of records have age below 60 seconds.” Percentiles are preferable when occasional long delays matter operationally.

3.3 Design risks

A real-time lakehouse can become a costly “everything platform” if teams ingest data without ownership, retention limits, or access purpose. Small-file proliferation, excessive compaction, uncontrolled schema changes, and high-frequency metadata operations can degrade performance. Streaming ingestion does not automatically create real-time value; the consumer must also query or act at the required frequency.

4. Unified Batch-Stream Processing

4.1 The unification problem

Historically, organizations built separate batch and streaming pipelines. Batch systems are convenient for complete recomputation and large historical transformations. Streaming systems support continuous processing but require difficult decisions about state, time, late events, and recovery. Separate implementations often produce semantic divergence: the dashboard and the monthly report calculate “revenue” differently.

Unified processing seeks one logical transformation model that can operate over bounded data, which has a known end, and unbounded data, which continues indefinitely. The objective is not necessarily one physical job. It is consistent business semantics across modes.

4.2 Time semantics

Three notions of time must be distinguished:

• Event time: when an event actually occurred;
• Processing time: when the platform processed the event;
• Ingestion time: when the platform received the event.

Event-time processing is usually preferable for analytical correctness because network delay should not change the period to which an event belongs. However, it requires watermarks and policies for late data.

A window partitions events into intervals. For a fixed window of width w, an event with timestamp t_e may be assigned to a window according to a defined boundary rule. The rule must specify whether endpoints are inclusive and how time zones and daylight-saving changes are handled.

A watermark is an estimate that events earlier than a particular event-time threshold are unlikely to arrive. It is not proof that no later record will arrive. Late-event handling may include correction of prior aggregates, side outputs for review, or a bounded lateness policy.

4.3 Correctness and delivery

At-most-once processing may lose records but avoids duplicates. At-least-once processing can retry records but may produce duplicates unless operations are idempotent or deduplicated. Exactly-once claims must be interpreted carefully: they generally refer to a defined scope involving state updates and sink commits, not to the entire external world.

For a record with unique identifier id, an idempotent sink ensures that repeated processing produces the same final state:

This expression is a property, not a universal implementation. It requires a stable identifier, deterministic interpretation, and a sink capable of conditional insert, upsert, or transactional handling.

A robust unified pipeline specifies:

• event identity and deduplication rules;
• ordering requirements and acceptable disorder;
• watermark generation;
• lateness tolerance;
• replay and backfill procedures;
• state checkpointing and recovery;
• correction and retraction semantics;
• data-quality responses.

5. Data Sharing and Data Products

Data sharing enables another team, organization, supplier, regulator, or customer to use data without receiving uncontrolled copies of the underlying storage. Sharing may occur through governed views, APIs, secure exchange, data clean rooms, marketplace products, or federated query mechanisms.

The data-product approach assigns ownership for a useful, discoverable data asset. A product should have a purpose, producer, schema, quality indicators, documentation, service-level expectations, access policy, and lifecycle. “A table exists” is not sufficient evidence that a data product exists.

Important sharing models include:

• Copy-based exchange: simple and broadly compatible, but creates duplication and stale copies.
• Access-based sharing: consumers query a governed source, reducing duplication but creating dependency on availability and policy enforcement.
• Federated analysis: data remains in different domains while computation is coordinated; this is useful for sovereignty but technically demanding.
• Clean-room analysis: parties compute approved statistics or intersections without revealing raw records; leakage risks still require careful query controls.

Data sharing requires contracts covering purpose limitation, permitted fields, retention, accuracy responsibilities, incident notification, intellectual property, and deletion. Privacy-preserving methods such as aggregation, masking, tokenization, differential privacy, or secure multiparty computation have different utility and protection characteristics. No method should be described as anonymous without considering linkage attacks and auxiliary information.

6. Confidential Computing

Encryption traditionally protects data at rest and in transit. Confidential computing aims to protect data while it is being processed by using hardware-supported trusted execution environments or related mechanisms. Code and data execute within an isolated environment, and remote attestation can allow a party to verify that an approved workload is running before releasing secrets.

A confidential-computing design must identify its threat model. It may reduce exposure to a hostile cloud operator or an untrusted host operating system, but it does not automatically protect against malicious application code, compromised credentials, side-channel attacks, denial of service, or incorrect outputs. Attestation evidence must be validated, keys must be released conditionally, and software supply chains must be controlled.

Confidential computing can support cross-organization analytics in which parties are unwilling to reveal raw data. The performance cost may be significant because of memory constraints, encryption overhead, limited accelerator support, and more difficult debugging. A sound design compares this cost with the legal, commercial, or security value of reducing exposure.

7. Autonomous Operations

Autonomous operations apply detection, diagnosis, prediction, and automated remediation to platform management. Examples include scaling workers, restarting failed tasks, changing query resources, quarantining malformed data, or rerouting traffic.

A useful control loop contains:

1. Observation: metrics, logs, traces, quality signals, and business indicators;
2. Analysis: anomaly detection, causal investigation, or policy evaluation;
3. Decision: selection of an action under constraints;
4. Execution: controlled change to infrastructure or data flow;
5. Verification: confirmation that the action improved the target without unacceptable side effects.

Automation should be graduated. Low-risk actions, such as restarting an idempotent worker, may be fully automatic. High-impact actions, such as deleting data, changing access rights, or publishing a model, should require approval or dual control. Every action requires an audit record, rollback mechanism, scope limit, and emergency disablement.

A control system must avoid optimizing a proxy at the expense of the real objective. For example, scaling down to reduce cost may violate freshness objectives. Operational objectives should therefore be expressed together, such as availability, freshness, error rate, security, and cost.

8. AI-Assisted Engineering

Generative AI can assist with schema documentation, SQL drafts, test generation, code migration, incident summarization, query optimization suggestions, and infrastructure templates. It can increase productivity, particularly for repetitive tasks, but generated output is not evidence of correctness.

An engineering workflow should include:

• approved models and data-handling rules;
• prohibition of confidential data in unapproved prompts;
• repository and dependency scanning;
• deterministic tests and representative test data;
• human review for security, correctness, and performance;
• provenance of generated artefacts;
• evaluation against known failure cases;
• monitoring after deployment.

AI assistants may hallucinate APIs, omit edge cases, reproduce insecure patterns, or encode bias from training material. In big data systems, subtle errors can be especially damaging: a wrong time-zone conversion, join condition, deduplication key, or access predicate can produce plausible but false results. AI should therefore be treated as a junior collaborator whose work requires evidence-based review, not as an autonomous authority.

9. Governance, Ethics, and Regulation

Emerging architectures increase the number of locations and actors involved in data processing. Governance must therefore be designed as an operating capability rather than a document stored after implementation.

Core controls include:

• data classification and purpose limitation;
• identity, least privilege, and separation of duties;
• encryption and key management;
• catalogues, lineage, and provenance;
• retention and defensible deletion;
• schema and policy change management;
• quality measurement and issue ownership;
• incident response and notification;
• model risk management and explainability;
• jurisdiction and cross-border transfer controls.

Ethical evaluation asks not only whether processing is legal, but whether it is proportionate, fair, understandable, and contestable. Edge cameras, location streams, and behavioral analytics may create surveillance risks. Automated decisions may disadvantage groups if data reflects historical inequalities. A data-sharing arrangement may be technically secure yet socially unacceptable if individuals lack meaningful choice or if the purpose expands beyond the original expectation.

Governance should be measurable. Examples include the percentage of critical datasets with owners, the proportion of sensitive fields protected by policy, lineage coverage for regulated reports, access-review completion, unresolved quality incidents, and time to revoke access.

10. Economic Evaluation and Architectural Trade-offs

Technical architecture is also an economic decision. Total cost of ownership includes infrastructure, storage, network transfer, software, support, security, compliance, engineering time, and the cost of failure.

A simple annualized value model is:

All terms must be expressed in the same monetary period and currency. Benefit may include additional revenue, avoided loss, reduced labor, or improved safety. Risk cost is an expected cost estimate and should not be treated as a precise prediction.

For a recurring decision, return on investment may be expressed as:

ROI is dimensionless, usually reported as a percentage. It is meaningful only when the benefit and cost boundaries, time horizon, discounting assumptions, and counterfactual are explicit. For long-lived investments, discounted cash flow is more appropriate than an undiscounted ratio.

Architectural trade-offs commonly include:

11. Capstone Design Method

A defensible capstone architecture begins with requirements rather than products. The following method can be used for an academic project or professional design review.

Step 1: Define the decision and users

State who needs which decision, at what frequency, and with what consequence. “Real time” is inadequate; specify a latency target, freshness percentile, availability target, and error tolerance.

Step 2: Characterize data

Describe sources, volume, velocity, variety, sensitivity, ownership, event-time behavior, retention, and legal constraints. Identify authoritative sources and expected data-quality failures.

Step 3: Allocate processing locations

For each operation, decide whether it belongs on the device, at the site, in a regional platform, or in a central cloud. Keep safety-critical or latency-critical actions close to the source, while retaining centrally governed records needed for audit and learning.

Step 4: Specify semantics

Define identifiers, schemas, time zones, windows, watermarks, deduplication, corrections, replay, and delivery guarantees. A diagram without these semantics is not a complete streaming design.

Step 5: Design the lakehouse and serving paths

Separate raw replayable data from validated and curated products. Define transactional boundaries, partitioning or clustering, query paths, APIs, feature stores, and operational sinks. Avoid building multiple conflicting definitions of key measures.

Step 6: Add governance and security

Map classifications to controls. Specify identity, authorization, key management, lineage, retention, consent or lawful basis, audit, and incident response. If data crosses organizations or jurisdictions, define contractual and technical boundaries.

Step 7: Engineer failure and recovery

List failures involving devices, networks, brokers, storage, schemas, models, credentials, and downstream systems. For each, specify detection, containment, retry, replay, rollback, and human escalation.

Step 8: Define measurable value

Use technical and business metrics. A balanced scorecard may include:

• p95 and p99 end-to-end latency;
• percentage of records meeting freshness objectives;
• duplicate, missing, and late-event rates;
• query success and availability;
• cost per million events or per analytical query;
• incident detection and recovery time;
• decision accuracy and false-action rate;
• energy use where sustainability is material;
• revenue uplift, loss avoided, or labor hours saved.

Metrics require baselines and measurement methods. A target such as “reduce latency by 50%” is incomplete without identifying the baseline population, percentile, workload, and measurement point.

Step 9: Defend alternatives

Present at least one credible alternative. Explain why the selected architecture is preferable under the stated requirements, and identify conditions under which the rejected alternative would be better. This demonstrates engineering judgment rather than technology advocacy.

12. Illustrative Case: Cold-Chain Logistics

Background

A fictional pharmaceutical distributor transports temperature-sensitive products in refrigerated vehicles. Sensors produce temperature, location, door-state, and battery events. A shipment may be damaged if temperature remains outside its permitted range. The distributor needs immediate local alerts, central compliance reporting, predictive maintenance, and controlled sharing with manufacturers and regulators.

Problem

The existing system uploads sensor data every fifteen minutes. It loses records during network outages, produces inconsistent daily reports, and cannot reliably distinguish a late event from a current event. Raw location and health data also require strict access control.

Analysis

The safety decision requires seconds, while compliance reporting requires durable and reconstructable history. Therefore, a single processing location is unsuitable. The design must support local action, store-and-forward behavior, event-time correction, and governed external sharing.

Approach

Each vehicle runs an authenticated edge agent. It validates sensor messages, maintains a bounded local buffer, detects immediate temperature excursions, and records the rule version used for each alert. It sends signed events and local summaries when connectivity is available. The central platform receives events through a durable ingestion service and stores immutable raw records in a lakehouse bronze layer.

A stream processor performs schema validation, deduplication using sensor-event identifiers, unit normalization, and event-time windows. Watermarks allow a defined lateness interval. Late corrections update silver and gold tables and are marked in compliance reports. A batch recomputation can rebuild aggregates from the bronze layer after a rule change or prolonged outage.

The gold layer exposes shipment status, excursion intervals, route performance, and equipment-health features. A governed sharing interface gives manufacturers only the fields and shipments authorized by contract. Regulators receive signed reports with lineage to source events. Sensitive driver information is separated from shipment analytics and protected through role-based access.

A confidential execution environment is considered for joint manufacturer analysis of failure patterns. It is adopted only for the specific collaborative workload because the additional performance and operational cost is not justified for ordinary internal reporting.

Autonomous operations may scale ingestion workers and quarantine malformed messages automatically. An excursion-triggered refrigeration shutdown is not fully automated until safety validation is complete; the initial design raises an alert and requires an approved operational procedure.

Outcome and measures

The architecture is evaluated against a baseline rather than an unqualified claim of success. Proposed targets are: p99 alert latency below 5 seconds while connected; preservation of all accepted sensor events after reconnection; less than 0.1% unresolved duplicate events; 99.9% availability for central ingestion; and complete lineage for regulatory reports. Business measures include reduced product loss, fewer manual investigations, and faster release of compliant shipments.

Learning points

The case illustrates several principles. Edge analytics is appropriate for urgent local decisions, but central storage remains essential for audit and recomputation. Event-time semantics are necessary when connectivity is intermittent. Sharing is a governed product, not a file transfer. Confidential computing should be selected for a defined threat model. Finally, automation must be proportionate to the consequences of error.

13. Limitations and Contemporary Directions

No emerging architecture eliminates fundamental constraints. The speed of light, network capacity, finite storage, imperfect sensors, changing regulations, and limited human attention remain relevant. Distributed systems can improve availability in one dimension while increasing consistency and operational complexity in another.

Current directions include semantic layers that standardize business meaning across engines, lakehouse-native vector and feature workloads, stream processing with increasingly strong transactional guarantees, privacy-enhancing computation, carbon-aware scheduling, and platform engineering based on reusable internal products. These developments should be evaluated empirically. Benchmark results may not transfer across workloads, and vendor terminology often hides important differences in semantics and cost.

Research and professional practice should therefore emphasize reproducible evaluation: representative workloads, stated data distributions, failure injection, security testing, cost measurement, and comparison with a credible baseline. A prototype that works in a demonstration is not evidence of production readiness.

Best-Practice Checklist

1. Start with decisions, users, constraints, and measurable objectives.
2. Use edge processing only where latency, resilience, privacy, or bandwidth justifies it.
3. Preserve sufficient raw data for audit, replay, and correction.
4. Define event-time, lateness, deduplication, and correction semantics explicitly.
5. Treat data products as owned services with contracts and quality indicators.
6. Match confidential computing to a documented threat model.
7. Graduate automation according to the consequence of incorrect action.
8. Review AI-generated code and infrastructure with tests, security analysis, and human accountability.
9. Measure total cost and business value, not only throughput.
10. Defend the chosen design against at least one credible alternative.

Chapter Summary

Emerging big data architectures distribute computation, governance, and decision authority across devices, streams, lakehouses, organizations, and protected execution environments. Edge analytics can reduce latency and network dependence, but it increases fleet and consistency challenges. Real-time lakehouses combine governed analytical storage with continuous ingestion, while unified batch-stream processing seeks consistent semantics across bounded and unbounded data.

Data sharing requires products, contracts, access controls, and privacy-aware exchange models. Confidential computing protects data during processing under a defined threat model, but does not remove all security risks. Autonomous operations can improve resilience when bounded by policy, verification, audit, and rollback. AI-assisted engineering can improve productivity only when generated artefacts are tested and reviewed.

A capstone architecture must connect requirements to processing locations, data semantics, storage, serving, governance, security, failure recovery, cost, and measurable value. The strongest design is not the one containing the most technologies; it is the one that makes assumptions explicit and demonstrates why its trade-offs are acceptable.

Key Terms

Edge analytics: Processing data near its source to reduce latency, bandwidth use, or central dependence.

Real-time lakehouse: A governed analytical storage architecture that supports continuous ingestion and low-latency use.

Data plane: The part of a system that transports, stores, and processes operational data.

Control plane: The part that manages policy, identity, configuration, deployment, and governance.

Event time: The time at which an event actually occurred.

Processing time: The time at which a system processes an event.

Watermark: An estimate of the event-time progress of a stream.

Late event: A record arriving after the system’s expected processing boundary.

Idempotence: A property in which repeated application of the same operation has the same final effect as one application.

Exactly-once processing: A bounded processing guarantee whose precise scope must be explicitly defined.

Data product: A governed, owned, documented, and reusable data asset.

Data contract: An agreement describing schema, meaning, quality, access, and change expectations.

Confidential computing: Protection of data during processing using isolated execution and related mechanisms.

Remote attestation: Evidence that a specified software configuration is running in a trusted environment.

Autonomous operations: Automated observation, decision, and remediation of platform conditions.

Human-in-the-loop: A control arrangement requiring human review for selected decisions or actions.

Lineage: A record of data origins, transformations, dependencies, and destinations.

Freshness: A measure of how current available data is relative to its source or event time.

Total cost of ownership: The full cost of acquiring, operating, governing, securing, and retiring a system.

Service-level objective: A measurable target for reliability, latency, freshness, quality, or another service property.

Review Questions

Short Answer Questions

1. Why might an architecture retain raw edge data rather than transmit only aggregates?
2. Distinguish event time from processing time.
3. What is the purpose of a watermark?
4. Why is remote attestation important in confidential computing?
5. Name two controls needed for safe autonomous operations.

Long Answer Questions

1. Compare edge analytics and centralized cloud analytics, including latency, governance, resilience, security, and cost.
2. Explain how a real-time lakehouse can support both operational dashboards and historical recomputation.
3. Discuss why exactly-once processing claims must be scoped carefully.
4. Evaluate the benefits and risks of data products and governed data sharing.
5. Develop a governance model for AI-assisted engineering in a regulated organization.

Analytical / Application Questions

1. Design an edge and cloud architecture for a smart-grid fault-detection system. Specify latency, event-time, replay, security, and human-escalation requirements.
2. A streaming platform has low average latency but poor p99 latency and frequent late events. Diagnose possible causes and propose measurements and architectural remedies.
3. Defend whether confidential computing and autonomous remediation should be used in a cross-company fraud-detection platform. State the threat model, value assumptions, controls, and conditions for adoption.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly justifies processing analytics at the edge?

A. The need to reduce response latency when network connectivity is uncertain

B. The desire to centralize all raw data immediately

C. The need to eliminate device management

D. The requirement that every query use historical data only

Correct Answer: A. The need to reduce response latency when network connectivity is uncertain

Question 2

In a distributed event-processing system, event time refers to:

A. The time a record is written to the database

B. The time a record is processed by a worker

C. The time the underlying event occurred

D. The time a dashboard is refreshed

Correct Answer: C. The time the underlying event occurred

Question 3

What is the primary purpose of a watermark?

A. To encrypt events before storage

B. To estimate stream progress and manage late events

C. To replace schema validation

D. To guarantee that no future event will arrive late

Correct Answer: B. To estimate stream progress and manage late events

Question 4

Which property helps a sink avoid changing its final state when the same event is retried?

A. Partitioning

B. Compression

C. Idempotence

D. Sampling

Correct Answer: C. Idempotence

Question 5

Which statement about confidential computing is most accurate?

A. It protects data during processing under a defined threat model

B. It guarantees that application code cannot be malicious

C. It eliminates the need for key management

D. It prevents all denial-of-service attacks

Correct Answer: A. It protects data during processing under a defined threat model

Question 6

Which is an appropriate control for high-impact autonomous remediation?

A. Removing all audit logs to improve performance

B. Allowing unrestricted deletion by the controller

C. Requiring approval, rollback, or dual control

D. Disabling verification after the action

Correct Answer: C. Requiring approval, rollback, or dual control

Question 7

A data product should normally include which combination?

A. Only a table name and storage location

B. An owner, documented meaning, quality indicators, and access expectations

C. A copied file with no retention policy

D. An undocumented query shared by email

Correct Answer: B. An owner, documented meaning, quality indicators, and access expectations

Question 8

Why is event-time processing valuable when network connectivity is intermittent?

A. It assigns records according to occurrence rather than arrival delay

B. It guarantees that records cannot be duplicated

C. It removes the need for state management

D. It makes all processing instantaneous

Correct Answer: A. It assigns records according to occurrence rather than arrival delay

Question 9

Which metric is most suitable for describing tail latency?

A. Mean only

B. Median only

C. p99 latency

D. Total storage capacity

Correct Answer: C. p99 latency

Question 10

What is the strongest reason to require human review of AI-generated data-engineering code?

A. Generated code is always syntactically invalid

B. AI cannot produce documentation

C. Plausible generated code can contain semantic, security, or privacy errors

D. Human review makes testing unnecessary

Correct Answer: C. Plausible generated code can contain semantic, security, or privacy errors