Data Acquisition, Storage and Architecture

Introduction

Data science begins before analysis. A model, dashboard, or decision-support system is only as reliable as the data entering it and the architecture used to preserve, transform, govern, and deliver it. Data acquisition concerns obtaining data from relevant sources; data storage concerns retaining it securely and efficiently; and data architecture concerns organizing technologies, processes, and responsibilities so that data can be used throughout its life cycle.

Modern organizations collect structured data from relational databases and enterprise applications, semi-structured data from APIs and JSON documents, and unstructured data from text, images, audio, video, logs, and documents. Sensors and operational systems may generate data continuously, while web sources may change without notice. These differences affect data quality, cost, latency, privacy, processing, and storage design.

This chapter examines acquisition methods and compares relational databases, NoSQL systems, data warehouses, data lakes, and lakehouses. It also introduces ingestion pipelines, metadata, data governance, security, and architectural decision-making.

Learning Objectives

After studying this chapter, you should be able to:

1. Distinguish structured, semi-structured, and unstructured data.
2. Explain how databases, files, APIs, web sources, sensors, and enterprise systems are acquired.
3. Compare batch, streaming, and change-data-capture ingestion.
4. Explain the purposes and trade-offs of relational, NoSQL, warehouse, lake, and lakehouse architectures.
5. Design a basic acquisition and storage architecture for a practical business problem.
6. Evaluate issues of quality, metadata, security, privacy, governance, and responsible use.

1. Data as an Organizational Resource

Data is a recorded representation of observations, transactions, events, or entities. A data value becomes useful when it has context, meaning, and an identifiable source. For example, “42” is ambiguous without knowing whether it represents age, temperature, quantity, or a product code. Metadata supplies this context by describing fields, types, units, ownership, origin, and time of capture.

A useful data life cycle is:

1. Create or capture: A transaction, sensor reading, web event, or human action produces data.
2. Acquire and ingest: Data is transferred into an organization’s controlled environment.
3. Store: Data is retained in an operational database, file system, warehouse, lake, or another repository.
4. Prepare and integrate: Data is cleaned, standardized, joined, and documented.
5. Analyze and visualize: Analysts and applications use the data.
6. Archive or delete: Data is retained, archived, or removed according to business and legal requirements.

Data can be classified by structure:

The distinction is practical rather than absolute. A photograph is unstructured in its pixels, but its filename, location, timestamp, and camera model may be structured metadata.

2. Sources and Methods of Data Acquisition

2.1 Databases

Operational databases support applications such as banking, retail, healthcare, education, and inventory management. Relational databases organize data into tables and usually expose it through SQL. Acquisition may occur through:

• scheduled SQL extraction;
• read-only replicas, which reduce load on production systems;
• database export files;
• views that expose only approved fields; and
• change-data capture (CDC), which records inserts, updates, and deletes.

Directly querying a production database for intensive analytics can slow business operations. A safer design uses replicas, controlled extraction windows, throttling, or CDC. Extraction must also account for transaction consistency: related tables should represent a coherent point in time rather than a mixture of different states.

2.2 Files and Documents

Files include CSV, spreadsheets, JSON, XML, Parquet, images, PDFs, and application logs. File-based acquisition is common because it is simple and compatible with many systems. However, files may contain inconsistent column names, missing values, duplicate records, hidden spreadsheet formulas, encoding differences, and ambiguous dates.

A robust file-ingestion process should validate:

• file name, size, format, and expected arrival time;
• schema and data types;
• character encoding and delimiter;
• record counts and duplicate identifiers;
• date, time-zone, and unit conventions; and
• malware and access permissions.

Columnar formats such as Parquet are often efficient for analytical workloads because related values are stored together by column and compression can reduce storage and scanning costs. This does not make them universally superior: row-oriented formats may be more appropriate when complete records are frequently inserted or retrieved.

2.3 APIs

An application programming interface (API) provides a controlled way for one software system to request data from another. APIs commonly use HTTP methods and return JSON or XML. Acquisition engineers must understand authentication, authorization, rate limits, pagination, retries, versioning, and error responses.

A reliable API client should:

1. authenticate using an approved method;
2. request only necessary fields;
3. follow pagination rather than assuming one response contains all records;
4. record request time, endpoint, status code, and source version;
5. retry transient failures with bounded exponential backoff; and
6. avoid repeated requests that violate provider policies.

API data is not automatically authoritative. A provider may revise historical values, omit records, change definitions, or impose a quota. The acquisition system should preserve raw responses or reproducible extracts where permitted.

2.4 Web Sources

Web data may be obtained from public datasets, open-data portals, feeds, or web pages. Web scraping should be undertaken only when legally and ethically permitted. A responsible process respects terms of service, robots policies where relevant, copyright, privacy, access limits, and attribution requirements.

Web pages are designed for presentation, not necessarily stable extraction. A change in HTML structure can silently break a scraper. Therefore, source monitoring, test cases, extraction logs, and validation rules are essential. When an official API or downloadable dataset exists, it is generally preferable to scraping.

2.5 Sensors and Internet-of-Things Systems

Sensors produce observations such as temperature, pressure, location, vibration, or energy consumption. Sensor acquisition is characterized by high volume, repeated time measurements, intermittent connectivity, device failures, and the need for time synchronization.

Each reading should ideally include a device identifier, event timestamp, ingestion timestamp, measurement, unit, quality indicator, and possibly location or calibration version. Event time is when the observation occurred; ingestion time is when the platform received it. The distinction is important when messages arrive late or out of order.

Edge processing can filter noise, aggregate readings, detect threshold events, or operate during network outages. It reduces bandwidth but may discard information needed for later analysis. The design should therefore distinguish lossless retention from intentional summarization.

2.6 Enterprise Systems

Enterprise resource planning, customer relationship management, human-resource, finance, point-of-sale, and supply-chain systems contain valuable but heterogeneous data. Integration requires mapping identifiers and business definitions. For instance, “customer,” “active account,” and “revenue” may have different meanings in different systems.

Enterprise acquisition should involve data owners and subject-matter experts, not only technical teams. Technical integration without semantic agreement can produce a large but misleading dataset.

3. Ingestion Patterns

Batch Ingestion

Batch ingestion transfers data at scheduled intervals, such as hourly, nightly, or weekly. It is suitable when decisions do not require immediate updates, when sources provide periodic files, or when processing efficiency is more important than low latency. Batch systems are generally easier to test and operate.

Streaming Ingestion

Streaming ingestion processes events continuously or in small time windows. It is useful for fraud detection, equipment monitoring, clickstream analysis, and operational alerts. Streaming introduces additional concerns: ordering, late events, duplicates, replay, back-pressure, and exactly-once or at-least-once delivery semantics.

A system described as “real time” should specify a measurable latency target. If an event is available for analysis within 30 seconds, that is a different service level from availability within 100 milliseconds.

Change Data Capture

CDC captures changes in a source system, commonly from transaction logs or database mechanisms. It avoids repeatedly scanning entire tables and can preserve the sequence of inserts, updates, and deletes. CDC must address schema changes, transaction boundaries, deleted records, and reprocessing. It is especially valuable for maintaining analytical copies of operational data.

ETL and ELT

In extract–transform–load (ETL), data is transformed before loading into the target repository. In extract–load–transform (ELT), raw or lightly processed data is loaded first and transformed within the analytical platform. ETL may be appropriate when the target has strict schema or when sensitive fields must be removed before storage. ELT is attractive when scalable storage and compute allow multiple transformations and preservation of raw data.

A sound pipeline is observable. It records run status, input and output counts, rejected records, processing duration, schema versions, and error details. Idempotence is also important: rerunning a job should not unintentionally duplicate data. A common strategy is to assign stable event identifiers and use controlled merge operations.

4. Architectural Foundations

A data architecture is the organized arrangement of data sources, ingestion mechanisms, storage layers, processing services, governance controls, and consumers. Architecture should be selected from requirements rather than fashion. Important requirements include:

• latency: how quickly data must be available;
• volume and velocity: how much data arrives and how quickly;
• variety: the number and diversity of formats;
• consistency and transaction needs;
• query patterns and concurrency;
• retention and legal obligations;
• security and residency requirements;
• cost and staffing; and
• recovery, availability, and scalability.

The following simplified flow illustrates a common analytical architecture.

Figure: A conceptual flow from data sources through ingestion and storage to analytical use, with governance and security spanning the architecture.

5. Relational Databases

A relational database stores data in tables with rows and columns. A schema defines attributes and data types, while primary keys identify records and foreign keys represent relationships. SQL supports selection, joining, aggregation, insertion, and updates.

Relational systems are strong when data is highly structured, relationships are important, and transactions require consistency. The ACID properties are commonly summarized as atomicity, consistency, isolation, and durability. Atomicity means a transaction is completed fully or not at all. Consistency means committed transactions preserve defined constraints. Isolation controls the interaction of concurrent transactions. Durability means committed changes survive failures.

Normalization reduces unnecessary duplication by separating facts into related tables. It improves update consistency but may require joins. Analytical warehouses often use denormalized star schemas, consisting of a central fact table and surrounding dimension tables, to simplify and accelerate reporting.

Relational systems are less convenient for rapidly changing schemas, massive unstructured objects, or highly distributed workloads that do not fit table relationships. They remain, however, central to operational and analytical computing.

6. NoSQL Architectures

NoSQL systems are designed for particular access patterns and large-scale or flexible data. Major categories include:

NoSQL does not mean “no structure.” It usually means the structure is not constrained by one universal relational schema. Benefits may include horizontal scalability, flexible records, and efficient access for known patterns. Trade-offs may include weaker cross-record transactions, duplicated data, application-managed relationships, vendor-specific query languages, or more complicated consistency models.

The correct question is not whether NoSQL is better than SQL, but whether its data model and consistency behavior match the application’s requirements.

7. Data Warehouses

A data warehouse is a curated repository optimized for analytical queries across integrated historical data. It typically uses schemas, governed definitions, dimensional models, and workload management. Warehouses are appropriate for dashboards, financial reporting, performance analysis, and structured decision support.

A warehouse generally emphasizes conformed dimensions, data quality, stable business definitions, and efficient aggregation. It may receive data through ETL or ELT. A warehouse is not simply a large database: its design, governance, and analytical purpose distinguish it from an operational transaction system.

Warehouse limitations include the effort needed to model new data, possible costs associated with intensive queries, and reduced convenience for raw multimedia or rapidly changing unstructured content.

8. Data Lakes

A data lake stores large amounts of raw or lightly processed data in economical storage, often using open file formats and object storage. It can accommodate structured, semi-structured, and unstructured data. A lake supports exploratory analysis, machine learning, archival, and later reprocessing.

The flexibility of a lake creates risks. Without cataloging, quality rules, ownership, and lifecycle controls, it can become a “data swamp”: data exists but cannot be found, trusted, interpreted, or governed. Common controls include zones such as raw, refined, and curated; schema documentation; quality checks; access policies; retention rules; and lineage.

9. Lakehouses

A lakehouse seeks to combine the openness and variety of a lake with the governance, reliability, and analytical management associated with a warehouse. It commonly uses object storage, open or broadly supported file formats, metadata layers, schema enforcement or evolution, transaction support, and table-management features.

A lakehouse can reduce duplication by allowing multiple workloads to use governed data in one environment. Nevertheless, it does not remove engineering challenges. Performance tuning, concurrency, metadata management, access control, cost monitoring, and interoperability still require careful design. The label “lakehouse” should describe actual capabilities rather than serve as a marketing substitute for architecture.

Comparative View

10. Data Quality, Metadata, and Governance

Data quality is multidimensional. Important dimensions include accuracy, completeness, consistency, timeliness, validity, uniqueness, and fitness for purpose. No dataset is simply “high quality” in the abstract: quality depends on a use case. A postal address may be complete enough for regional analysis but insufficient for delivery.

Quality controls should occur as near to the source as practical and again at important processing boundaries. Examples include range checks, referential-integrity checks, duplicate detection, schema validation, reconciliation to source totals, and monitoring of missingness over time.

Metadata should identify:

• business meaning and owner;
• technical type, format, and schema;
• source and transformation history;
• update frequency and time zone;
• sensitivity classification;
• quality measures and known limitations; and
• permitted uses and retention period.

A data catalog helps users discover assets. Data lineage shows where a field originated and how it changed. Governance assigns decision rights and responsibilities. A data steward may coordinate definitions and quality, while a security team manages technical controls and a legal or compliance function interprets obligations.

11. Security, Privacy, and Ethical Responsibilities

Data acquisition must follow purpose limitation and data minimization: collect what is needed for a legitimate purpose, not everything technically accessible. Personal data should be classified and protected with appropriate authentication, authorization, encryption, logging, masking, and retention controls. Access should follow least privilege, meaning users receive only the permissions needed for their roles.

Privacy risks can remain after direct identifiers are removed. Combinations of location, time, demographics, and behavior may re-identify individuals. Consent, notice, lawful basis, deletion rights, cross-border transfer requirements, and sector-specific rules vary by jurisdiction; organizations must obtain appropriate legal guidance rather than assume that public availability eliminates responsibility.

Ethical acquisition also concerns representativeness and power. Data from online platforms may exclude less-connected populations. Sensor deployment may affect workers or communities. Scraped content may reflect individuals’ expectations differently from the organization’s intended use. Provenance, fairness analysis, documentation, and human review are therefore part of professional data practice.

12. Worked Application: University Student-Success Platform

Background and problem

This illustrative case concerns a university that wants to identify students who may benefit from academic support. Relevant information exists in the student information system, learning-management system, attendance devices, advising notes, and student surveys. The university must avoid using the platform as an automatic disciplinary mechanism and must protect sensitive information.

Analysis

The student information system contains structured enrollment and course records in a relational database. Learning events are available through an API and include semi-structured JSON. Attendance devices generate time-stamped events. Advising notes may contain free text and sensitive personal information. Surveys are submitted as files.

The requirements are daily indicators for advisors, historical reporting, and exploratory research. Immediate automated intervention is not required. This suggests batch acquisition for most sources, with optional near-real-time ingestion only for operational alerts.

Approach and solution

A read-only replica supplies approved academic fields. CDC transfers changes to enrollment and course registrations. The learning API is collected incrementally using a documented watermark such as the last successful event time, with overlap to capture late updates. Attendance events are validated for device identifier, event time, course, and duplicate event ID. Survey files enter a quarantined area until schema and malware checks pass.

Raw data is retained in a restricted lake zone. Curated, de-identified analytical tables are stored in a warehouse or governed lakehouse. A catalog documents definitions such as “active enrollment” and “course engagement.” Access to advising notes is separated from the predictive feature table. Advisors see explanations and support recommendations, while students are informed about the purpose and given an avenue to challenge incorrect records.

Quality monitoring reconciles daily enrollment counts with the source system, measures missing event rates, and flags sudden changes in API volume. The system records lineage from source fields to dashboard indicators. Retention and deletion schedules are aligned with institutional policy and applicable law.

Outcome

The architecture provides reliable daily information without copying unrestricted sensitive notes into general analytics storage. Advisors receive earlier signals, but a human makes the support decision. The design can be expanded later, provided that new data sources undergo the same privacy, quality, and governance review.

Learning points

1. Architecture follows purpose and latency requirements.
2. Different sources require different acquisition controls.
3. Raw retention is useful only when metadata and access controls exist.
4. Predictive use does not remove the need for human judgment.
5. Data definitions and governance are as important as technology.

13. Best Practices and Contemporary Developments

Effective data architecture is increasingly supported by automated data quality monitoring, data contracts, observability platforms, workflow orchestration, and privacy-enhancing techniques. A data contract specifies expected fields, types, meanings, service levels, and change procedures between producers and consumers. Data observability monitors freshness, volume, schema, distribution, and lineage so that failures are detected before users rely on incorrect outputs.

Organizations are also adopting domain-oriented ownership models, sometimes called data mesh approaches. Such models distribute responsibility to domain teams while requiring shared standards for interoperability, security, and governance. Distribution can improve local knowledge but can also fragment definitions if central policies and federated coordination are weak.

Artificial intelligence increases demand for trustworthy acquisition. Training data requires provenance, licensing review, deduplication, bias assessment, versioning, and protection against leakage of confidential information. Generative AI systems can also create synthetic or transformed data, but synthetic data is not automatically private, accurate, or representative. It must be evaluated against the intended use.

A practical architecture review should ask:

• What decision or service will the data support?
• Which sources are authoritative for each business fact?
• What latency, volume, availability, and recovery targets apply?
• How will late, duplicate, invalid, or changed records be handled?
• Who owns definitions, quality, access, and retention?
• Can the organization explain the origin and permitted use of each important field?
• What happens when a source, pipeline, or storage service fails?

Challenges and Limitations

No architecture solves poor source processes automatically. Data may be incomplete, biased, contradictory, or unavailable. Integrating identifiers across systems is difficult, especially when entities change names or have multiple accounts. Streaming improves freshness but increases operational complexity. Lakes reduce initial modeling effort but demand stronger discovery and governance. Warehouses improve consistency but can encourage premature exclusion of valuable unstructured data. NoSQL systems can scale specific access patterns while transferring complexity to application developers.

Cost is also multidimensional. Storage, compute, network transfer, support staff, licensing, security, downtime, and the cost of wrong decisions all matter. A technically elegant design may be unsuitable if the organization cannot operate it reliably. Incremental architecture—starting with clear requirements, measurable service levels, and limited scope—often produces better outcomes than adopting every available technology.

Chapter Summary

Data acquisition collects information from databases, files, APIs, web sources, sensors, and enterprise applications. Each source has distinct issues involving structure, authentication, frequency, reliability, semantics, and legal or ethical use. Batch, streaming, and CDC ingestion support different latency and consistency requirements, while ETL and ELT place transformation at different points in the pipeline.

Relational databases provide structured schemas, relationships, and strong transactional behavior. NoSQL systems support specialized models and scalable access patterns. Warehouses provide curated analytical performance; data lakes retain diverse raw data at scale; and lakehouses seek to combine lake flexibility with warehouse-like governance and reliability. The choice must reflect workload, quality, latency, security, cost, and organizational capability.

Metadata, lineage, quality monitoring, access control, privacy protection, and responsible governance are foundational rather than optional additions. A professional data architecture makes data discoverable, trustworthy, secure, reproducible, and fit for its intended purpose.

Key Terms

Data acquisition: The process of obtaining data from internal or external sources.

Metadata: Information describing data’s meaning, structure, origin, quality, and use.

Structured data: Data organized according to a fixed schema, commonly in tables.

Semi-structured data: Data with flexible tags, keys, or nested organization.

Unstructured data: Data without a consistent tabular structure, such as images or audio.

Batch ingestion: Transfer and processing of data at scheduled intervals.

Streaming ingestion: Continuous or near-continuous processing of events.

Change data capture: Recording inserts, updates, and deletes made in a source system.

ETL: Extract, transform, and load.

ELT: Extract, load, and transform.

Relational database: A database using related tables, schemas, keys, and SQL.

NoSQL: A family of non-relational database models designed for particular access patterns.

Data warehouse: A governed repository optimized for integrated analytical queries.

Data lake: Scalable storage for raw or lightly processed data of varied types.

Lakehouse: An architecture combining lake storage flexibility with warehouse-like management features.

Data catalog: An inventory describing available data assets and their characteristics.

Data lineage: A record of data origins and transformations.

Data quality: The degree to which data is accurate, complete, consistent, timely, valid, and fit for purpose.

Data governance: Policies, roles, processes, and controls governing data use.

Idempotence: The property that repeating an operation does not create unintended additional effects.

Review Questions

Short Answer Questions

1. Distinguish structured, semi-structured, and unstructured data, giving one example of each.
2. What is the difference between event time and ingestion time?
3. State two advantages and two limitations of a data warehouse.
4. What does change-data capture record?
5. Why are metadata and data lineage important?

Long Answer Questions

1. Explain how a university or retail organization could acquire data from databases, APIs, files, and sensors. Include validation and security controls.
2. Compare relational, NoSQL, warehouse, lake, and lakehouse architectures in terms of data model, strengths, limitations, and appropriate use.
3. Discuss batch, streaming, and CDC ingestion and explain how latency and reliability requirements influence the choice.
4. Explain how a data lake can become a data swamp and propose governance measures to prevent this outcome.
5. Discuss privacy, ethical, and professional responsibilities involved in acquiring and storing personal data.

Analytical / Application Questions

1. A manufacturer receives machine readings every second, monthly maintenance spreadsheets, and transactional data from an ERP database. Design an acquisition and storage architecture, justifying ingestion patterns and storage choices.
2. An API begins returning duplicate and out-of-order events after a provider upgrade. Analyze possible causes and propose validation, deduplication, replay, and monitoring procedures.
3. A company wants one platform for dashboards, machine learning, raw documents, and historical audit data. Evaluate whether a warehouse, lake, or lakehouse is most suitable, and identify the governance risks of your recommendation.

Multiple Choice Questions with Answers

Question 1

Which acquisition method records inserts, updates, and deletes from a source system?

A. Change data capture

B. Web scraping

C. Manual spreadsheet entry

D. Static visualization

Correct Answer: A. Change data capture

Question 2

Which architecture is primarily optimized for governed analytical queries and integrated historical data?

A. Operational cache

B. Data warehouse

C. Key-value store

D. Sensor gateway

Correct Answer: B. Data warehouse

Question 3

What is the main purpose of metadata?

A. To replace all source data

B. To describe data’s meaning, origin, structure, and use

C. To eliminate the need for security

D. To guarantee that data is unbiased

Correct Answer: B. To describe data’s meaning, origin, structure, and use

Question 4

Which ingestion pattern is generally most suitable when data is needed continuously for immediate alerts?

A. Streaming ingestion

B. Annual batch processing

C. Manual file upload

D. Archival deletion

Correct Answer: A. Streaming ingestion

Question 5

Which NoSQL model represents entities as nodes and their relationships as edges?

A. Key-value

B. Document

C. Graph

D. Wide-column

Correct Answer: C. Graph

Question 6

A data lake becomes a data swamp primarily when it lacks:

A. Any raw data

B. Governance, cataloging, ownership, and quality controls

C. All forms of object storage

D. Internet connectivity

Correct Answer: B. Governance, cataloging, ownership, and quality controls

Question 7

In a sensor system, event time refers to:

A. When the observation occurred

B. When a dashboard was opened

C. When a database backup ran

D. When a user changed a password

Correct Answer: A. When the observation occurred

Question 8

Which property means that rerunning an ingestion operation does not unintentionally create duplicate effects?

A. Normalization

B. Idempotence

C. Pagination

D. Encryption

Correct Answer: B. Idempotence

Question 9

Which practice best supports least-privilege access?

A. Giving every analyst administrator rights

B. Allowing access only to data required for a user’s role

C. Publishing all raw data publicly

D. Disabling access logs

Correct Answer: B. Allowing access only to data required for a user’s role

Question 10

A lakehouse is intended primarily to combine:

A. Spreadsheet formulas and email systems

B. Lake-scale flexible storage with warehouse-like governance and reliability

C. Only relational transactions with no analytics

D. Web scraping with manual reporting

Correct Answer: B. Lake-scale flexible storage with warehouse-like governance and reliability