6. Data Sources, Acquisition and Management

Introduction and Context

Data science begins long before modelling. A predictive model, dashboard, or artificial intelligence system is only as credible as the data on which it depends. Data must be located, acquired, interpreted, stored, protected, documented, and made available for appropriate use. These activities are often treated as technical preparation, but they also involve organizational judgment, legal responsibility, information architecture, and professional ethics.

Data may be generated by transaction-processing systems, sensors, mobile devices, websites, scientific instruments, surveys, documents, social platforms, or external organizations. It may appear as a neatly structured table, a JSON response from an application programming interface, an email attachment, an image, or a stream of events. Each form creates different requirements for acquisition and management. A data scientist therefore needs to understand not only how to read data, but also where it came from, what it represents, how it changes, and whether it is fit for purpose.

This chapter examines the main forms and sources of data, methods of acquisition, database and file technologies, data models, metadata, data quality, privacy, security, governance, and lineage. It also considers the practical decisions involved in building reliable data pipelines.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish structured, semi-structured, and unstructured data.

2. Evaluate internal and external sources of data for a stated analytical purpose.

3. Use the main principles of relational databases, SQL, APIs, and file-based acquisition.

4. Explain data models, schemas, metadata, data quality, lineage, and governance.

5. Design a basic acquisition and management workflow.

6. Identify privacy, security, ethical, and regulatory risks.

7. Select appropriate storage and integration approaches for professional data-science projects.

The Data Landscape: Forms, Sources, and Meaning

Structured, semi-structured, and unstructured data

Structured data is organized according to a predefined schema. A customer table, for example, may contain columns such as customer_id, date_of_birth, and account_status, with each row representing one customer. Relational databases are the most familiar environment for structured data. Its strengths include consistency, efficient querying, defined relationships, and strong transaction controls. Its weakness is that a rigid schema may be unsuitable for rapidly changing or highly varied information.

Semi-structured data does not conform to a single tabular structure but contains tags, keys, labels, or hierarchical relationships. JSON and XML are common examples. A JSON record may contain a customer identifier, an address object, and an array of transactions, with different records containing different optional fields. Semi-structured formats are common in web services and event systems because they are flexible and portable. Their flexibility, however, can conceal inconsistent field names, types, nesting, and missing values.

Unstructured data lacks a predefined analytical structure. Text documents, images, audio, video, scanned forms, and many emails are examples. Such data often contains substantial information, but analysis normally requires additional processes such as optical character recognition, natural language processing, image analysis, or speech recognition. Unstructured does not mean meaningless; it means that meaning is not represented in a fixed relational schema.

The distinction is useful but not absolute. A video file may be unstructured internally while its catalogue contains highly structured metadata. A document repository may therefore combine all three forms. Effective data management recognizes both the content and the descriptive information surrounding it.

Internal and external sources

Internal data is produced within an organization. Examples include sales transactions, employee records, financial ledgers, customer-service interactions, machine logs, and website events. Internal data often aligns closely with organizational processes, but it may be fragmented across departments and systems.

External data is obtained from outside the organization. It may include government open data, commercial datasets, partner information, public APIs, research repositories, satellite observations, or social-media data. External sources can enrich analysis, but their definitions, collection methods, licensing conditions, and quality may be difficult to verify.

A source should be evaluated in relation to a specific purpose. Important questions include: Who collected the data? For what purpose? When and where was it collected? What population does it represent? Are important groups missing? Is the definition of a field stable? What permissions govern its use? Data that is adequate for operational reporting may be inappropriate for causal research or automated decision-making.

Data Models and Storage Choices

A data model describes the entities, attributes, relationships, constraints, and rules that organize data. Conceptual models describe business ideas at a high level; logical models specify entities and relationships without committing to a particular technology; physical models describe implementation in tables, indexes, partitions, files, or storage services.

In the relational model, data is represented in relations, commonly implemented as tables. A primary key uniquely identifies a row, while a foreign key refers to a key in another table. Normalization reduces unnecessary duplication and update anomalies by separating related facts into appropriately designed tables. For example, customer details should generally not be repeated in every order record. However, analytical systems may deliberately denormalize data to improve query performance.

SQL is the principal language for querying relational data. A simple query might be written as:

```sql

SELECT customer_id, SUM(amount) AS total_spend

FROM orders

WHERE order_date >= '2025-01-01'

GROUP BY customer_id;

```

This statement filters records, groups them by customer, and computes an aggregate. Joins allow information from related tables to be combined. Analysts must understand join types because an incorrect join can duplicate records, omit entities, or produce misleading totals. Constraints, data types, indexes, transactions, and access permissions are also part of responsible database use.

NoSQL technologies are designed for use cases in which relational structures are insufficient or inefficient. Document databases store JSON-like documents; key-value stores associate a key with a value; column-family systems organize data by column groups; and graph databases represent nodes and relationships. The term NoSQL covers different designs rather than a single model. Selection should be driven by access patterns, consistency requirements, scale, relationship complexity, and team capability—not by fashion.

Data warehouses are structured repositories optimized for analytical queries, often integrating historical data from multiple operational systems. Data lakes store large volumes of raw or lightly transformed data in varied formats. A lakehouse attempts to combine the flexibility of a lake with the management and analytical features of a warehouse. These architectures may coexist, but they require clear ownership, cataloguing, security, and lifecycle controls.

Acquiring Data: Methods and Workflow

A sound acquisition process begins with a purpose and a data specification. The specification should identify the population, variables, time period, granularity, acceptable latency, quality expectations, legal basis, and required output. Acquisition should not begin merely because data is available.

Databases and SQL extraction

Data may be extracted through read-only database accounts, views, scheduled queries, or replication mechanisms. A view can expose only necessary fields and reduce the risk of directly accessing sensitive tables. Incremental extraction is usually preferable to repeatedly copying an entire database. It may use a timestamp, sequence number, change-data-capture log, or other reliable change indicator.

Extraction must account for transaction timing. A query executed while records are changing may create an inconsistent snapshot unless the database provides appropriate isolation. Analysts should record extraction time, query logic, source version, and row counts. These details support reproducibility and investigation when results change.

APIs

An application programming interface provides a defined method for one system to request data or services from another. APIs commonly use HTTP methods, authentication tokens, parameters, pagination, rate limits, and JSON responses. A robust API client should handle authentication securely, check status codes, respect rate limits, retry temporary failures with backoff, validate the response schema, and record request metadata.

Pagination is especially important. Retrieving only the first page silently produces incomplete data. APIs may also return data in coordinated time zones, use changing field definitions, or impose limits on historical access. A data scientist should read the provider's documentation and preserve the terms of use and version information.

Files and documents

Common file formats include CSV, Excel, JSON, XML, Parquet, images, and PDF. CSV is simple and widely supported but may suffer from delimiter conflicts, ambiguous encodings, inconsistent quoting, and loss of data types. Excel files can contain multiple sheets, formulas, hidden rows, and presentation formatting. Parquet is a columnar format well suited to analytical workloads because it supports compression and efficient column selection.

File ingestion should include validation of filenames, extensions, encoding, schema, record counts, duplicate files, and checksums where appropriate. Original files should be retained in a controlled location, while transformed copies should be clearly distinguished. Documents and images may require text extraction, classification, or annotation before analysis.

Streaming and event data

Batch acquisition collects data at intervals; streaming acquisition processes events as they are generated. Streaming is useful for fraud detection, monitoring, industrial control, and personalization. It introduces additional concerns: event ordering, duplicate delivery, late-arriving events, replay, idempotence, and state management. A system should define whether it offers at-most-once, at-least-once, or effectively exactly-once processing, recognizing that these guarantees have operational consequences.

A practical acquisition workflow usually contains the following stages: define the use case; identify and assess sources; obtain authorization; extract or receive data; preserve the raw input; validate and profile it; transform and integrate it; document lineage and metadata; publish an approved dataset; monitor quality; and archive or delete data according to policy.

Metadata, Quality, and Lineage

Metadata is data about data. Technical metadata describes types, schemas, formats, partitions, and storage locations. Business metadata explains definitions, ownership, permitted use, and reporting meaning. Operational metadata records refresh times, processing status, row counts, and failures. Without metadata, data may exist but remain difficult to discover or interpret.

A data catalogue brings metadata together so that users can locate and evaluate datasets. A useful catalogue entry identifies the owner, steward, source system, update frequency, field definitions, sensitivity classification, quality indicators, and access procedure. A data dictionary provides more detailed definitions, including valid values, units, and relationships.

Data quality is fitness for a defined use rather than an abstract property. Common dimensions include:

• Accuracy: whether values represent the real-world phenomenon.
• Completeness: whether required records or fields are present.
• Consistency: whether values agree across systems and time.
• Timeliness: whether data is available and current when needed.
• Validity: whether values conform to permitted formats and rules.
• Uniqueness: whether records are unnecessarily duplicated.

Quality controls may include range checks, referential-integrity checks, duplicate detection, reconciliation with trusted totals, anomaly detection, and sampling against source documents. Cleaning should not automatically replace unusual values. An extreme value may be an error, but it may also be the most important observation. The treatment and rationale should be documented.

Data lineage records the origin, movement, transformation, and use of data. It may show that a dashboard metric was derived from a source table, filtered by a particular rule, joined to another table, and aggregated by a scheduled job. Lineage supports impact analysis, debugging, auditability, regulatory response, and trust. It is particularly important when a model or decision depends on multiple transformations.

Privacy, Security, and Governance

Data management is inseparable from responsible use. Personal data is information relating to an identifiable person, either directly or indirectly. Sensitive categories may receive additional legal protection depending on jurisdiction. Organizations should determine the applicable law and policy rather than assuming that publicly accessible data is unrestricted.

Privacy principles commonly include purpose limitation, data minimization, transparency, accuracy, retention limitation, security, and accountability. Collecting data for one stated purpose and later using it for an unrelated purpose may be ethically or legally problematic. De-identification reduces risk but does not guarantee anonymity, especially when datasets can be linked. Pseudonymization replaces direct identifiers with codes while retaining a means of re-identification; it therefore remains subject to controls.

Security controls should operate across the data lifecycle. Encryption protects data in transit and at rest. Authentication verifies identity, while authorization determines permitted actions. Least privilege limits access to what a user needs. Logging, segregation of duties, secure backups, vulnerability management, and incident-response procedures reduce operational risk. Sensitive data should not be placed in personal notebooks, uncontrolled spreadsheets, or public repositories.

Governance assigns decision rights and responsibilities. A data owner is accountable for a domain or asset; a data steward helps maintain definitions, quality, and appropriate use; a custodian manages technical storage and access. Governance should define classification, retention, sharing, quality thresholds, change management, and disposal. It should also address the use of data in machine-learning systems, including bias assessment, explainability requirements, and human oversight where decisions affect individuals.

An Illustrative Case: Integrating Retail Data for Demand Planning

Background

A fictional retail organization, Northstar Stores, operates physical shops and an online channel. Managers want a weekly demand-planning dataset to reduce stockouts and excess inventory.

Situation and analysis

Sales transactions are stored in a relational system, online events are available through an API, inventory snapshots arrive as CSV files from stores, and product descriptions are maintained in spreadsheets. Initial reports disagree because product identifiers differ between systems, store files contain duplicate rows, and online timestamps use UTC while store systems use local time. Some products are discontinued but remain in historical files.

The analytical requirement is not simply to combine every available record. The team must define the unit of analysis—product, location, and day—along with the meaning of a sale, return, cancellation, stockout, and available inventory. It must also decide how to treat late-arriving transactions and whether online demand should be attributed to a fulfilment location.

Approach

The team creates a logical data model with product, location, calendar, sales, inventory, and channel entities. A controlled product master supplies a stable product key. Database extracts are performed incrementally using a transaction timestamp, while API responses are stored in their original JSON form. Incoming CSV files are checked for encoding, required columns, duplicate keys, and valid product codes.

A raw storage area preserves source files and responses. Transformation jobs standardize time zones, map identifiers, remove confirmed duplicates, and record rejected rows for review rather than silently deleting them. The analytical table is published only after reconciliation against finance-approved sales totals. A catalogue documents field definitions, refresh schedules, owners, quality checks, and known limitations. Access to customer-level information is restricted because demand planning requires aggregated transactions rather than names or contact details.

Outcome

The organization obtains a reproducible daily dataset with documented lineage from each source to the final planning table. Managers can distinguish a genuine demand decline from a stockout-related absence of sales. Data-quality reports expose late files and identifier failures before they affect forecasts. The system does not eliminate all uncertainty, but it makes assumptions visible and enables controlled improvement.

Learning points

The case demonstrates that integration is a semantic problem as well as a technical one. Stable identifiers, time definitions, business rules, raw-data preservation, quality monitoring, and access control are all necessary. A sophisticated forecasting algorithm would not correct ambiguous definitions or undocumented transformations.

Common Difficulties and Professional Best Practice

Frequent mistakes include collecting data without a defined question, trusting a file because it opens successfully, assuming column names have universal meanings, ignoring time zones, using inner joins without checking lost records, overwriting raw data, hard-coding credentials, and treating missing values as zero. Another common error is to focus on a pipeline's successful execution rather than on whether its output remains meaningful.

Good practice includes profiling unfamiliar data before analysis; separating raw, cleaned, and published layers; versioning code and schemas; using automated tests; documenting assumptions; applying incremental processing where possible; monitoring freshness and volume; and establishing clear ownership. Sensitive fields should be identified early, not after a breach or inappropriate analysis. Data retention and disposal should be planned at the beginning of a project.

For postgraduate practitioners, a further principle is critical: every dataset is an argument about reality. Its categories reflect institutional processes, collection incentives, measurement choices, and exclusions. Technical competence must therefore be combined with critical questioning. When a dataset is used for high-impact decisions, practitioners should examine representation, possible discrimination, consent, contestability, and the consequences of error.

Current and Emerging Developments

Modern data platforms increasingly automate catalogue generation, schema detection, lineage capture, quality monitoring, and access workflows. Data contracts are being used to specify the structure and service expectations that producing systems must provide to consuming teams. Data products treat curated datasets as managed products with users, owners, documentation, service levels, and feedback mechanisms.

Privacy-enhancing technologies, including federated analysis, secure computation, differential privacy, and synthetic-data methods, are being explored where direct data sharing creates risk. These techniques are not universal solutions: they can reduce utility, introduce statistical distortion, or create false confidence if poorly implemented. Artificial intelligence is also being used to classify documents, map schemas, detect anomalies, and assist catalogue search, but automated suggestions require human review because they may propagate incorrect definitions or sensitive inferences.

The direction of development is toward more observable, policy-aware, and interoperable data environments. Nevertheless, automation does not remove accountability. Organizations remain responsible for the provenance, quality, security, and appropriate use of the data they manage.

Chapter Summary

Data acquisition and management provide the foundation for trustworthy data science. Data may be structured, semi-structured, or unstructured, and may originate internally or externally. Relational databases and SQL support structured querying, while APIs, files, streaming systems, and NoSQL technologies support varied acquisition and storage needs. A data model describes entities and relationships; metadata explains what data means and how it is managed; quality dimensions help assess fitness for purpose; and lineage records movement and transformation. Effective practice requires validation, reproducibility, security, privacy protection, governance, and explicit ownership. The retail case illustrates that successful integration depends on semantics, identifiers, time, quality controls, and documentation as much as on software. Emerging tools can automate aspects of management, but professional judgment and accountability remain essential.

Key Terms

Structured data: Data organized according to a predefined schema, commonly in relational tables.

Semi-structured data: Data with organizational markers such as keys, tags, or hierarchical nesting, but without a fixed tabular schema.

Unstructured data: Content such as text, images, audio, or video without a predefined analytical structure.

Data model: A representation of entities, attributes, relationships, and constraints.

Schema: The formal structure defining fields, types, relationships, and rules for a dataset.

Primary key: An attribute or combination of attributes that uniquely identifies a record.

Foreign key: An attribute that references a key in another table.

Normalization: The organization of relational data to reduce redundancy and update anomalies.

API: A defined interface through which software systems request or exchange data or services.

Metadata: Information describing a dataset’s structure, meaning, origin, status, or use.

Data catalogue: A searchable inventory containing metadata about available data assets.

Data quality: The degree to which data is fit for a specified purpose.

Data lineage: A record of data origin, movement, transformation, and use.

Data warehouse: A repository optimized for integrated, historical analytical querying.

Data lake: A repository that stores large volumes of data in varied formats, often in relatively raw form.

Data minimization: Collecting and using only the data necessary for a defined purpose.

Pseudonymization: Replacing direct identifiers with codes while retaining a controlled means of re-identification.

Data governance: The policies, roles, decision rights, and controls governing data assets.

Incremental extraction: Retrieving only new or changed records rather than copying all source data.

Data contract: An explicit agreement describing the structure, meaning, and service expectations of shared data.

Review Questions

Short Answer Questions

1. What distinguishes structured, semi-structured, and unstructured data?

2. Why are primary keys and foreign keys important in relational databases?

3. Define metadata and give two examples of metadata categories.

4. What is meant by data lineage?

5. Explain the difference between pseudonymization and anonymization.

Long Answer Questions

1. Explain the stages of a responsible data-acquisition workflow, from defining a use case to publishing an approved dataset.

2. Compare relational databases, data warehouses, data lakes, and NoSQL systems in terms of structure and typical use.

3. Discuss the main dimensions of data quality and describe suitable validation procedures.

4. Explain how privacy, security, and governance should influence the design of a data pipeline.

5. Discuss why data integration is a semantic problem as well as a technical problem, using the Northstar Stores case.

Analytical / Application Questions

1. A company receives customer data through an API, monthly Excel files, and a relational database. Design an acquisition and validation plan that preserves provenance and detects incomplete or inconsistent data.

2. An analyst joins a sales table to a product table and obtains totals twice as large as the finance report. Diagnose possible causes and propose tests to identify the error.

3. A university wants to use student activity logs to predict withdrawal risk. Recommend governance, privacy, quality, and ethical controls before modelling begins.

22. Multiple Choice Questions with Answers

Question 1

A JSON response contains nested customer objects and optional fields that vary between records. How should this data primarily be classified?

A. Structured data

B. Semi-structured data

C. Unstructured data

D. Relational data only

Correct Answer: B. Semi-structured data

Question 2

Which practice most directly supports reproducibility when extracting data from a database?

A. Deleting the original source after extraction

B. Recording the query, extraction time, and source version

C. Exporting every table without documentation

D. Replacing database keys with row numbers

Correct Answer: B. Recording the query, extraction time, and source version

Question 3

A data scientist needs only records added since the previous extraction. Which approach is most appropriate?

A. Full extraction of every historical record

B. Incremental extraction using a reliable change indicator

C. Manual copying from screen reports

D. Random sampling of current records

Correct Answer: B. Incremental extraction using a reliable change indicator

Question 4

Which statement best describes data lineage?

A. A list of users authorized to access a database

B. A measure of the number of missing values

C. A record of data origin, transformations, movement, and use

D. A method for encrypting stored data

Correct Answer: C. A record of data origin, transformations, movement, and use

Question 5

An analyst uses an inner join when combining customer and order tables. What should be checked first?

A. Whether customers without orders have been excluded

B. Whether all text has been converted to images

C. Whether the database has no primary keys

D. Whether the API rate limit has increased

Correct Answer: A. Whether customers without orders have been excluded

Question 6

Which action best reflects data minimization?

A. Collecting every available customer attribute for possible future use

B. Keeping personal identifiers when only aggregated demand is required

C. Collecting only the fields necessary for the defined analytical purpose

D. Publishing raw records so users can choose relevant fields

Correct Answer: C. Collecting only the fields necessary for the defined analytical purpose

Question 7

A dataset passes format checks but contains product codes that do not exist in the product master. Which quality dimension is most directly affected?

A. Referential consistency

B. Timeliness only

C. Compression efficiency

D. Encryption strength

Correct Answer: A. Referential consistency

Question 8

Why might a data warehouse deliberately contain denormalized tables?

A. To eliminate the need for any data definitions

B. To improve analytical query performance in suitable circumstances

C. To prevent all duplicate observations in source systems

D. To make transactional updates more complex

Correct Answer: B. To improve analytical query performance in suitable circumstances

Question 9

A streaming system may deliver the same event more than once. Which design concern is especially relevant?

A. Idempotent processing and duplicate handling

B. Spreadsheet formatting

C. Removal of all event timestamps

D. Avoidance of data validation

Correct Answer: A. Idempotent processing and duplicate handling

Question 10

Which statement about de-identification is most accurate?

A. It always guarantees that individuals cannot be re-identified

B. It removes the need for access controls

C. It can reduce privacy risk, but linkage with other data may still create re-identification risk

D. It means that data no longer requires a defined purpose

Correct Answer: C. It can reduce privacy risk, but linkage with other data may still create re-identification risk