Data Sources, Structures and Acquisition

Introduction

Data preparation begins before a dataset is downloaded, queried, or loaded into a programming environment. Analysts must first understand where data originates, how it is represented, why it was collected, who controls it, and what limitations may affect its use. A technically clean dataset can still produce misleading conclusions if it is irrelevant, biased, poorly documented, duplicated, or detached from its original context.

In contemporary data science, information is acquired from relational databases, spreadsheets, documents, application programming interfaces (APIs), sensors, websites, social platforms, operational systems, research studies, and commercial providers. These sources differ in structure, update frequency, accessibility, reliability, ownership, and ethical risk. A sound acquisition strategy therefore combines technical knowledge with statistical reasoning, domain expertise, governance, and critical evaluation.

This chapter develops a framework for evaluating data sources and structures. It distinguishes structured, semi-structured, and unstructured data; explains transactional, master, observational, and experimental data; examines acquisition methods; and introduces provenance, relevance, quality, and risk assessment.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish structured, semi-structured, and unstructured data.
2. Evaluate databases, files, APIs, sensors, platforms, and external providers.
3. Differentiate transactional, master, observational, and experimental data.
4. Assess data provenance, relevance, quality, accessibility, and acquisition risk.
5. Design an appropriate acquisition strategy for a data science project.
6. Recognise ethical, legal, operational, and statistical consequences of source selection.

1. Data as a Representational and Organizational Concept

Data is a recorded representation of entities, events, measurements, interactions, or observations. A useful distinction is between:

• Data value: the recorded content, such as 42.7 or “approved”.
• Metadata: information describing the data, such as units, definitions, source, timestamp, and permitted values.
• Schema: the formal organization of fields, records, relationships, and constraints.
• Context: the circumstances in which the data was generated and the purpose for which it was collected.

A dataset should not be treated as a neutral mirror of reality. It is the result of measurement, classification, recording, storage, and selection decisions. For example, a customer database may represent “customer” as a person who has completed a purchase, while a marketing platform may define a customer as anyone who has submitted an email address. The same label can therefore describe different populations.

Data acquisition is the process of obtaining data for analysis while preserving sufficient information about its origin, meaning, transformations, and permitted use. Acquisition includes discovery, authorization, extraction, transfer, validation, storage, and documentation.

2. Data Structures

2.1 Structured Data

Structured data follows a defined schema, commonly represented as rows and columns. Relational database tables are the canonical example. Each row usually represents an entity or event, and each column represents an attribute.

A relational table may contain a primary key that uniquely identifies each row and foreign keys that refer to records in other tables. Constraints may enforce uniqueness, valid data types, mandatory fields, or referential integrity.

Structured data supports efficient querying and joining. SQL allows analysts to filter, aggregate, sort, and combine tables. However, structural regularity does not guarantee validity. A table can contain inaccurate, outdated, duplicated, or systematically biased records.

Examples include:

• sales transactions;
• payroll records;
• inventory tables;
• banking account records;
• examination results; and
• standardized survey responses.

2.2 Semi-structured Data

Semi-structured data does not conform to one rigid tabular schema but contains organizational features such as keys, tags, nesting, or delimiters. JSON, XML, YAML, application logs, and many event streams are common examples.

A JSON record might contain a customer identifier, an address object, a list of purchases, and optional fields that appear only for some customers. This flexibility is valuable when systems evolve or when different observations contain different attributes. It also creates analytical difficulties: nested objects must be flattened, data types may vary, and missing fields may be ambiguous.

Semi-structured data requires attention to:

• schema evolution;
• optional versus missing attributes;
• nested arrays and objects;
• inconsistent naming conventions;
• version differences; and
• encoding and parsing errors.

2.3 Unstructured Data

Unstructured data lacks a predetermined tabular organization. Text documents, emails, photographs, audio, video, scanned forms, and some social-media content are examples. “Unstructured” does not mean that no structure exists. Language, visual objects, speakers, timestamps, and document layouts provide latent or extractable structure.

Analysis often requires additional processing such as optical character recognition, natural-language processing, speech recognition, image classification, or embedding generation. These transformations may introduce errors and should be documented as part of provenance.

2.4 A Practical Comparison

The categories are not absolute. A database may store structured fields alongside documents, and a JSON record may contain both highly regular and highly variable elements. Classification should therefore reflect the analytical task, not only the storage technology.

3. Major Data Sources

3.1 Relational and Analytical Databases

Operational databases support routine business processes such as orders, payments, reservations, and patient administration. They are often optimized for frequent inserts and updates. Analytical warehouses and data lakes are optimized for reporting, historical analysis, or large-scale processing.

When acquiring database data, an analyst should establish:

• table and column definitions;
• primary and foreign keys;
• update and deletion policies;
• time zones and timestamp semantics;
• historical retention;
• access privileges;
• query cost and performance limits; and
• whether data is a current snapshot or a historical extract.

A major danger is joining tables without understanding grain. The grain is the level represented by one row, such as one order, one order line, one customer, or one daily measurement. Joining an order table to an order-line table can intentionally create multiple rows per order; failing to account for this may inflate revenue or counts.

3.2 Files

Files remain a common source because they are portable and easy to exchange. CSV files are convenient but underspecified: delimiters, quoting, character encoding, decimal conventions, date formats, and missing-value symbols may vary. Spreadsheet files may contain formulas, hidden rows, multiple tables, manual adjustments, and presentation formatting that obscures the intended schema.

Good practice includes preserving the original file, recording its checksum or version, inspecting encoding and delimiters, and validating row counts and data types after import.

3.3 APIs

An API provides programmatic access to data or services. APIs may expose REST endpoints, GraphQL queries, streaming interfaces, or specialized protocols. Important acquisition considerations include authentication, authorization, pagination, rate limits, response schemas, versioning, status codes, retries, and usage restrictions.

An API response is not necessarily a complete dataset. It may represent only the latest records, a filtered view, a sample, or records visible to the requesting account. Analysts should document query parameters and retrieval times so that the extraction can be reproduced.

3.4 Sensors and Internet-of-Things Devices

Sensors generate observations of physical phenomena such as temperature, location, pressure, vibration, or heart rate. Sensor data is often time-stamped and high-volume, but it may contain drift, calibration error, outages, duplicated messages, irregular sampling, and transmission delays.

The recorded value should be distinguished from the underlying phenomenon. A sensor reading is a measurement produced by an instrument under conditions. Metadata should include device identity, location, units, calibration history, sampling interval, firmware version, and quality flags.

3.5 Platforms and Digital Traces

Websites, mobile applications, social networks, advertising platforms, and payment systems generate digital traces. These may include clicks, views, sessions, searches, posts, reactions, and device events. Platform data is often abundant but shaped by platform design, ranking algorithms, user incentives, moderation rules, and access policies.

A platform metric such as “engagement” is operationally defined by the platform. It may not correspond to attention, satisfaction, or influence. Analysts should avoid treating proprietary metrics as self-explanatory scientific variables.

3.6 External Providers and Open Data

External providers may supply demographic, economic, geospatial, credit, industry, weather, or market data. Providers can offer valuable coverage and specialized expertise, but acquisition introduces contractual, financial, legal, and methodological dependencies.

Before purchase or use, evaluate licensing, update schedules, geographic coverage, population definitions, revision policies, quality guarantees, support arrangements, and the provider’s data-collection methods. Open access does not necessarily mean unrestricted reuse. Terms may limit redistribution, commercial use, or personal-data processing.

4. Data Classifications by Business and Research Role

Structure describes how data is represented. Another classification describes what role the data plays.

4.1 Transactional Data

Transactional data records events or exchanges, such as purchases, payments, bookings, claims, or account changes. It usually has an event time, an entity or entities involved, and an outcome or amount.

Transactions may be amended, cancelled, refunded, or reversed. Therefore, the latest row is not always the final truth. Analysts must understand event states, correction records, and whether totals are recorded gross or net of discounts, taxes, returns, and fees.

4.2 Master Data

Master data describes relatively stable entities shared across processes: customers, products, suppliers, employees, locations, or accounts. It provides reference information for interpreting transactions.

Master data is rarely perfectly stable. Customers change addresses, products are reclassified, and organizations merge. Slowly changing dimensions and historical versions are important when analysis requires the attributes that were valid at the time of an event rather than the attributes known today.

4.3 Observational Data

Observational data is recorded without the researcher assigning the exposure or intervention. Examples include medical records, website behavior, census data, traffic observations, and natural environmental measurements.

Observational data is often realistic and large-scale, but associations may be confounded by variables related to both exposure and outcome. Data availability can also reflect who interacted with a system, sought treatment, owned a device, or consented to collection.

4.4 Experimental Data

Experimental data arises when an investigator or organization deliberately assigns an intervention, treatment, policy, interface, or condition according to a design. Randomized experiments can support stronger causal conclusions when implementation is valid and outcomes are measured appropriately.

Experiments still require careful acquisition. Treatment assignment, exposure, non-compliance, attrition, spillover, timing, and protocol deviations must be recorded. A randomized design does not repair inaccurate measurement or selective missing outcomes.

These categories can overlap. A clinical trial may generate experimental treatment data, transactional billing data, and master patient data simultaneously.

5. Provenance and Data Lineage

Provenance is the documented history of data origin, collection, transformation, and use. Lineage emphasizes the path from source through processing stages to an analytical output. Provenance supports reproducibility, accountability, debugging, and defensible interpretation.

A useful provenance record answers:

1. Who created or collected the data?
2. When and where was it collected?
3. Under what protocol, instrument, or business process?
4. What population or process does it represent?
5. What transformations have occurred?
6. Which versions, filters, and queries were used?
7. What permissions and restrictions apply?
8. What known errors, missingness, or revisions exist?

A minimal acquisition log should record source identifier, retrieval timestamp, query or file version, software version, schema version, checksum where feasible, responsible person or system, and validation results.

Provenance is not merely administrative. Suppose a model trained on sales data performs poorly after a pricing-system change. Lineage may reveal that the meaning of “discount” changed halfway through the historical period. Without provenance, the change may be mistaken for a genuine market effect.

6. Assessing Relevance and Data Quality

A source is relevant when it measures the concepts required for the decision or research question, for the appropriate population, period, and unit of analysis. Relevance should be assessed before convenience.

A practical evaluation examines:

• Coverage: Does the source include the target population or process?
• Granularity: Are observations sufficiently detailed in time, space, and entity level?
• Timeliness: Is the data current enough for the intended use?
• Validity: Do variables measure what their labels imply?
• Completeness: Are required records and fields present?
• Accuracy: Are values close to the underlying quantities or classifications?
• Consistency: Do definitions and formats remain stable?
• Uniqueness: Are duplicate entities or events controlled?
• Accessibility: Can authorized users obtain and use the data reliably?
• Interoperability: Can it be combined with other sources?

A simple completeness rate for a field can be expressed as:

The calculation assumes that the set of expected records is known and that a blank, null, or unavailable value has been consistently identified. Completeness does not imply accuracy or representativeness.

For example, if 9,600 of 10,000 expected customer records contain a usable postal code, completeness is 96%. This says nothing about whether the postal codes are correct or whether customers without postal codes differ systematically from others.

7. Acquisition Risk

Acquisition risk is the possibility that obtaining or using a source will create analytical, operational, legal, ethical, financial, or security problems. Important risk categories include:

• Representativeness risk: the source excludes relevant groups or overrepresents accessible users.
• Measurement risk: variables are inaccurate, ambiguously defined, or affected by instrument failure.
• Temporal risk: definitions, systems, populations, or collection practices change over time.
• Access risk: credentials, pricing, rate limits, or provider decisions interrupt availability.
• Security risk: data may be exposed during transfer, storage, or processing.
• Privacy risk: individuals may be identifiable or harmed by inappropriate use.
• Legal and contractual risk: collection or reuse may violate law, license, consent, or contract.
• Vendor risk: a provider may discontinue a feed, revise historical values, or change its methodology.
• Reproducibility risk: dynamic sources may not produce the same result on later retrieval.

A useful professional response is to maintain a source-risk register. Each source can be rated by impact and likelihood, with controls such as validation checks, backup providers, contractual review, access controls, encryption, retention limits, and documented fallback procedures.

8. Designing an Acquisition Strategy

Acquisition should be driven by the analytical question rather than by the most convenient available file. The following sequence is effective:

Step 1: Define the decision and target population

Specify the outcome, relevant entities, time period, geography, and unit of analysis. Clarify whether the objective is description, prediction, monitoring, or causal evaluation.

Step 2: Identify required variables and their meanings

Create a data dictionary that defines names, types, units, valid ranges, missing-value conventions, and business rules. Distinguish directly observed variables from derived indicators.

Step 3: Search for candidate sources

Consider internal systems, public data, research repositories, APIs, sensors, surveys, and commercial providers. Assess whether the source measures the intended construct rather than merely offering a convenient proxy.

Step 4: Confirm authority and permissions

Check ownership, lawful basis, consent, license, contractual terms, retention requirements, and restrictions on linkage or redistribution.

Step 5: Obtain a sample or metadata before full extraction

Inspect schema, record examples, distributions, timestamps, missingness, and documentation. A small controlled extraction can reveal problems before substantial cost is incurred.

Step 6: Acquire reproducibly

Use versioned scripts or documented procedures where possible. Preserve raw data as read-only, record retrieval details, and separate raw, standardized, and analytical layers.

Step 7: Validate immediately

Check file integrity, row counts, uniqueness, referential integrity, data types, ranges, timestamps, and expected totals. Compare with an independent control total where available.

Step 8: Document limitations and approve use

Record known biases, unresolved ambiguities, transformations, and risks. Obtain appropriate review for sensitive or high-impact applications.

9. Worked Application: Retail Demand Data

Consider an illustrative retailer seeking to forecast weekly demand for household products.

Background and problem

The retailer wants forecasts by store and product. Available sources include point-of-sale transactions, product master data, store master data, promotional calendars, inventory records, weather data, and an external demographic provider.

Analysis of sources

Point-of-sale data is transactional and structured. Its grain is ideally one product-store transaction line, but cancelled sales and returns must be represented correctly. Product and store tables are master data. Promotional calendars are semi-structured because different campaigns contain different attributes. Weather data is observational, time-indexed, and externally provided.

The team discovers that product identifiers changed after a packaging redesign. A current product master table cannot correctly interpret older transactions. Historical product versions are therefore required. The team also finds that recorded sales fall when inventory is unavailable. Observed sales consequently measure purchases, not unconstrained customer demand.

Approach and solution

The team establishes a canonical product identifier, preserves historical mappings, and aggregates transaction lines to store-product-week after handling returns and cancellations. It joins promotions using effective dates rather than current campaign labels. Inventory availability is retained as a feature and used to identify periods in which sales were censored by stockouts.

Weather data is acquired through a versioned API process. Retrieval dates, API parameters, units, and provider revisions are logged. External demographic data is used only at an aggregated store-area level after licensing and privacy review.

Outcome

The resulting analytical dataset has a documented grain, source lineage, temporal alignment, and quality checks. Forecast performance improves not merely because a sophisticated model was chosen, but because source definitions and acquisition errors were addressed. The team also reports that forecasts are less reliable for newly opened stores because historical observations are sparse.

Learning points

1. A source must be interpreted at its true grain.
2. Master-data versioning is essential for historical analysis.
3. Availability constraints can distort observed outcomes.
4. External data requires methodological and contractual review.
5. Provenance improves both model development and managerial trust.

10. Ethical, Legal, and Governance Considerations

Data acquisition can affect people who did not directly participate in the analytical project. Sensitive personal data may include health, financial, biometric, precise location, or inferred behavioral information. Ethical acquisition requires purpose limitation, data minimization, fairness, transparency, security, and appropriate retention.

Consent should be meaningful where required, but consent alone does not eliminate risks of re-identification, discrimination, or unexpected secondary use. Data protection obligations vary by jurisdiction, and organizations must consult applicable law and institutional policy. Analysts should not assume that publicly visible data is ethically unrestricted.

Linking datasets increases analytical value but also increases re-identification risk and the possibility of incorrect matches. Identifiers should be governed carefully, and linkage quality should be evaluated rather than assumed. Access should follow least privilege: users receive only the permissions required for their role.

For high-impact decisions such as credit, employment, healthcare, education, or public services, source selection should include fairness and harm assessment. A dataset may be statistically large yet systematically exclude vulnerable populations. Documentation should disclose such limitations to decision-makers.

11. Contemporary Developments

Modern data architectures increasingly combine warehouses, data lakes, lakehouses, streaming systems, and event platforms. These technologies support large-scale and near-real-time acquisition but make schema management, lineage, quality monitoring, and cost control more important.

AI systems have increased demand for large collections of text, images, audio, code, and interaction logs. Their usefulness depends on provenance, licensing, annotation quality, duplication control, and representation of relevant populations. Synthetic data can support testing or privacy protection, but it does not automatically reproduce the statistical properties, rare cases, or social biases of real data.

Data contracts are an emerging organizational practice in which producers and consumers agree on schemas, definitions, quality expectations, ownership, and change procedures. Automated observability tools can monitor freshness, volume, distributions, schema changes, and failed pipelines. These tools assist governance but do not replace domain judgment.

Best-Practice Checklist

Before approving a data source, ask:

• What exactly does one record represent?
• Who generated the record and for what purpose?
• Which population is included or excluded?
• Are definitions, units, and time zones documented?
• Can the source be legally and ethically used?
• What changes have occurred over time?
• How will extraction be reproduced?
• What quality checks will detect failure?
• What happens if the provider or system becomes unavailable?
• Can the resulting analysis be explained and defended?

Chapter Summary

Data acquisition is a methodological activity, not a mechanical prelude to analysis. Structured data provides explicit schemas, semi-structured data provides flexible organization, and unstructured data requires extraction of latent structure. Databases, files, APIs, sensors, platforms, and external providers each offer distinct benefits and risks.

Transactional data records events, master data describes shared entities, observational data records naturally occurring conditions, and experimental data arises from assigned interventions. These categories can coexist in one project and should be distinguished from representational structure.

Provenance and lineage preserve the history of data origin and transformation. Relevance depends on population, time, unit, granularity, and construct validity; completeness alone is insufficient. Acquisition risk includes representativeness, measurement, access, security, privacy, legal, vendor, and reproducibility risks.

A defensible acquisition strategy defines the analytical purpose, evaluates candidate sources, confirms permissions, inspects samples, acquires reproducibly, validates immediately, and documents limitations. Strong data science therefore begins with critical source evaluation and accountable data stewardship.

Key Terms

API: A programmatic interface through which software requests data or services.

Acquisition: The process of discovering, obtaining, transferring, validating, and documenting data.

Data lineage: The recorded path of data from source through transformations to outputs.

Data lake: A storage environment designed to retain data in diverse formats, often including raw data.

Data warehouse: A structured analytical repository designed for integrated querying and reporting.

Experimental data: Data generated when an intervention or condition is deliberately assigned.

Grain: The entity, event, or level represented by one record in a dataset.

Master data: Relatively stable reference information about entities such as products or customers.

Metadata: Information describing data, including definitions, units, source, and timestamps.

Observational data: Data recorded without researcher-controlled assignment of exposure or treatment.

Provenance: The history of data origin, collection, transformation, and use.

Representativeness: The extent to which a dataset reflects the target population or process.

Schema: The formal organization of fields, records, relationships, and constraints.

Semi-structured data: Data with flexible organization using keys, tags, nesting, or delimiters.

Structured data: Data organized according to an explicit schema, commonly rows and columns.

Transactional data: Data recording events or exchanges such as purchases or payments.

Unstructured data: Data without a predetermined tabular organization, such as text, images, or audio.

Validity: The extent to which a variable measures the intended concept.

Versioning: Maintaining identifiable historical versions of data, definitions, schemas, or records.

Review Questions

Short Answer Questions

1. What is the difference between structured and semi-structured data?
2. Why is the grain of a database table important when joining tables?
3. Define data provenance and state two reasons it matters.
4. Distinguish transactional data from master data.
5. Why does high completeness not guarantee high data quality?

Long Answer Questions

1. Compare databases, APIs, sensors, and external providers as sources of data. Discuss their benefits and risks.
2. Explain the differences among observational and experimental data, including implications for causal interpretation.
3. Develop a framework for evaluating the relevance and quality of a candidate dataset.
4. Describe a reproducible data-acquisition workflow from source discovery through validation.
5. Discuss ethical, legal, privacy, and governance issues in acquiring platform or personal data.

Analytical / Application Questions

1. A company joins one customer row to multiple transaction rows and reports that the number of customers has doubled. Diagnose the likely error and propose validation steps.
2. A sensor network shows a sudden reduction in readings after a firmware update. Design an investigation using provenance, metadata, and quality checks.
3. Select and justify data sources for a project predicting hospital readmission. Address data structure, data role, relevance, acquisition risk, privacy, and fairness.

Multiple Choice Questions with Answers

Question 1

Which characteristic most directly distinguishes structured data from semi-structured data?

A. Structured data is always more accurate.

B. Structured data follows a more explicit and rigid schema.

C. Structured data cannot contain missing values.

D. Structured data is always stored in the cloud.

Correct Answer: B. Structured data follows a more explicit and rigid schema.

Question 2

A record describing a product name, category, and supplier is best classified as which type of data?

A. Master data

B. Experimental data

C. Transactional data

D. Streaming data

Correct Answer: A. Master data

Question 3

What is the primary purpose of data provenance?

A. To increase the number of variables

B. To eliminate all missing values

C. To document origin, transformations, and use

D. To guarantee causal conclusions

Correct Answer: C. To document origin, transformations, and use

Question 4

Which issue is especially associated with sensor data?

A. Calibration drift

B. Foreign-key enforcement

C. Spreadsheet formatting

D. Contractual licensing only

Correct Answer: A. Calibration drift

Question 5

If 9,600 of 10,000 expected records have a usable value for a field, what is the completeness rate?

A. 9.6%

B. 40%

C. 94%

D. 96%

Correct Answer: D. 96%

Question 6

Which statement about observational data is most accurate?

A. It always results from random assignment.

B. It records conditions without researcher-controlled assignment.

C. It cannot contain timestamps.

D. It is necessarily unstructured.

Correct Answer: B. It records conditions without researcher-controlled assignment.

Question 7

Why can joining an order table to an order-line table inflate totals?

A. The join changes all numeric values to text.

B. One order may match multiple order-line records.

C. Foreign keys remove duplicate records.

D. Master data cannot be joined to transactions.

Correct Answer: B. One order may match multiple order-line records.

Question 8

Which API feature helps an analyst reproduce an extraction?

A. Undocumented default filters

B. A recorded query, retrieval time, and API version

C. Randomly changing page sizes

D. Deleting the raw response

Correct Answer: B. A recorded query, retrieval time, and API version

Question 9

Which is an example of acquisition risk?

A. A documented data dictionary

B. A stable primary key

C. A provider discontinuing a critical data feed

D. A validated timestamp format

Correct Answer: C. A provider discontinuing a critical data feed

Question 10

Why is a current product master table potentially insufficient for historical retail analysis?

A. Products never change attributes.

B. Historical identifiers and classifications may differ from current ones.

C. Transactions cannot contain product identifiers.

D. Master data is always unstructured.

Correct Answer: B. Historical identifiers and classifications may differ from current ones.