3. Data Sources, Acquisition and Data Architecture

Introduction and context

Data analysis begins before a dataset is opened in a statistical package or visualised in a dashboard. The quality, relevance, structure, provenance and timeliness of the data determine what can be inferred from it. A sophisticated analytical method cannot compensate for a poorly defined population, biased sample, incomplete source, broken data pipeline or misunderstood data architecture.

Organisations now acquire data from operational databases, application programming interfaces, files, surveys, sensors, transaction systems, social and digital platforms, and third-party providers. These sources differ substantially. A relational database may enforce a precise schema, whereas social-media content may combine text, images, links and metadata. A sensor may produce high-volume time-series observations, while a survey may provide carefully designed but relatively infrequent responses. Effective data professionals must therefore evaluate not only the content of data but also its origin, collection mechanism, structure, legal basis, reliability and suitability for a decision.

This chapter examines the main types and sources of data, methods of acquisition, sampling and integration principles, and the architectures that support analytical work. It connects source-level decisions with data warehouses, data lakes, lakehouses, cloud analytics and data pipelines. The central argument is that data architecture is not merely an information-technology concern: it shapes analytical possibilities, organisational accountability and the credibility of evidence.

Learning objectives

By the end of this chapter, the learner should be able to:

1. Distinguish structured, semi-structured and unstructured data and evaluate their analytical implications.

2. Compare databases, APIs, files, surveys, sensors and digital platforms as data sources.

3. Design appropriate acquisition strategies, including sampling and data-quality controls.

4. Explain data integration, metadata, lineage and common architectural patterns.

5. Compare warehouses, lakes, lakehouses and cloud analytics environments.

6. Describe the stages and governance requirements of a modern data pipeline.

7. Identify technical, ethical, legal and organisational risks in data acquisition and use.

The nature of data and its sources

Structure as an analytical property

Structured data is organised according to a predefined model, usually rows and columns with specified data types. Customer accounts, invoices and examination records stored in relational tables are familiar examples. Structured data is comparatively easy to query, validate and join, provided that keys and definitions are reliable. Its apparent order, however, does not guarantee accuracy or fairness.

Semi-structured data does not conform fully to a tabular schema but contains tags, keys, delimiters or other organisational markers. JSON documents returned by an API, XML records and event logs are common examples. Different records may contain different attributes, and nested objects may require flattening before analysis. Semi-structured data offers flexibility but creates challenges involving schema evolution, missing fields and inconsistent representations.

Unstructured data lacks a fixed analytical schema. Documents, email, audio, video, images and free-form web content fall into this category. Analysis generally requires additional processing such as natural language processing, speech recognition, image analysis or information extraction. Unstructured data often contains valuable context, but interpretation is more dependent on models, annotation and domain knowledge.

The categories are not absolute. A digital photograph is unstructured as an image but may have structured metadata such as timestamp, location and device identifier. A customer-support conversation may contain free text alongside structured case fields. In practice, data estates are often multimodal, and useful analysis depends on relating these forms without confusing their different meanings.

Operational databases

Operational databases support routine business transactions. Relational database management systems use tables, constraints, keys and transaction controls to preserve consistency. They are designed for online transaction processing, or OLTP, where many users create and update individual records. Examples include banking transactions, inventory movements and hospital appointments.

Analysts should normally avoid placing intensive analytical workloads directly on production systems because complex queries may affect operational performance. Data may instead be replicated into an analytical store through scheduled extraction, change-data capture or streaming. Before use, the analyst should understand whether a field records a current state, a historical event or an application-specific code.

Non-relational databases are useful where data is highly variable, distributed or large in scale. Document stores manage flexible records, key-value stores support rapid lookup, column-family systems handle wide distributed tables, and graph databases represent entities and relationships. Choosing among them is an architectural decision based on access patterns, consistency requirements, scale and analytical needs rather than popularity alone.

APIs and digital services

An API provides a controlled interface through which one system requests data or functionality from another. APIs may return JSON, XML, tabular responses or streamed events. Acquisition requires attention to authentication, rate limits, pagination, versioning, status codes, quotas and terms of use. A successful request is not necessarily a successful acquisition: the response must also be checked for completeness, duplicate pages, changed field definitions and error messages embedded in otherwise valid responses.

API documentation should be preserved with the acquisition record. Analysts should record the endpoint, parameters, retrieval time, version, authentication method and transformation steps. Reproducibility is improved when raw responses are retained, subject to privacy and storage requirements.

Files and documents

CSV, spreadsheet, Parquet, text, image and log files remain important sources. Files are convenient but frequently contain hidden assumptions: multiple sheets may use different definitions, dates may be ambiguous, missing values may be encoded as blank strings or special symbols, and column names may change between reporting periods. Delimited text is portable but may lose types and formatting. Columnar formats such as Parquet are efficient for analytical processing because they permit selective reading of columns and support typed data.

File-based acquisition should include validation of filename, size, encoding, schema, row count, date range and checksum where appropriate. A data dictionary and a controlled landing area help distinguish an original file from a cleaned or transformed version.

Surveys and experiments

Survey data is generated through questions posed to a selected population. Its quality depends on the sampling frame, wording, mode of administration, response rate, non-response pattern and treatment of missing answers. Leading questions, double-barrelled questions and unclear recall periods can create measurement error. A survey may be statistically representative in design yet still suffer from non-response bias.

Experimental data is produced when treatments are deliberately assigned, ideally using randomisation and controls. It can support stronger causal claims than observational data, although practical, ethical and external-validity constraints remain. Analysts must distinguish association from causation when working with records that merely observe behaviour.

Sensors and machine-generated data

Sensors produce observations about physical or digital conditions, often as time series. Internet-of-Things devices, manufacturing equipment, mobile devices and application telemetry may generate data at high velocity. Important properties include sampling frequency, calibration, precision, timestamp source, unit of measurement, battery or connectivity failures and device identity.

Sensor readings are not direct reality; they are measurements produced by an instrument. Drift, noise, outliers and missing intervals should be investigated rather than automatically removed. A sudden change may indicate a genuine event, a maintenance action or a faulty device. Contextual metadata is therefore essential.

Digital platforms and third-party data

Search systems, e-commerce platforms, social networks, advertising systems and mobile applications produce behavioural and interaction data. Such data can reveal sequences, engagement and network relationships, but it often reflects platform design as much as user preference. Algorithms determine what is shown, recorded or made available. Third-party data may have uncertain provenance, changing coverage and contractual restrictions. Its apparent scale should not be mistaken for representativeness.

Acquisition as a controlled process

A defensible acquisition process starts with an analytical question and a data specification. The specification identifies the unit of analysis, required variables, time period, population, acceptable latency, quality thresholds, ownership and access constraints. Collecting everything is rarely efficient or ethical; collection should be proportionate to a legitimate purpose.

A typical workflow contains the following linked activities:

• identify and evaluate candidate sources;
• obtain authorisation and document purpose and conditions of use;
• extract or receive data into a controlled landing zone;
• preserve the original version where lawful and appropriate;
• profile structure, completeness, uniqueness, validity and timeliness;
• transform and integrate data using documented rules;
• publish an approved analytical dataset; and
• monitor quality, lineage, access and changes over time.

This workflow is iterative. Profiling may reveal that the source cannot answer the original question, requiring a revised specification or a different collection method.

Sampling and representativeness

Sampling is the selection of a subset from a target population. Probability sampling gives units a known, non-zero probability of selection. Simple random sampling, systematic sampling, stratified sampling and cluster sampling are common designs. Stratification can improve precision when important subgroups differ, while cluster sampling may reduce cost when geographically or organisationally grouped units are easier to reach.

Non-probability methods include convenience, purposive, quota and snowball sampling. These may be appropriate for exploratory or hard-to-reach populations, but estimation of population parameters requires caution. A large sample does not automatically remove selection bias. Coverage error occurs when the sampling frame excludes part of the population; non-response bias occurs when respondents differ systematically from non-respondents.

Sampling error reflects chance variation in a sample, whereas measurement and non-sampling errors arise from wording, recording, processing or coverage. Weighting can adjust for known differences in selection probabilities, but it cannot reliably repair an unknown or severely biased sampling process. Analysts should report the target population, frame, method, response rate, weighting and limitations.

Data quality and metadata

Data quality is multidimensional. Accuracy concerns correspondence with the phenomenon of interest; completeness concerns required values or records; consistency concerns agreement across representations; validity concerns conformity to rules; uniqueness concerns unwanted duplication; and timeliness concerns availability relative to use. Quality is fitness for purpose, so a small amount of missing data may be acceptable for one decision and unacceptable for another.

Metadata describes data. Business metadata explains meaning and ownership; technical metadata records types, formats and schemas; operational metadata records refresh time, pipeline status and usage; and provenance metadata describes origin and transformations. A data dictionary alone is insufficient if definitions, lineage and responsible owners are absent.

Integrating data from different sources

Data integration combines data so that it can be analysed coherently. The difficult part is usually semantic rather than mechanical. Two systems may use the same word differently, different words for the same concept, or incompatible units and time zones.

Extract, transform and load, or ETL, transforms data before loading it into a target store. Extract, load and transform, or ELT, loads relatively raw data into a powerful analytical platform and transforms it there. ETL can be useful when strict control and target-specific preparation are required; ELT supports flexible exploration and takes advantage of cloud computing. Neither is universally superior.

Integration commonly requires:

• schema mapping between source and target fields;
• entity resolution, such as determining that two customer records refer to one person;
• standardisation of dates, currencies, units and codes;
• deduplication and survivorship rules;
• management of slowly changing attributes;
• handling of late-arriving and corrected records; and
• preservation of keys, timestamps and lineage.

A star schema separates measurable facts, such as sales amount, from descriptive dimensions, such as product, customer and date. It supports understandable and efficient reporting. A normalised operational model reduces update anomalies but may require more joins for analysis. Data modelling should therefore reflect the intended use, not simply reproduce the source system.

From storage to architecture

Warehouses, lakes and lakehouses

A data warehouse is a curated analytical repository containing integrated, structured and historically oriented data. It commonly applies schema-on-write: data is modelled and validated before broad use. Warehouses support governed reporting, consistent metrics and high-performance SQL, but extensive preparation may slow the inclusion of novel or unstructured data.

A data lake stores large volumes of raw or lightly processed data in object storage. It supports schema-on-read, allowing structure to be applied when data is used. Lakes are valuable for exploration, machine learning, logs and multimodal content, but unmanaged lakes can become data swamps marked by duplication, uncertain quality and poor discoverability.

A lakehouse combines lake-style low-cost, flexible storage with warehouse-like management, transactional reliability, cataloguing and analytical performance. In practice, the distinction between products is less important than the capabilities provided: governance, interoperability, scalability, performance, security and usability.

Cloud analytics

Cloud analytics separates or loosely couples storage and computation, allowing resources to scale according to demand. Managed services can reduce infrastructure administration and provide distributed processing, orchestration, catalogues and machine-learning tools. However, cloud adoption introduces costs related to data movement, storage growth, compute consumption, vendor dependence, regional restrictions and configuration security.

A sound cloud architecture uses least-privilege access, encryption, network controls, environment separation, budgets, monitoring and lifecycle policies. Cost and performance should be treated as design variables from the beginning rather than as operational surprises.

Data pipelines and processing patterns

A data pipeline moves data from sources to destinations and performs required validation or transformation. Batch pipelines process accumulated data on a schedule. Streaming pipelines process events continuously or in small intervals, supporting use cases such as fraud detection and equipment monitoring. Event time, processing time, ordering, duplication, replay and late arrival are central streaming concerns.

A robust pipeline is observable. It records input and output counts, failures, latency, schema changes, quality-rule results and lineage. Idempotent processing allows a failed operation to be safely repeated without creating duplicate effects. Orchestration tools manage dependencies, retries, alerts and schedules, while data contracts specify expectations between producers and consumers.

Modern architectures may include a raw or bronze layer, a validated or silver layer, and a business-ready or gold layer. These names are conventions, not universal standards. Their value lies in separating preservation, refinement and consumption while making transformations understandable.

An illustrative case: integrating municipal mobility data

Background

Consider a fictional metropolitan transport authority seeking to reduce delays and improve bus reliability. It holds scheduled timetable data in a relational database, vehicle-location events from GPS devices, passenger counts from onboard sensors, service complaints from a web form and weather observations obtained through an API.

Situation and problem

Initial reports showed conflicting delay figures. The timetable used local time, GPS devices occasionally transmitted duplicate events, passenger counters were missing during equipment faults, and complaints represented only passengers motivated to submit them. Leaders wanted a dashboard and a model predicting congestion, but no shared definition of a trip, stop or delay existed.

Analysis and approach

The analytical team first defined the unit of analysis: a vehicle journey between scheduled stops. A data contract specified identifiers, timestamps, units, acceptable latency and required fields. GPS messages were retained in a raw lake area, validated for device and timestamp plausibility, and deduplicated using event identifiers. Timetable and route data were modelled as dimensions, while arrivals, boardings and delay intervals formed fact tables in a warehouse.

Time zones and daylight-saving changes were standardised. Sensor gaps were flagged rather than silently converted to zero. Complaints were analysed as a separate, self-selected source and were not treated as a population estimate of all passenger experiences. Weather observations were joined by location and time window, with the retrieval date and API version recorded.

A batch process refreshed historical performance each night, while a streaming process produced operational alerts for severe disruption. Quality monitoring checked event volume, missingness, duplicate rates, late arrivals and route-code changes. Access was restricted because some complaint records contained personal information.

Outcome

The resulting dashboard distinguished scheduled, observed and estimated measures. Managers could identify routes with persistent delays, while analysts could evaluate whether apparent improvements reflected route changes or missing sensors. The architecture supported both governed reporting and exploratory modelling without allowing unverified raw data to appear as official performance information.

Learning points

The case illustrates that integration is a meaning-management problem as much as a technical one. Clear units of analysis, metadata, source-specific limitations, layered storage, quality monitoring and governance were necessary before visualisation could be trusted.

Risks, governance and professional judgement

Common failures include collecting data without a defined purpose, treating missing values as zero, joining tables on non-unique names, overwriting raw data, ignoring time zones, assuming API completeness, using convenience samples as population evidence, and reporting model-generated estimates as observations. These errors may produce polished but misleading dashboards.

Privacy requires attention throughout the lifecycle. Personal data should be collected lawfully, used for specified purposes, minimised, protected and retained only as necessary under applicable regulation and organisational policy. Pseudonymisation reduces direct identification but does not guarantee anonymity, especially when datasets can be linked. Access controls, audit trails, encryption, secure transfer and breach procedures are technical and organisational necessities.

Governance should assign data owners, stewards and custodians; define quality standards; maintain catalogues and lineage; establish retention and deletion rules; and provide processes for correction and access. Ethical analysis also asks whether collection is coercive, whether affected groups are represented, whether surveillance is proportionate and whether automated decisions may create discriminatory outcomes. Consent, contractual permission and public availability are not interchangeable concepts.

Emerging directions and best practice

Data architectures increasingly support real-time analytics, data products, privacy-enhancing techniques, federated analysis, semantic layers and machine-learning operations. Generative AI can assist with schema discovery, documentation and code generation, but its outputs require validation and should not be allowed to invent lineage or silently alter definitions. Automated data-quality agents can detect anomalies, yet domain experts remain necessary to judge whether an anomaly reflects reality.

A practical best-practice checklist is to begin with the decision and population, document source limitations, preserve provenance, use stable identifiers, validate at ingestion, separate raw from curated data, automate repeatable transformations, monitor quality and cost, apply least-privilege access, and communicate uncertainty. The most valuable architecture is not the most elaborate one; it is the architecture that delivers reliable, explainable and appropriately governed data for its intended use.

Chapter Summary

Data may be structured, semi-structured, unstructured or multimodal, and each form creates different requirements for storage, preparation and analysis. Databases, APIs, files, surveys, sensors and digital platforms vary in reliability, coverage, timeliness, provenance and representativeness. Acquisition should begin with a clear analytical purpose and continue through authorisation, extraction, profiling, transformation, integration, publication and monitoring.

Sampling determines how evidence relates to a target population. Probability designs support stronger population inference, while non-probability designs require explicit limitations. Data integration depends on semantic definitions, entity resolution, standardisation, keys, time and lineage. Warehouses provide curated analytical consistency; lakes provide flexible raw storage; lakehouses combine capabilities; and cloud platforms offer scalable but economically and operationally complex environments. Pipelines may be batch or streaming and should be observable, repeatable and governed. Privacy, security, fairness, transparency and professional judgement are essential to trustworthy data analysis.

Key Terms

Structured data: Data organised in a predefined schema, commonly rows and columns.

Semi-structured data: Data containing organisational markers such as keys, tags or nested fields without a rigid tabular schema.

Unstructured data: Data without a fixed analytical structure, such as text, audio, video or images.

API: A programmatic interface through which one system requests data or services from another.

OLTP: Online transaction processing designed for frequent operational updates.

Metadata: Information describing data, including its meaning, structure, ownership and provenance.

Sampling frame: The operational list or mechanism from which a sample is selected.

Coverage error: Bias caused when the sampling frame omits part of the target population.

Non-response bias: Bias arising when respondents differ systematically from non-respondents.

ETL: Extract, transform and load; transformation occurs before loading into the target system.

ELT: Extract, load and transform; raw or lightly processed data is loaded before transformation.

Entity resolution: The process of determining which records refer to the same real-world entity.

Data warehouse: A governed, integrated repository optimised for analytical querying.

Data lake: A repository that stores large volumes of raw or lightly processed data, often using schema-on-read.

Lakehouse: An architecture combining flexible lake storage with warehouse-style management and performance.

Data lineage: A record of data origin, movement and transformation.

Data pipeline: A sequence of processes that moves, validates and transforms data.

Idempotence: The property that repeating an operation produces the same intended result without unwanted duplication.

Data contract: An explicit agreement about a dataset's schema, meaning, quality and delivery expectations.

Data governance: Policies, roles and controls that ensure data is managed responsibly and effectively.

Review Questions

Short Answer Questions

1. Distinguish structured, semi-structured and unstructured data.

2. What information should be recorded when acquiring data from an API?

3. Explain the difference between sampling error and coverage error.

4. What is data lineage, and why is it useful?

5. Define ETL and ELT.

Long Answer Questions

1. Compare operational databases, data warehouses and data lakes in terms of purpose, structure and analytical use.

2. Explain how a survey's sampling design and response process affect the validity of its findings.

3. Discuss the principal stages of a controlled data-acquisition workflow.

4. Explain the technical and semantic challenges involved in integrating data from multiple sources.

5. Describe the architecture and governance requirements of a reliable batch and streaming data pipeline.

Analytical / Application Questions

1. A retailer combines point-of-sale records, mobile-app events and third-party demographic data. Design an integration plan that addresses identifiers, provenance, quality and privacy.

2. A sensor system reports a sudden fall in production output, but half of the devices stopped transmitting. Explain how you would determine whether the fall is real and how it should be reported.

3. A city uses voluntary online complaints to measure public transport satisfaction. Evaluate the sampling limitations and propose a more defensible evidence strategy.

22. Multiple Choice Questions with Answers

Question 1

Which characteristic most clearly distinguishes semi-structured data from structured data?

A. It cannot contain metadata

B. It uses organisational markers but does not require a rigid tabular schema

C. It is always generated by sensors

D. It cannot be queried computationally

Correct Answer: B. It uses organisational markers but does not require a rigid tabular schema

Question 2

An API request returns the correct status code but only the first 100 records. Which acquisition issue is most likely?

A. Lack of encryption

B. Failure to handle pagination

C. Entity resolution

D. Schema-on-write

Correct Answer: B. Failure to handle pagination

Question 3

A survey excludes people without internet access, although they belong to the target population. This is primarily an example of:

A. Coverage error

B. Processing latency

C. Duplicate measurement

D. Randomisation

Correct Answer: A. Coverage error

Question 4

Which approach is most appropriate for protecting production performance while enabling complex historical analysis?

A. Running all analytical queries directly on the OLTP database

B. Deleting old operational records

C. Replicating or transforming operational data into an analytical store

D. Replacing all structured data with images

Correct Answer: C. Replicating or transforming operational data into an analytical store

Question 5

A data lake becomes a data swamp chiefly when:

A. It stores any raw data

B. It lacks discoverability, quality controls and clear ownership

C. It uses object storage

D. It supports machine learning

Correct Answer: B. It lacks discoverability, quality controls and clear ownership

Question 6

Which statement best describes ELT?

A. Data is transformed before extraction

B. Data is extracted, loaded into the analytical platform, and transformed there

C. Only streaming data can use it

D. It eliminates the need for metadata

Correct Answer: B. Data is extracted, loaded into the analytical platform, and transformed there

Question 7

A pipeline is designed so that rerunning a failed load does not create duplicate records. This property is called:

A. Latency

B. Stratification

C. Idempotence

D. Denormalisation

Correct Answer: C. Idempotence

Question 8

Why should a sensor reading not automatically be replaced with zero when it is missing?

A. Zero is never a valid value

B. The missingness may represent equipment failure rather than a measured zero

C. Sensors cannot produce numeric values

D. Zero prevents schema evolution

Correct Answer: B. The missingness may represent equipment failure rather than a measured zero

Question 9

Which architecture most directly combines flexible raw storage with warehouse-like governance and analytical management?

A. Lakehouse

B. OLTP-only system

C. Spreadsheet network

D. Sampling frame

Correct Answer: A. Lakehouse

Question 10

A complaints dataset is used to estimate satisfaction for all passengers, but only dissatisfied users tend to submit complaints. The main analytical danger is:

A. Non-response or self-selection bias

B. Excessive encryption

C. Columnar storage

D. Successful random sampling

Correct Answer: A. Non-response or self-selection bias