Data Sources, Collection and Management

Introduction

Data analysis begins before any chart, statistical test, or machine-learning model is created. Analysts must first identify where data come from, how they were collected, what they represent, and whether they can be trusted. Poorly collected data cannot usually be rescued by sophisticated algorithms. In contrast, carefully designed data collection and management create a reliable foundation for analysis, visualization, and artificial intelligence.

This chapter examines structured, semi-structured, and unstructured data; common sources such as files, databases, APIs, surveys, sensors, and digital platforms; sampling and study design; metadata and data lineage; and the major dimensions of data quality. It also introduces practical workflows for acquiring, documenting, storing, validating, protecting, and preparing data for analysis.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish structured, semi-structured, and unstructured data.
2. Select suitable data sources for an analytical problem.
3. Explain the difference between observational and experimental data.
4. Describe probability and non-probability sampling methods.
5. Design a basic data-collection and management workflow.
6. Interpret metadata and trace data lineage.
7. Assess data quality using recognized dimensions.
8. Identify ethical, privacy, security, and governance requirements.

1. Understanding Data as an Analytical Resource

Data are recorded representations of observations, events, measurements, transactions, or communications. A value such as 27 is not meaningful by itself. Its meaning depends on context: 27 degrees Celsius, 27 customers, or an age of 27 are different data values. Therefore, useful data require both values and information about those values.

A practical data resource has at least four components:

• Content: the recorded values, text, images, sound, or signals.
• Structure: the organization and relationships among values.
• Context: information explaining what the data represent.
• Provenance: evidence about their origin and transformation history.

In AI and machine learning, these components affect feature engineering, model validity, bias assessment, reproducibility, and deployment risk. Data that are statistically large but poorly defined may be less useful than a smaller, well-designed dataset.

2. Types of Data by Structure

2.1 Structured data

Structured data follow a defined schema. They are commonly arranged in tables with rows representing records and columns representing attributes. Examples include student records, sales transactions, and inventory tables.

A relational table may contain a unique identifier, such as student_id, and fields such as programme, enrolment date, and assessment score. Structured data are convenient for SQL queries, validation rules, aggregation, and visualization.

A table should normally satisfy clear rules. Each column should have a defined data type, each record should have a stated meaning, and keys should identify records reliably. A database may also enforce constraints such as uniqueness, allowable ranges, and relationships between tables.

2.2 Semi-structured data

Semi-structured data do not use one rigid tabular schema, but contain organizational features such as keys, tags, or nested objects. JSON, XML, email headers, and many application logs are examples.

For example, one JSON record might contain a customer name and address while another contains an additional loyalty-status field. This flexibility helps software systems evolve, but creates analytical challenges. Analysts may need to flatten nested structures, manage missing fields, and reconcile changing names or data types.

2.3 Unstructured data

Unstructured data lack a fixed analytical format. Text documents, photographs, audio, video, scanned forms, and social-media posts are common examples. They often require specialized processing, such as optical character recognition, natural-language processing, speech recognition, image analysis, or manual coding.

Unstructured does not mean unusable. It means that the analytical structure must often be created during preparation. A photograph may become a record containing detected objects, confidence scores, and location metadata; a review may become sentiment, topic, and language features.

3. Major Data Sources

3.1 Files

Files include CSV, spreadsheet, JSON, XML, Parquet, text, image, and audio files. They are inexpensive and easy to exchange, making them common in small projects and data pipelines. However, file-based data often suffer from ambiguous column names, undocumented revisions, duplicate copies, inconsistent date formats, and accidental overwriting.

A CSV file should be accompanied by a data dictionary describing each column, units, permitted values, encoding, missing-value conventions, and time zone. File names should include meaningful version or date information, but version control should not rely only on filenames.

3.2 Databases and data warehouses

Operational databases support routine transactions such as orders or registrations. Analytical databases and warehouses are optimized for queries across large historical datasets. A database provides controlled access, constraints, indexing, backup, and concurrent use.

A relational database represents entities in related tables. A primary key identifies a record within a table, while a foreign key links it to a record in another table. Good database design reduces unnecessary duplication, although analytical systems may deliberately use denormalized structures for performance.

3.3 Application programming interfaces

An API is a defined interface through which one software system requests data or services from another. APIs may return JSON or XML and commonly require authentication tokens. Analysts must examine documentation for endpoint meaning, pagination, rate limits, timestamp conventions, error responses, and licensing conditions.

A successful API response does not guarantee a valid dataset. The analyst should record the request time, endpoint, parameters, response status, and software version. Repeated requests may return different results if the source is updated.

3.4 Surveys and questionnaires

Surveys collect information directly from respondents. They can measure opinions, self-reported behaviour, experiences, or characteristics that may not exist in administrative records. Their quality depends on population definition, question wording, response options, ordering, mode of administration, and non-response management.

Leading questions, double-barrelled questions, unexplained technical language, and unbalanced response scales can produce measurement error. A pilot survey can reveal confusing wording and excessive completion time. Sensitive questions require clear consent and appropriate protection.

3.5 Sensors and Internet-of-Things devices

Sensors generate observations such as temperature, pressure, location, acceleration, or energy use. Sensor data are often time series and may arrive at regular or irregular intervals. Important metadata include device identity, calibration status, measurement unit, sampling interval, location, time zone, and firmware version.

Sensor data may contain noise, drift, missing intervals, duplicated transmissions, or impossible readings. A value outside an expected range is not automatically an error: it may represent a genuine event. Validation should combine domain rules with examination of the measurement process.

3.6 Digital platforms

Websites, mobile applications, transaction systems, search systems, and social platforms generate behavioural and interaction data. These data can be detailed and timely, but they are shaped by platform design. Clicks measure recorded interaction, not necessarily understanding or satisfaction. Users who do not access a platform are absent from the dataset, creating coverage bias.

Platform data may also change when interface designs, recommendation algorithms, privacy settings, or tracking technologies change. Such changes must be recorded as potential breaks in comparability.

4. Study Design: Observational and Experimental Data

4.1 Observational data

In an observational study, researchers record conditions without assigning an intervention. Examples include analysing hospital admissions, examining customer churn, or observing traffic flows. Observational data are valuable for describing real-world behaviour, but association does not by itself establish causation.

Confounding occurs when a third variable is related to both a presumed cause and an outcome. For example, an association between exercise-app use and health may partly reflect age, income, or prior health motivation. Statistical adjustment can reduce confounding, but cannot automatically remove all unmeasured differences.

4.2 Experimental data

In an experiment, investigators assign an intervention or treatment according to a protocol and measure outcomes. Random assignment, when feasible, helps balance both known and unknown confounding factors between groups. A control group provides a comparison, and blinding may reduce expectancy effects.

A randomized experiment can support stronger causal conclusions than a simple observational comparison, but it may be expensive, infeasible, or ethically inappropriate. Experiments must also define the unit of assignment, treatment, outcome, duration, and analysis population.

4.3 Administrative and secondary data

Primary data are collected for the current research or business question. Secondary data were collected previously, often for another purpose. Administrative records, open-government datasets, and public research repositories are secondary sources.

Secondary data can reduce cost and collection time, but analysts must verify whether the original definitions, population, time period, and recording procedures match the current purpose. A variable called “income,” for example, may mean taxable income, household income, salary, or a self-reported category.

5. Sampling and Representativeness

A population is the complete group of interest. A sample is the subset actually observed. A sampling frame is the operational list or mechanism from which the sample is selected. The target population and sampling frame may differ, producing coverage error.

5.1 Probability sampling

In probability sampling, each eligible population unit has a known, non-zero probability of selection. Common designs include:

• Simple random sampling: units are selected randomly from the frame.
• Systematic sampling: every kth unit is selected after a random start.
• Stratified sampling: the population is divided into strata and samples are drawn within them.
• Cluster sampling: naturally occurring groups are sampled, and units within selected groups are studied.

Stratification can ensure that important subgroups are represented. Cluster sampling can reduce fieldwork costs, but observations within a cluster may be similar, reducing the information gained from each additional observation.

5.2 Non-probability sampling

Convenience, purposive, quota, and voluntary-response sampling do not generally provide known selection probabilities. They may be practical for exploratory work, specialist populations, or early user research, but estimates may not generalize to the wider population.

A large sample is not automatically representative. If an online survey receives responses mainly from highly engaged users, increasing its size may make the biased estimate more precise rather than more accurate.

5.3 Sampling error and non-response

Sampling error is the difference between a sample-based estimate and the corresponding population value caused by observing a sample rather than the entire population. Non-response occurs when selected units do not provide usable data. Non-response can be especially serious when responders differ systematically from non-responders.

Analysts should report the target population, frame, selection method, sample size, response rate where applicable, exclusions, and limitations. Weighting may improve representation when appropriate auxiliary information exists, but weighting does not repair every source of bias.

6. Data Collection and Management Workflow

A professional workflow is planned rather than improvised:

1. Define the purpose. State the decision, research question, population, unit of analysis, and required time period.
2. Identify candidate sources. Compare direct collection, internal records, external datasets, APIs, sensors, and surveys.
3. Assess feasibility and legality. Check access, cost, licensing, consent, privacy, security, and retention requirements.
4. Specify the data. Create a data dictionary, schema, coding rules, units, identifiers, and missing-value policy.
5. Collect and record provenance. Preserve source details, timestamps, versions, parameters, and responsible personnel.
6. Ingest safely. Store raw data separately from cleaned and derived data. Do not overwrite the raw source.
7. Validate. Check types, ranges, uniqueness, referential integrity, completeness, duplicates, and unexpected changes.
8. Transform reproducibly. Use documented scripts or workflows rather than undocumented manual edits.
9. Secure and control access. Apply authentication, authorization, encryption, backups, and audit logs.
10. Publish or use with documentation. State definitions, limitations, quality results, and appropriate uses.

A useful layered arrangement is:

• Raw layer: original received data, preserved exactly where possible.
• Validated layer: data that have passed structural and basic quality checks.
• Curated layer: standardized and integrated data for analysis.
• Feature or reporting layer: variables and summaries created for a particular purpose.

This separation improves traceability and allows transformations to be repeated.

7. Metadata and Data Lineage

Metadata are data about data. Structural metadata describe fields, types, keys, and relationships. Descriptive metadata explain subject, source, and coverage. Administrative metadata record ownership, access rights, licensing, retention, and security classification. Quality metadata record validation results and known limitations.

A data dictionary might define temperature_c as “ambient temperature in degrees Celsius, measured by device D17 every 60 seconds.” This is more useful than a column name alone.

Data lineage describes where data originated and how they changed. A lineage chain may be represented as:

Each arrow should be supported by records such as code version, execution time, input files, parameters, and output location. Lineage supports reproducibility, debugging, audit, impact analysis, and responsible communication. If a source field is later found to be unreliable, lineage helps identify every report and model affected by it.

8. Data Quality Dimensions

Quality is fitness for a stated purpose, not an absolute property. A dataset can be adequate for a rough exploratory chart but unsuitable for safety-critical prediction.

A simple completeness rate for a field can be calculated as:

The formula assumes that missingness has been defined consistently and that all records in the denominator are eligible for the field. For example, if 940 of 1,000 required email values are present, completeness is 94%.

Quality checks should be proportionate to risk. A missing optional comment is different from a missing dosage in a clinical record. Analysts should distinguish an error, a legitimate missing value, and a value that was not applicable.

9. Data Cleaning Without Destroying Evidence

Cleaning may involve standardizing dates, correcting data types, resolving duplicate records, handling missing values, and detecting outliers. Every change should have a stated rule and, where possible, preserve the original value.

Missing data can be:

• Missing completely at random: missingness is unrelated to observed and unobserved values.
• Missing at random: missingness is related to observed information after relevant variables are considered.
• Missing not at random: missingness depends on the missing value or unobserved factors.

These categories are assumptions about a data-generating process, not labels that can always be proven from the dataset alone. Deleting incomplete records may reduce sample size and introduce bias. Imputation, explicit missing categories, or model-based methods may be appropriate depending on context.

Outlier detection should combine statistical evidence and domain knowledge. An unusually high payment may be fraud, a legitimate corporate purchase, or a data-entry error. Automatic deletion is unsafe.

10. Ethics, Privacy, Security, and Governance

Data collection affects people, organizations, and communities. Ethical practice requires a legitimate purpose, proportional collection, transparency, respect for consent where required, and protection against foreseeable harm.

Important principles include:

• Data minimization: collect only what is necessary.
• Purpose limitation: do not silently repurpose data beyond reasonable expectations.
• Access control: allow access according to role and need.
• Pseudonymization: replace direct identifiers while recognizing that re-identification may remain possible.
• Retention limitation: delete or archive data when no longer justified.
• Fairness: examine whether collection excludes or disadvantages groups.
• Accountability: assign owners and maintain audit evidence.

Personal data may include direct identifiers, quasi-identifiers, location traces, device identifiers, and inferred attributes. Removing names alone does not guarantee anonymity. Legal obligations vary by jurisdiction and sector; organizations should consult applicable privacy and records-management requirements.

Security measures include encryption in transit and at rest, strong authentication, backups, patching, secrets management, monitoring, and incident-response procedures. Data governance establishes policies, roles, standards, and decision rights. Typical roles include data owner, data steward, custodian, analyst, and privacy or security officer.

11. Applied Case: Predicting Equipment Failure

Background

This illustrative case concerns a manufacturing company that wants to predict failures in industrial pumps and schedule maintenance before production stops.

Problem

The company has temperature and vibration sensors, maintenance work orders, equipment specifications, and operator reports. Early analysis produced a model with high apparent accuracy, but managers discovered that many failures were missed in practice.

Analysis

The data team examined the sources. Sensor readings were stored every minute, but some devices used local time and others used UTC. Work orders recorded repair completion time rather than failure time. Operator reports were free text and frequently used different names for the same pump. Several pumps had been replaced, but their sensor identifiers were reused. The model had also been trained using records created after maintenance, allowing information that would not be available at prediction time to leak into the features.

Approach and Solution

The team created a data dictionary and a device-identity table. Timestamps were converted to a common time standard, while original timestamps were retained. A lineage record connected each feature to its sensor, work order, or report. Maintenance events were redefined using the best available failure-time evidence, and uncertain labels were flagged rather than treated as exact.

Raw sensor data were preserved in a raw layer. Validation checks identified impossible values, missing intervals, duplicate messages, and sensor drift. Text reports were standardized and converted into documented categorical indicators. The model-training dataset was built using only information available before each prediction time.

The team reported precision, recall, false alarms, and missed failures rather than accuracy alone. Maintenance engineers reviewed alerts before operational deployment.

Outcome

The revised model had lower headline accuracy but detected more actionable failures and produced fewer misleading alerts. Engineers trusted it because its data definitions, limitations, and alert history were documented.

Learning Points

1. Data quality includes timing, identity, and meaning, not only valid numbers.
2. Reused identifiers can corrupt joins and create false histories.
3. Leakage can make a model appear successful while failing in real use.
4. Domain experts are essential when labels and anomalies are ambiguous.
5. Lineage and preserved raw data support audit and improvement.

12. Contemporary Developments and Professional Practice

Modern data platforms increasingly support streaming ingestion, cloud storage, data catalogs, automated quality monitoring, and data contracts. A data contract specifies expected fields, types, freshness, ownership, and acceptable changes between producers and consumers.

Machine learning introduces additional concerns. Training data should be versioned, labelled consistently, and evaluated for representation and historical bias. Synthetic data may help development or privacy protection, but synthetic records can reproduce or amplify errors and should not automatically be treated as equivalent to real observations.

Generative AI and automated extraction can convert documents, audio, and images into structured fields. Such systems require validation because extraction errors, fabricated content, and inconsistent confidence scores can enter downstream datasets. Human review remains important for high-impact decisions.

Best-Practice Checklist

Before using a dataset, ask:

• What exact population and period does it cover?
• What is one row or observation?
• Who collected it, and for what original purpose?
• Are definitions, units, time zones, and codes documented?
• How were missing values, duplicates, and corrections handled?
• Can every derived field be traced to its sources?
• What selection, measurement, and non-response biases may exist?
• Is the collection lawful, ethical, secure, and proportionate?
• Are quality limitations communicated to users and decision-makers?
• Can the workflow be repeated from preserved inputs and code?

Chapter Summary

Reliable analysis depends on responsible data sourcing and management. Structured data use fixed schemas, semi-structured data use flexible keys or tags, and unstructured data require analytical structure to be extracted. Files, databases, APIs, surveys, sensors, and digital platforms differ in cost, accessibility, bias, timeliness, and governance requirements.

Observational data record naturally occurring conditions, whereas experimental data involve assigned interventions and can provide stronger evidence for causal claims under suitable designs. Sampling determines how well observations represent a target population; large convenience samples may still be biased. Metadata explain data meaning and management, while lineage records origin and transformation history.

Data quality is multidimensional. Accuracy, completeness, consistency, validity, uniqueness, timeliness, integrity, and relevance should be evaluated in relation to purpose and risk. Professional practice preserves raw data, documents transformations, controls access, protects privacy, and communicates limitations. These foundations are essential for trustworthy visualization, statistical analysis, and machine-learning systems.

Key Terms

API: A defined interface through which software requests data or services.

Accuracy: Agreement between a recorded value and the real-world value.

Administrative data: Records created during organizational operations or service delivery.

Completeness: The extent to which required data are present.

Confounding: Distortion of an apparent relationship by another variable.

Data dictionary: Documentation defining fields, types, units, and permitted values.

Data lineage: A record of data origin, movement, and transformation.

Data quality: Fitness of data for a stated purpose.

Data warehouse: A system designed for integrated historical analysis.

Experimental data: Data produced when an intervention is assigned according to a study design.

Metadata: Information describing data content, structure, context, or management.

Observational data: Data recorded without assigning an intervention.

Population: The complete group about which conclusions are intended.

Primary data: Data collected for the current investigation or purpose.

Sampling frame: The operational list or mechanism used to select a sample.

Semi-structured data: Data with flexible organization using keys, tags, or nested objects.

Structured data: Data organized according to a defined schema, commonly in tables.

Unstructured data: Data without a fixed analytical format, such as images or free text.

Validity: Conformity to defined formats, rules, or permitted values.

Pseudonymization: Replacing identifying fields with artificial identifiers while retaining a controlled possibility of linkage.

Review Questions

Short Answer Questions

1. What distinguishes structured, semi-structured, and unstructured data?
2. Define metadata and give two examples.
3. What is the difference between observational and experimental data?
4. Name four dimensions of data quality.
5. Why should raw data be preserved separately from cleaned data?

Long Answer Questions

1. Explain the advantages and limitations of files, databases, APIs, surveys, sensors, and digital platforms as data sources.
2. Describe probability sampling methods and explain why a large convenience sample may still be unrepresentative.
3. Explain how data lineage supports reproducibility, governance, and error investigation.
4. Discuss the major ethical, privacy, security, and governance issues in data collection.
5. Design a complete workflow for collecting and managing data for an AI project.

Analytical / Application Questions

1. A survey of 2,000 selected people receives 800 responses, mainly from frequent users of a service. Identify likely sources of bias and propose improvements.
2. A sensor dataset contains duplicate readings, mixed time zones, missing intervals, and reused device identifiers. Develop a validation and lineage plan before analysis.
3. A predictive model performs well in testing but poorly after deployment. Using the chapter concepts, diagnose possible problems in sampling, data leakage, quality, metadata, and lineage.

Multiple Choice Questions with Answers

Question 1

Which type of data commonly uses keys, tags, or nested objects without requiring one rigid table schema?

A. Structured data

B. Semi-structured data

C. Experimental data

D. Aggregated data

Correct Answer: B. Semi-structured data

Question 2

Which source is most directly associated with measurement calibration, sampling interval, and firmware metadata?

A. Sensor data

B. Survey data

C. Database data

D. Textbook data

Correct Answer: A. Sensor data

Question 3

What is the defining feature of an experimental study?

A. It always uses a very large sample

B. It uses only secondary data

C. It assigns an intervention according to a protocol

D. It excludes a control group

Correct Answer: C. It assigns an intervention according to a protocol

Question 4

Which sampling method divides a population into subgroups and samples within each subgroup?

A. Cluster sampling

B. Convenience sampling

C. Voluntary-response sampling

D. Stratified sampling

Correct Answer: D. Stratified sampling

Question 5

If 940 of 1,000 required values are present, what is the completeness rate?

A. 9.4%

B. 40%

C. 94%

D. 106%

Correct Answer: C. 94%

Question 6

Which document primarily defines field names, data types, units, and permitted values?

A. Data dictionary

B. Incident report

C. Marketing plan

D. Access token

Correct Answer: A. Data dictionary

Question 7

What does data lineage record?

A. Only the number of rows in a table

B. The origin and transformations of data

C. Only the visual design of a dashboard

D. The salaries of data analysts

Correct Answer: B. The origin and transformations of data

Question 8

Why can a large convenience sample still produce biased results?

A. Large samples cannot contain missing values

B. Selection may systematically exclude parts of the target population

C. Probability samples are always smaller

D. Convenience samples automatically become experiments

Correct Answer: B. Selection may systematically exclude parts of the target population

Question 9

Which action best protects reproducibility during data cleaning?

A. Overwrite the original file manually

B. Remove all unusual values without review

C. Use documented, repeatable transformation scripts

D. Change definitions for each report

Correct Answer: C. Use documented, repeatable transformation scripts

Question 10

Why is data leakage especially dangerous in machine-learning projects?

A. It makes all datasets smaller

B. It uses information unavailable at prediction time and inflates apparent performance

C. It prevents data from being stored

D. It guarantees representative sampling

Correct Answer: B. It uses information unavailable at prediction time and inflates apparent performance