Data Acquisition, Storage and IntegrationIntroduction and ContextData science begins before modelling. A model is only as credible as the data supplied to it, and the quality of that data depends on how it is acquired, stored, described, integrated and governed. In professional environments, data rarely arrives as one clean table. It may be distributed across transactional databases, spreadsheets, JSON APIs, web pages, sensor streams, cloud object stores and specialist systems. These sources differ in structure, update frequency, reliability, access controls, semantics and legal status. Data acquisition is the controlled process of obtaining data from one or more sources. Data storage concerns the physical and logical arrangements used to retain data for operational use, analysis, audit and recovery. Data integration combines data from heterogeneous sources while preserving meaning, lineage and quality. In R, these activities involve packages such as `readr`, `readxl`, `DBI`, `RPostgres`, `httr2`, `jsonlite`, `rvest`, `arrow`, `dplyr` and `dbplyr`, although the principles are independent of any particular software. At postgraduate level, acquisition should be treated as an engineering and governance problem rather than merely a programming task. The analyst must ask: What is the source of truth? What does each field mean? Which records are valid at a given time? Can the transformation be reproduced? Is the collection lawful and ethical? Can another researcher reconstruct the analytical dataset six months later? Learning ObjectivesBy the end of this chapter, the reader should be able to: 1. Distinguish major data sources, storage architectures and file formats. 2. Acquire structured and semi-structured data using R. 3. Query relational databases using SQL and R database interfaces. 4. Explain schemas, keys, constraints, metadata and data provenance. 5. Integrate data while managing grain, types, missingness, duplicates and temporal validity. 6. Select storage formats according to analytical, operational and governance requirements. 7. Design a reproducible and secure acquisition pipeline. 8. Evaluate streaming, API and web-based acquisition methods and their limitations. 1. The Data Acquisition Landscape1.1 Sources and their characteristicsCommon sources include: - Files: CSV, TSV, Excel workbooks, XML, JSON, Parquet and Avro files.
- Relational databases: systems such as PostgreSQL, MySQL, SQL Server and Oracle, which store related tables and support SQL transactions.
- NoSQL databases: document, key-value, wide-column and graph stores, often used where schema flexibility or horizontal scale is important.
- APIs: programmatic interfaces that expose resources through HTTP, commonly returning JSON or XML.
- Web sources: HTML pages, downloadable files and public data portals. Web scraping must respect terms of service, robots policies where relevant, access controls and applicable law.
- Streaming systems: event brokers and message platforms that deliver records continuously, often with ordering, partitioning and replay characteristics.
- Devices and applications: sensors, logs, point-of-sale systems, mobile applications and enterprise systems.
A source should be assessed along several dimensions: structure, volume, velocity, variety, veracity, access reliability, latency, cost, sensitivity and semantic authority. A small CSV may be difficult to use if its encoding, units and definitions are undocumented. Conversely, a large Parquet data set may be efficient because it is columnar, typed and accompanied by a catalogue. 1.2 Batch and streaming acquisitionBatch acquisition retrieves a finite collection at scheduled intervals. It is appropriate for monthly financial extracts, nightly operational snapshots and historical archives. Streaming acquisition handles an ongoing sequence of events. A stream may be processed in real time, near real time or accumulated into periodic batches. A crucial distinction is between an event and a state. An event records something that happened, such as a payment authorised at 10:03. A state records a current condition, such as the account balance. Replaying events can reconstruct state, but only if event ordering, correction events and business rules are understood. Analysts should not treat an event log as a current-state table without performing the necessary aggregation. 2. Relational Principles and SQL2.1 Relations, keys and grainA relational table represents a relation: rows are tuples and columns are attributes. In practice, relational systems also enforce constraints, indexes, transactions and permissions. The grain of a table states what one row represents. For example, one row may represent one order, one order line, one customer per day or one sensor reading. A primary key uniquely identifies a row. A foreign key references a key in another table and expresses a relationship. A candidate key is any minimal set of attributes capable of uniquely identifying rows; one candidate key is selected as the primary key. A surrogate key is an artificial identifier, whereas a natural key has business meaning. Consider an order system: ```sql CREATE TABLE customer ( customer_id INTEGER PRIMARY KEY, email TEXT NOT NULL UNIQUE, created_at TIMESTAMP NOT NULL ); CREATE TABLE orders ( order_id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL REFERENCES customer(customer_id), order_date DATE NOT NULL, status TEXT NOT NULL ); CREATE TABLE order_line ( order_id INTEGER REFERENCES orders(order_id), product_id INTEGER NOT NULL, quantity INTEGER NOT NULL CHECK (quantity > 0), unit_price NUMERIC(12,2) NOT NULL CHECK (unit_price >= 0), PRIMARY KEY (order_id, product_id) ); ``` The primary key of `order_line` indicates that, in this design, a product occurs at most once per order. That assumption would be incorrect if the business permits separate lines for different discounts or fulfilment locations. Database design must therefore reflect business semantics, not only technical convenience. 2.2 Normalisation and analytical modelsNormalisation reduces update anomalies by separating facts into logically coherent tables. In a simplified interpretation, first normal form requires atomic values, second normal form removes dependencies on part of a composite key, and third normal form removes non-key attributes that depend on other non-key attributes. Normalisation is valuable for transactional integrity, but analytical workloads often use dimensional models or denormalised structures to improve query simplicity and performance. A star schema contains a central fact table surrounded by dimension tables. A sales fact may have one row per order line and foreign keys to date, product, customer and store dimensions. Measures such as quantity and revenue are additive or semi-additive according to their business meaning. A data warehouse may also use slowly changing dimensions to preserve historical descriptions, rather than overwriting them. 2.3 SQL operations and semantic riskA relational query commonly uses `SELECT`, `FROM`, `WHERE`, `GROUP BY`, `HAVING`, `JOIN` and `ORDER BY`. The logical order of evaluation is conceptually different from the written order: rows are selected from sources, filtered, grouped, aggregated, projected and ordered. Understanding this order prevents errors such as filtering an aggregate in `WHERE` instead of `HAVING`. ```sql SELECT c.customer_id, COUNT(o.order_id) AS order_count, SUM(ol.quantity * ol.unit_price) AS sales_value FROM customer AS c JOIN orders AS o ON o.customer_id = c.customer_id JOIN order_line AS ol ON ol.order_id = o.order_id WHERE o.status = 'completed' GROUP BY c.customer_id HAVING SUM(ol.quantity * ol.unit_price) > 1000; ``` The query has one row per customer after grouping. A frequent integration error is joining tables at incompatible grains. Joining an order-level table to an order-line table and then summing an order-level amount duplicates that amount once for every line. Analysts must identify grain before joining and aggregate at the correct level. `INNER JOIN` retains matching rows only. `LEFT JOIN` retains every row from the left table and adds matching values from the right table, using missing values where no match exists. A left join is often appropriate when a master list must be retained, but it can silently produce duplicate rows if the right-hand key is not unique. 3. Schemas, Metadata and Data Provenance3.1 Schema as a contractA schema describes the structure and permitted interpretation of data. It may specify field names, data types, nullability, constraints, units, permitted codes and relationships. A schema can be explicit, as in a database definition, or implicit, as in a loosely documented CSV file. Schema-on-write validates and transforms data before storage. It supports strong governance and predictable queries but may reject novel data. Schema-on-read applies interpretation when data is consumed. It offers flexibility, but inconsistent interpretations may proliferate. Modern data platforms often combine both approaches: raw data is retained with limited alteration, while curated layers enforce contracts. Schema evolution occurs when fields are added, removed, renamed or changed in meaning. Backward-compatible changes, such as adding an optional field, are generally safer than changing the type or meaning of an existing field. Versioning and migration procedures are essential when producers and consumers are updated independently. 3.2 MetadataMetadata is data about data. Structural metadata records types, columns, keys and formats. Descriptive metadata explains business meaning, such as the distinction between order date and shipment date. Operational metadata records refresh time, row counts, processing status and failures. Security metadata records classification, ownership, access policy and retention requirements. A useful data catalogue should identify the owner, steward, source system, definition, grain, refresh schedule, quality rules, sensitivity classification and permitted uses of each important data asset. Metadata should be maintained as part of the pipeline rather than written once in an abandoned document. 3.3 Provenance and lineageData provenance is the record of where data came from, what happened to it and when those operations occurred. Lineage may be represented as a directed acyclic graph: source files and tables are upstream nodes, transformations are edges or processing nodes, and analytical outputs are downstream nodes. A robust provenance record includes source URI or table, extraction timestamp, source version or request parameters, software version, transformation code, validation results and output identifier. Checksums can help establish whether a downloaded file has changed. Provenance supports reproducibility, debugging, regulatory inspection and scientific credibility. It also exposes risks: an apparently authoritative result may depend on a web page that changed without notice or on a manually edited spreadsheet. 4. Acquiring Data with R4.1 Files and formatsDelimited files are accessible but often ambiguous. Problems include inconsistent delimiters, quotation, encoding, decimal conventions, dates, duplicate headers and missing-value codes. Prefer explicit parsing and validation rather than relying on automatic guessing. ```r library(readr) sales <- read_csv( "data/sales.csv", col_types = cols( order_id = col_integer(), order_date = col_date(format = "%Y-%m-%d"), quantity = col_double(), unit_price = col_double() ), na = c("", "NA", "NULL") ) ``` Excel files may contain multiple sheets, formulas, formatting and hidden assumptions. They are useful for exchange but are not ideal as a system of record. For large analytical data, Parquet is usually preferable to CSV because it preserves types, supports compression and permits column projection. Arrow enables R to query Parquet data efficiently without loading every column into memory. ```r library(arrow) sales_ds <- open_dataset("warehouse/sales", format = "parquet") result <- sales_ds |> dplyr::filter(order_date >= as.Date("2025-01-01")) |> dplyr::select(order_id, customer_id, quantity, unit_price) |> dplyr::collect() ``` The choice of format should consider interoperability, compression, random access, schema support, archival stability and the capabilities of downstream users. 4.2 Relational databases through DBIThe Database Interface (DBI) provides a common R framework for connecting to database back ends. Queries should be parameterised, credentials should not be placed in source code, and large results should be filtered or aggregated in the database before collection. ```r library(DBI) con <- dbConnect( RPostgres::Postgres(), dbname = Sys.getenv("PGDATABASE"), host = Sys.getenv("PGHOST"), user = Sys.getenv("PGUSER"), password = Sys.getenv("PGPASSWORD") ) cutoff <- as.Date("2025-01-01") q <- sqlInterpolate( con, "SELECT customer_id, COUNT(*) AS n FROM orders WHERE order_date >= ?cutoff GROUP BY customer_id", cutoff = cutoff ) customer_counts <- dbGetQuery(con, q) dbDisconnect(con) ``` `dbplyr` allows dplyr code to be translated into SQL, but translation should be inspected because not every R operation has an identical database implementation. Connection pooling, transaction management, indexes and query plans become important in production systems. 4.3 APIs and JSONAn API request should be treated as an external dependency. The pipeline should record endpoint, parameters, authentication method, response status, retrieval time and relevant headers. It should implement timeouts, rate-limit handling, retries with backoff and validation of the returned schema. Retries should not blindly repeat non-idempotent operations. ```r library(httr2) library(jsonlite) req <- request("https://api.example.org/v1/records") |> req_url_query(limit = 100, page = 1) |> req_timeout(30) |> req_headers(`User-Agent` = "research-client/1.0") resp <- req_perform(req) resp_check_status(resp) records <- resp_body_json(resp, simplifyVector = TRUE) ``` JSON is hierarchical. A response may contain nested arrays, optional fields and mixed types. Flattening it into a rectangular table requires a declared interpretation. For example, one record with three transactions should become either one record row with a nested transaction list or three transaction rows with a repeated record identifier. The correct choice depends on the intended grain. Pagination may be page-based, offset-based or cursor-based. Cursor pagination is generally safer when records are changing during extraction. Incremental acquisition should use stable watermarks such as an update timestamp or source sequence number, while allowing for late-arriving records. 4.4 Web sourcesWeb scraping is technically possible but methodologically fragile. HTML layout changes, client-side rendering, anti-bot mechanisms and ambiguous definitions can compromise a collection. Prefer an official API or downloadable dataset where available. Respect authorisation, terms of use, copyright, privacy obligations and reasonable request rates. Preserve the retrieved page or a permitted archival representation when reproducibility requires it. 5. Storage Architectures and Format SelectionOperational databases optimise transaction consistency and concurrent updates. Data warehouses optimise governed analytical queries, often using columnar storage and dimensional or relational models. Data lakes store diverse objects, frequently in cloud object storage. A lakehouse combines low-cost object storage with table formats, schema management, transactions and warehouse-like query capabilities. A useful architecture separates: 1. Landing or raw layer: immutable or append-oriented copies of source data. 2. Standardised layer: corrected types, standard naming, parsed timestamps and validated structures. 3. Curated layer: integrated, documented datasets designed for analysis or reporting. 4. Serving layer: marts, extracts, models or APIs tailored to consumers. Raw does not mean uncontrolled. Access must be restricted, sensitive fields protected and retention defined. A raw copy should not be used to justify retaining data indefinitely. Storage format selection involves trade-offs. CSV is human-readable and widely supported but weakly typed and inefficient for large analytical workloads. JSON represents hierarchy well but can be verbose and difficult to query consistently. Parquet is typed, compressed and column-oriented, making it suitable for analytical scans. Relational tables provide constraints, indexes and transactions. A message log is appropriate for ordered event distribution but is not automatically a durable analytical archive. 6. Data Integration as Semantic EngineeringIntegration is more than concatenating files. It requires agreement about entities, identifiers, time, units, categories and business definitions. 6.1 Schema matching and entity resolutionSchema matching maps fields with different names or structures, such as `cust_id` and `customer_number`. Entity resolution determines whether records refer to the same real-world entity. Exact matching on a stable identifier is preferable. If identifiers are absent, probabilistic or rule-based matching may use names, addresses and dates, but false matches and missed matches must be measured and reviewed. A crosswalk table should preserve source identifier, canonical identifier, matching method, confidence or review status and effective dates. Never overwrite the original identifier: it is needed for audit and correction. 6.2 Type, unit and time alignmentA date without a time zone may not be comparable to a UTC timestamp. Monetary values require currency and whether tax or discounts are included. A field called `temperature` is unusable without a unit and measurement context. Standardisation should record conversion rules and assumptions. Temporal integration requires distinguishing event time, ingestion time and processing time. Late data may arrive after a reporting period has closed. Slowly changing dimensions and valid-from/valid-to intervals can preserve historical context. Analysts must avoid using future information when constructing a past prediction dataset; this is a form of temporal leakage. 6.3 Quality controlsUseful controls include uniqueness, referential integrity, range checks, accepted-value checks, completeness, freshness, distribution monitoring and reconciliation to authoritative totals. A quality rule should specify its purpose, threshold, action and owner. Not every anomaly is an error: a sudden sales increase may be a genuine event, so automated checks should distinguish blocking failures from warnings requiring investigation. 7. Streaming Data and Incremental PipelinesA streaming system commonly contains producers, a broker, consumers and a durable sink. Events may be partitioned by key, and ordering is normally guaranteed only within a partition or defined scope. Delivery semantics are often described as at-most-once, at-least-once or exactly-once. At-least-once delivery can produce duplicates, so consumers should be idempotent: processing the same event twice should not change the final result after the first successful application. A practical event should have an event identifier, event type, producer, event time, schema version, entity key and payload. Watermarks and windows support calculations over time, but out-of-order and late events require explicit policy. A streaming pipeline should provide replay, dead-letter handling, monitoring, back-pressure control and a route for schema evolution. R is particularly effective for consuming micro-batches and analysing stored events; high-throughput production ingestion is often implemented with specialised brokers and services, with R used downstream for quality analysis, feature creation and modelling. 8. Security, Ethics and GovernanceAcquisition must follow least privilege: a process receives only the access required for its task. Credentials belong in a secret manager or protected environment, not in scripts or notebooks. Connections should use encryption, and logs should avoid exposing tokens or personal data. Personally identifiable information should be minimised, masked, pseudonymised or encrypted where appropriate. Governance includes lawful basis, purpose limitation, retention, access rights, deletion procedures, cross-border transfer controls and records of processing. Regulations vary by jurisdiction, but privacy principles such as transparency, proportionality and accountability are broadly relevant. Public availability does not automatically make a dataset ethically unrestricted. Web data may contain vulnerable individuals, inferred attributes or information collected outside the expectations of the people represented. Data provenance also has ethical significance. An integrated dataset can create new risks through linkage, even when each source appears harmless. Access decisions should consider re-identification risk, group harms, discriminatory use and the possibility that errors will affect individuals. 9. Illustrative Case: Integrating Retail Transactions and Customer SupportBackgroundA fictional national retailer wants to understand whether delivery problems predict customer churn. Transaction data are stored in PostgreSQL, support interactions are delivered through a JSON API, and delivery scans arrive as daily Parquet files. ProblemThe initial analysis produced implausibly high customer revenue and inconsistent churn rates. Investigation found that transactions were at order-line grain, support data used an email address rather than a customer identifier, and delivery timestamps used local time while the API used UTC. In addition, the API returned pages of records and occasionally repeated the final record of one page on the next. Analysis and approachThe team first wrote a data contract. It defined a customer, an order, an order line, a support interaction and a delivery event. It documented grain, identifiers, currency, time zones and refresh schedules. PostgreSQL metadata and row counts were captured at extraction. API requests recorded cursor values, status codes and retrieval times. Files were stored in a restricted raw area with checksums. A crosswalk mapped verified support email addresses to customer identifiers. Ambiguous matches were placed in a review table rather than automatically assigned. Delivery event timestamps were converted to UTC while preserving the original timestamp and source time zone. Duplicate API events were removed using the source event identifier, and the operation was made repeatable. For revenue, the team aggregated order lines to order grain before joining order-level delivery outcomes. Customer-level features were then calculated using only information available before the churn observation window. Validation reconciled completed-order totals against the finance system and compared event counts with the delivery provider’s daily summaries. OutcomeThe revised pipeline reduced apparent revenue by 8.4 percent, because duplicated order amounts had been removed, and excluded 3.1 percent of support interactions whose identity could not be established reliably. The resulting churn model was less optimistic but more defensible. The organisation retained an audit trail showing source versions, matching decisions, transformations and validation results. Learning pointsThe central lesson is that integration quality is semantic. Correct syntax cannot repair an incorrect grain, an unverified identity match or an undefined time zone. The second lesson is that uncertainty should be represented explicitly. Excluding or flagging ambiguous records is often more responsible than silently assigning them. Finally, reproducibility requires both code and evidence about the inputs to which that code was applied. 10. Best Practices and Contemporary DevelopmentsA mature acquisition workflow is automated but observable. It should use version-controlled code, environment management, configuration separated from logic, automated tests, documented schemas, incremental extraction, idempotent writes, monitoring and alerting. Pipelines should be restartable and should distinguish transient failures from permanent data-quality failures. Data contracts are increasingly used between producers and consumers. Data observability tools monitor freshness, volume, distributions, schema changes and lineage. Open table formats and lakehouse systems provide transaction logs and schema evolution on object storage. Knowledge graphs and semantic layers can improve discovery by representing relationships among datasets, concepts and business definitions. Artificial intelligence can assist with schema matching, anomaly detection, metadata generation and natural-language query, but it should not be treated as an authority. AI-generated mappings require validation, particularly where personal data, regulated decisions or financial measures are involved. Large language models may also reproduce sensitive source content or hallucinate field meanings. Human ownership, testable rules and provenance remain necessary. Chapter SummaryData acquisition, storage and integration form the foundation of trustworthy data science. Sources differ in structure, authority, latency, scale and governance requirements. Relational principles—grain, keys, constraints, joins and normalisation—provide a rigorous basis for integrating structured data, while dimensional models support analytical use. Schemas act as contracts, metadata explains structure and meaning, and provenance records origin and transformation. R supports acquisition from files, databases, APIs, web sources and analytical storage formats, but sound engineering requires explicit parsing, parameterised queries, validation, secure credentials and reproducible records. Storage choices should reflect workload, typing, scale, interoperability and governance. Integration requires semantic alignment of entities, identifiers, units and time. Streaming introduces ordering, duplication, late data and delivery semantics. Ethical and regulatory duties include privacy, purpose limitation, access control, retention and consideration of linkage risks. The most reliable pipelines are observable, testable, versioned, idempotent and designed around clearly documented data contracts. Key TermsData acquisition: The controlled process of obtaining data from one or more sources. Data integration: Combining data sources while preserving semantic consistency and traceability. Grain: The business meaning of one row in a table. Primary key: An attribute or attribute set that uniquely identifies a row. Foreign key: An attribute that references a key in another table. Schema: A description of data structure, types, constraints and relationships. Metadata: Information describing data, its meaning, operation and governance. Data provenance: Evidence about data origin, transformations and processing history. Schema-on-write: Validating structure before data is stored. Schema-on-read: Applying interpretation when data is consumed. Normalisation: Structuring relational data to reduce redundancy and update anomalies. Star schema: A fact table linked to surrounding dimension tables. API: A programmatic interface through which software requests data or services. Idempotence: The property that repeated processing has the same final effect as one successful processing. Event time: The time at which an event occurred according to its source. Data warehouse: A governed system optimised for analytical queries. Data lake: Storage for diverse data objects, often in object storage. Parquet: A typed, compressed columnar file format for analytical data. Entity resolution: Determining which records refer to the same real-world entity. Data contract: An explicit agreement concerning a data product’s structure, meaning and quality. Review QuestionsShort Answer Questions1. Why is identifying table grain important before performing a join? 2. Distinguish schema-on-write from schema-on-read. 3. What information should a provenance record contain? 4. State two advantages of Parquet over CSV for analytical workloads. 5. Why should API credentials not be embedded directly in R scripts? Long Answer Questions1. Explain how primary keys, foreign keys and constraints support reliable relational integration. Include an example of a possible constraint failure. 2. Compare operational databases, data warehouses, data lakes and lakehouses in terms of purpose and storage behaviour. 3. Describe a reproducible R pipeline for acquiring and validating paginated JSON API data. 4. Discuss how event time, ingestion time, late data and duplicate delivery affect streaming analytics. 5. Explain the roles of metadata, data catalogues and lineage in organisational data governance. Analytical / Application Questions1. An order table contains one row per order and an order-line table contains five rows for a particular order. If an order-level shipping charge is joined to the order-line table and summed, what error can occur? Design a correction. 2. Design an integration strategy for customer records from two systems when 7 percent have missing stable identifiers and names are inconsistently formatted. Explain how you would represent uncertainty and evaluate matching quality. 3. A public API changes a field from numeric to text and begins returning duplicate records during pagination. Propose technical, testing and provenance controls that would prevent the change from silently corrupting a production dataset. Multiple Choice Questions with AnswersQuestion 1What does the grain of a table specify? A. The compression ratio of the table B. What one row represents C. The number of database indexes D. The physical storage location Correct Answer: B. What one row represents Question 2Which SQL clause is used to filter groups after aggregation? A. WHERE B. FROM C. HAVING D. ORDER BY Correct Answer: C. HAVING Question 3Which format is generally typed, compressed and column-oriented for analytical workloads? A. CSV B. Plain text C. Parquet D. HTML Correct Answer: C. Parquet Question 4What is the main purpose of a foreign key? A. To encrypt a table B. To reference a key in another table C. To sort rows alphabetically D. To compress duplicate values Correct Answer: B. To reference a key in another table Question 5Which practice best protects database credentials in an R pipeline? A. Store them in a public repository B. Put them in the script comments C. Use protected environment variables or a secret manager D. Include them in the output data Correct Answer: C. Use protected environment variables or a secret manager Question 6What problem can result from joining an order-level amount to multiple order-line rows and summing it? A. The amount may be duplicated B. The database becomes automatically normalised C. All foreign keys are removed D. Dates are converted to UTC Correct Answer: A. The amount may be duplicated Question 7What does data provenance primarily document? A. Only the colour scheme of a dashboard B. The origin and transformations of data C. The number of users of an application D. The network bandwidth available Correct Answer: B. The origin and transformations of data Question 8Which delivery model requires consumers to tolerate possible duplicate events? A. At-least-once delivery B. No-delivery processing C. Compile-time delivery D. Single-user delivery Correct Answer: A. At-least-once delivery Question 9Why is a cursor often useful for API pagination over changing data? A. It guarantees that the API has no schema B. It can provide a more stable continuation position C. It converts JSON to SQL automatically D. It removes the need for authentication Correct Answer: B. It can provide a more stable continuation position Question 10Which principle requires a process to receive only the access needed for its task? A. Least privilege B. Schema denormalisation C. Event sourcing D. Column projection Correct Answer: A. Least privilege
|