Business Data Sources and Data Management

Introduction

Business analytics depends on more than sophisticated dashboards or predictive algorithms. It depends first on the availability of appropriate, trustworthy, well-managed data. An analyst may use an excellent statistical model, but the resulting decision will still be unreliable if the underlying data are incomplete, duplicated, outdated, poorly defined or collected without regard for privacy.

Data management is the set of practices, technologies, roles and controls used to acquire, store, describe, protect, integrate and use data. In business analytics, data management connects operational activity with evidence-based decision-making. It helps an organisation answer questions such as:

• Where did the data come from?
• What does each field mean?
• Can different systems be combined reliably?
• Who is permitted to use the data?
• How current and accurate are the results?
• Can the organisation explain and reproduce an analytical conclusion?

This chapter introduces major business data sources and the principal structures used to manage them, including databases, data warehouses, data lakes, master data, metadata, data pipelines and foundational SQL.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish internal from external data and structured from unstructured data.
2. Explain the purposes of operational databases, data warehouses and data lakes.
3. Describe master data, reference data, metadata and data quality.
4. Explain the stages and controls of a modern data pipeline.
5. Read and construct basic SQL queries for analytical purposes.
6. Identify governance, ethical, privacy and security issues in data management.
7. Select an appropriate data-management approach for a business problem.

1. Understanding Business Data

Data are recorded representations of events, entities or observations. A sales transaction, a customer complaint, a product description and a web-page visit are all examples of business data. Information is data interpreted in a context that gives it meaning. A value of “250” is ambiguous until its field, unit, time period and business definition are known.

A useful distinction is between data, information, insight and decision. Data may show that monthly returns increased. Information may establish that returns increased from 4% to 7% in a particular product category. Insight may suggest that a packaging change contributed to the increase. A decision may be to redesign the packaging and monitor the effect.

Data used in analytics can be classified in several ways. These classifications are not mutually exclusive. For example, a customer relationship management system may contain internal, structured, operational data, while recorded customer-service calls may be internal, unstructured data.

Internal and External Data

Internal data are generated by an organisation's own activities and systems. Examples include:

• sales invoices and payment records;
• inventory movements;
• employee attendance and payroll records;
• customer accounts and service interactions;
• website and application usage logs;
• production, maintenance and quality records.

Internal data are often valuable because the organisation controls their collection and can connect them to business processes. However, they may reflect only the organisation's own customers and may contain inconsistent definitions across departments.

External data are obtained from sources outside the organisation. Examples include:

• government statistics and regulatory publications;
• market-research reports;
• public demographic or geographic datasets;
• supplier and partner data;
• social-media or online-review data;
• weather, economic and industry data.

External data can provide context and support benchmarking, demand forecasting or market segmentation. Important questions include whether the source is authoritative, whether the data may legally be used, how often they are updated and whether their definitions match internal measures.

First-Party, Second-Party and Third-Party Data

First-party data are collected directly by an organisation from its own interactions with customers, employees, suppliers or users. Second-party data are another organisation's first-party data shared through an agreement. Third-party data are obtained from an independent provider that aggregates or sells data from multiple sources.

This classification is especially relevant to marketing analytics. Data ownership, consent, contractual restrictions and permitted purposes must be considered before combining these sources.

Structured, Semi-Structured and Unstructured Data

Structured data fit a defined schema, usually consisting of fields and records. A table containing customer identification numbers, dates and transaction amounts is structured. Structured data are relatively easy to validate, query and aggregate.

Semi-structured data do not necessarily conform to one fixed table, but contain organisational labels or hierarchical features. JSON, XML and many application-programming-interface responses are common examples. A record may contain a customer identifier and address fields, while another record contains additional attributes.

Unstructured data do not have a predefined tabular structure. Examples include emails, contracts, photographs, audio recordings, videos, scanned documents and free-text reviews. Unstructured data can contain valuable business signals, but analysis generally requires techniques such as text processing, speech recognition, image analysis or natural-language processing.

The distinction concerns organisation, not usefulness. A customer review is unstructured in its original form, but analysts can derive structured variables such as sentiment, topic or complaint category. Such derived variables are interpretations and should not be confused with the original source evidence.

Data Quality Dimensions

Data quality is fitness for a particular purpose. Common dimensions include:

Quality is contextual. A stock balance updated once per day may be adequate for strategic planning but inadequate for real-time warehouse control. Data-quality controls should therefore be linked to the decision the data support.

2. Databases and Data Models

A database is an organised collection of data managed by software known as a database management system, or DBMS. A DBMS provides facilities for storing, retrieving, updating, securing and controlling access to data.

Relational Databases

A relational database stores data in tables. Each table consists of rows, called records or tuples, and columns, called attributes or fields. A primary key uniquely identifies each row. A foreign key is a field whose value refers to a key in another table, thereby representing a relationship.

For example, a simple order system might contain:

• Customers(CustomerID, CustomerName, Region)
• Orders(OrderID, CustomerID, OrderDate, OrderStatus)
• OrderLines(OrderID, ProductID, Quantity, UnitPrice)
• Products(ProductID, ProductName, Category)

CustomerID is the primary key of Customers and a foreign key in Orders. OrderID and ProductID together may form a composite primary key in OrderLines.

Relational design often uses normalisation, the organisation of data to reduce unnecessary duplication and update anomalies. In a normalised design, customer information is stored once in a customer table rather than repeatedly in every order row. Normalisation supports integrity, although analytical systems may deliberately use some denormalisation to improve query simplicity or performance.

A relational database commonly supports the ACID properties of transactions:

• Atomicity: a transaction is completed fully or not at all;
• Consistency: a transaction preserves defined rules;
• Isolation: concurrent transactions do not improperly interfere;
• Durability: committed changes persist despite a system failure.

These properties are particularly important for operational systems such as banking, order processing and payroll.

NoSQL and Other Database Approaches

NoSQL databases are designed for use cases in which a relational structure may not be the most suitable approach. Document databases store records such as JSON documents; key-value stores associate a key with a value; graph databases represent nodes and relationships; and column-family systems organise data by column groups. NoSQL does not mean “without data management” or “without any structure.” It usually means that the system uses a non-relational model or a more flexible schema.

The choice between relational and non-relational technologies depends on data structure, transaction requirements, scale, access patterns, latency and team capability. It is not correct to assume that one technology is universally superior.

3. Data Warehouses, Data Lakes and Lakehouses

Operational Databases

Operational databases, often called online transaction processing systems or OLTP systems, support routine business transactions. They prioritise fast inserts and updates, concurrency, validation and transaction integrity. Examples include point-of-sale, banking and booking systems.

Operational systems are not always suitable for complex analytics. Large analytical queries may compete with transactions for computing resources, and operational schemas are often designed for processing individual events rather than summarising long-term patterns.

Data Warehouses

A data warehouse is a centrally managed repository designed primarily for reporting and analysis. It normally integrates data from multiple operational systems, applies standard definitions and preserves historical information. Warehouse data are commonly organised using fact and dimension structures.

A fact table contains measurable events, such as sales amount, quantity or cost. Dimension tables describe the context of those events, such as product, customer, store and date. This arrangement is often called a star schema.

A warehouse improves consistency by providing governed measures. For example, the organisation can define “net sales” centrally rather than allowing every department to calculate it differently. A warehouse may use extract, transform and load (ETL), or extract, load and transform (ELT), depending on the platform and architecture.

Data Lakes

A data lake stores large quantities of data in relatively raw form. It may contain structured tables, JSON files, text, images, audio and logs. The schema is often applied when data are read, a principle known as schema-on-read. This enables exploratory work and supports data types that are difficult to fit into conventional warehouse tables.

A data lake without cataloguing, access controls, quality rules and lifecycle management can become a data swamp: a repository in which users cannot determine what data exist, whether they are trustworthy or how they may be used.

Lakehouses

A lakehouse attempts to combine the low-cost, flexible storage of a data lake with the management, reliability and analytical performance associated with a warehouse. The term describes an architectural approach rather than one universal product. In practice, organisations should evaluate governance, query performance, interoperability, cost and skills rather than select a design solely because of its label.

4. Master Data, Reference Data and Metadata

Master data describe important business entities shared across processes. Common domains include customers, products, suppliers, employees, locations and accounts. If one system identifies a product as “P-104” and another calls it “104-P,” integration becomes difficult unless a common identity is established.

Master data management (MDM) is the organisational and technical discipline used to create, maintain, govern and distribute reliable master records. MDM commonly involves matching duplicate records, defining a system of record, assigning ownership, managing changes and distributing approved values to consuming systems.

Reference data are controlled sets of permitted values used to classify or describe other data. Examples include country codes, currency codes, order statuses and industry categories. Reference data may appear simple, but inconsistent codes can produce serious analytical errors.

Metadata are data about data. They may describe:

• the meaning and definition of a field;
• its data type, unit and permitted values;
• its source system and owner;
• when it was created or updated;
• how it was transformed;
• its sensitivity classification and retention period;
• relationships to other datasets.

Technical metadata describe schemas and data types. Business metadata explain business meanings and rules. Operational metadata describe processing times, failures and volumes. A data catalogue makes metadata searchable and helps users discover appropriate datasets.

Data lineage records the origin and transformations of data from source to report or model. Lineage supports troubleshooting, auditability, impact analysis and trust. If a revenue dashboard changes, lineage helps identify which source tables, transformation rules and reports may be affected.

5. Data Pipelines and Integration

A data pipeline is a controlled sequence through which data move from source systems to destinations such as a warehouse, lake, dashboard or machine-learning system. A typical pipeline includes:

1. Source identification: determine systems, owners, access rights and collection conditions.
2. Extraction or ingestion: obtain data through files, database connections, APIs, event streams or sensors.
3. Validation: check formats, required fields, ranges, duplicates, referential relationships and record counts.
4. Transformation: standardise codes, convert types, join sources, derive fields and apply business rules.
5. Storage: write data to a suitable raw, staging, curated or serving layer.
6. Orchestration: schedule and coordinate dependent tasks.
7. Monitoring: record success, failure, latency, volumes and quality measures.
8. Publication: make approved data available to reports, analysts or applications.

ETL transforms data before loading them into the target. ELT loads data first and transforms them within the target platform. ETL may be appropriate when strict transformation and validation are required before storage; ELT may be attractive where the target platform provides scalable processing and raw data must be retained. Many organisations use both approaches.

Batch processing moves data at scheduled intervals, such as nightly. Streaming processing handles events continuously or with very low delay. Batch processing is often simpler and sufficient for financial reporting. Streaming may be necessary for fraud detection, operational alerts or real-time recommendations.

Reliable pipelines should be repeatable and, where possible, idempotent: rerunning the same processing step should not create unintended duplicate results. They should also maintain logs, retry transient failures, isolate invalid records and alert responsible staff.

A pipeline is not merely a technical connection. It is also a chain of accountability. Each stage should have an owner, documented assumptions, quality checks and a defined response when data do not meet expectations.

6. Foundational SQL for Analytics

Structured Query Language, or SQL, is widely used to retrieve and manipulate relational data. The following examples use the tables described earlier.

A basic query selects columns and rows:

SELECT CustomerID, CustomerName, Region
FROM Customers
WHERE Region = 'North';

SELECT identifies fields, FROM identifies the table, and WHERE filters rows. Text values are normally enclosed in single quotation marks.

Sorting is performed with ORDER BY:

SELECT OrderID, OrderDate, OrderStatus
FROM Orders
ORDER BY OrderDate DESC;

Aggregation summarises records. For example:

SELECT Region, COUNT(*) AS CustomerCount
FROM Customers
GROUP BY Region;

COUNT(*) counts rows. Other common aggregate functions include SUM, AVG, MIN and MAX. A WHERE condition filters individual rows before grouping, while HAVING filters groups after aggregation:

SELECT CustomerID, COUNT(*) AS OrderCount
FROM Orders
WHERE OrderStatus = 'Completed'
GROUP BY CustomerID
HAVING COUNT(*) >= 5;

A join combines related tables:

SELECT o.OrderID, c.CustomerName, o.OrderDate
FROM Orders AS o
JOIN Customers AS c
  ON o.CustomerID = c.CustomerID;

An INNER JOIN returns only matching records. A LEFT JOIN returns every row from the left table and matching rows from the right table; unmatched right-side fields usually appear as NULL. The choice of join affects the result and must reflect the analytical question.

To calculate revenue from order lines, an analyst may write:

SELECT ProductID,
       SUM(Quantity * UnitPrice) AS GrossSales
FROM OrderLines
GROUP BY ProductID;

The expression Quantity × UnitPrice is appropriate only if UnitPrice is expressed per unit and discounts, taxes, returns and currency conversions are either excluded intentionally or handled elsewhere. SQL syntax cannot by itself guarantee that a measure has a correct business meaning.

NULL means that a value is missing or unknown; it is not automatically equal to zero. For example, AVG generally ignores NULL values, which can affect interpretation. Analysts should determine whether missingness means “not collected,” “not applicable,” “unknown” or “zero.”

SQL also supports subqueries, common table expressions, window functions and data-definition commands. At diploma level, the most important habits are to inspect table definitions, qualify fields in joins, test row counts, check for duplicates and validate results against known totals.

7. Governance, Ethics, Privacy and Security

Data governance defines decision rights, responsibilities, policies and standards for managing data. Typical roles include data owners, data stewards, custodians, engineers, analysts and privacy or security officers. Governance should clarify who may create, change, approve, access and delete data.

Important controls include:

• role-based access and least privilege;
• encryption during transmission and storage;
• authentication and audit logging;
• retention and secure disposal schedules;
• backup and recovery procedures;
• quality monitoring and issue-management processes;
• documentation of definitions, lineage and transformations;
• incident-response procedures.

Privacy requires more than removing names. Location, device identifiers, transaction patterns and combinations of apparently harmless attributes may identify individuals. Organisations should collect data for specified purposes, limit collection to what is necessary, protect it appropriately and respect applicable privacy law and contractual obligations. Requirements differ by jurisdiction, so analysts must consult organisational policy and legal specialists rather than assume that one rule applies everywhere.

Ethical analytics also concerns fairness, transparency and human impact. A dataset may underrepresent certain groups; a scoring model may reproduce historical discrimination; or a legitimate prediction may be used in an unfair way. Data provenance, consent, purpose limitation and explainability are therefore relevant to data management as well as to modelling.

8. Illustrative Case: Retail Inventory Integration

Background

A fictional regional retailer operates 25 stores and an online shop. Store transactions are held in a relational point-of-sale system, online orders are stored in a cloud application, and suppliers send spreadsheet files. Managers receive different stock reports from different departments.

Problem

The same product appears under several codes. Some supplier files use kilograms and others use units. Store stock is updated nightly, while online stock is updated hourly. As a result, managers experience stockouts despite apparently high inventory, and the organisation cannot reliably compare store and online sales.

Analysis

The retailer identifies three data problems. First, product identity is not mastered across systems. Second, definitions and units are inconsistent. Third, the reporting process lacks documented lineage and quality monitoring. The problem is therefore not solved simply by purchasing a new dashboard.

Approach and Solution

The retailer establishes a product master containing an approved product identifier, description, category, unit of measure and active status. A data steward is assigned responsibility for product definitions, while systems retain their local identifiers in cross-reference tables.

Transaction data are ingested into a governed analytical store. A pipeline validates required fields, converts supplier quantities to standard units, detects duplicate transactions and records rejected rows for review. A warehouse model separates sales facts from product, store, channel and date dimensions. Metadata document the definition of “available stock,” the update frequency and the treatment of returns.

A dashboard displays data freshness and quality warnings alongside inventory measures. Access is restricted according to role, and supplier files are retained only as long as required by policy.

Outcome

The retailer gains a common product view, more consistent inventory measures and an auditable reporting process. Some discrepancies remain because the source systems are updated at different times; these are made visible rather than hidden. Managers can now distinguish a genuine stock problem from a data-latency problem.

Learning Points

1. Integration requires common identifiers and definitions.
2. Data quality must be measured in relation to business use.
3. A warehouse improves consistency, but it cannot correct unknown source errors automatically.
4. Metadata and lineage make analytical outputs explainable.
5. Governance involves people and processes as well as technology.

9. Contemporary Developments and Best Practices

Cloud platforms have made elastic storage and computing widely available, while APIs and event systems have increased the speed at which data can be collected. Self-service analytics enables more employees to explore data, but it also increases the risk of inconsistent definitions and uncontrolled data copies.

Artificial intelligence can assist with data classification, anomaly detection, entity matching, document extraction and natural-language querying. These uses require human review. An automated system may misclassify sensitive data, merge two different customers or generate SQL that appears plausible but applies the wrong business logic.

Effective practice includes:

• begin with the decision and its required data;
• define ownership and business terms before building reports;
• retain appropriate raw evidence while controlling access;
• use automated validation plus human investigation;
• test joins, filters and aggregates on known examples;
• document assumptions, units, time zones and exclusions;
• monitor freshness, completeness and pipeline failures;
• design privacy and security into the process from the beginning;
• retire unused datasets and reports;
• train users in interpretation, not only tool operation.

Data management is a continuing capability rather than a one-time implementation. Business processes, regulations, technologies and definitions change, so data assets require maintenance throughout their lifecycle.

Chapter Summary

Business analytics depends on data that are relevant, accessible, well-defined and trustworthy. Internal data arise from organisational operations, whereas external data provide market, economic, geographic or other contextual information. Structured data fit an explicit schema; semi-structured and unstructured data require more flexible storage or additional processing.

Operational databases support transactions and integrity. Data warehouses support governed, integrated analysis, while data lakes retain large volumes of raw and mixed-format data. Lakehouse approaches seek to combine flexibility with stronger analytical management. Master data establish consistent identities for entities such as customers and products, reference data provide controlled classifications, and metadata explain meaning, origin, quality and permitted use.

Data pipelines move data from sources to analytical destinations through ingestion, validation, transformation, storage, orchestration, monitoring and publication. SQL provides essential tools for selecting, filtering, joining, grouping and aggregating relational data, but correct syntax does not guarantee correct business interpretation. Governance, privacy, security, ethics and accountability must be built into every stage of the data lifecycle.

Key Terms

Data: A recorded representation of an event, entity or observation.

Internal data: Data generated by an organisation's own activities or systems.

External data: Data obtained from sources outside the organisation.

Structured data: Data organised according to a defined schema of fields and records.

Semi-structured data: Data with labels or hierarchy but without one rigid tabular schema.

Unstructured data: Data without a predefined tabular structure, such as text, images or audio.

Database management system: Software that stores, retrieves, updates and controls databases.

Primary key: A field or combination of fields that uniquely identifies a record.

Foreign key: A field that refers to a key in another table.

Normalisation: Designing tables to reduce unnecessary duplication and update anomalies.

Data warehouse: A governed repository designed primarily for integrated reporting and analysis.

Data lake: A repository that stores large quantities of raw or lightly processed data in multiple formats.

Master data: Shared, important entities such as customers, products and suppliers.

Reference data: Controlled values used to classify or describe other data.

Metadata: Data that describe the meaning, structure, origin, use or management of other data.

Data lineage: A record of data origins and transformations through processing stages.

Data pipeline: A controlled sequence that moves and processes data between systems.

ETL: Extract, transform and load; transforming data before loading the target system.

ELT: Extract, load and transform; loading data before transforming it in the target platform.

SQL: Structured Query Language used to work with relational databases.

Review Questions

Short Answer Questions

1. Distinguish internal data from external data and give one example of each.
2. What is the difference between a primary key and a foreign key?
3. Define metadata and state two types of metadata.
4. Why might an operational database be unsuitable for complex historical analysis?
5. What is the purpose of the GROUP BY clause in SQL?

Long Answer Questions

1. Compare structured, semi-structured and unstructured data, including the analytical opportunities and challenges associated with each.
2. Explain the differences among an operational database, a data warehouse and a data lake.
3. Describe the stages of a reliable data pipeline and explain why monitoring is necessary.
4. Explain master data management, reference data and data lineage, using a retail example.
5. Discuss the ethical, privacy and security responsibilities involved in managing business data.

Analytical / Application Questions

1. A company has customer data in a CRM system, purchase records in a point-of-sale database and online reviews in text files. Classify these sources by origin and structure, then recommend an architecture for analysis.
2. Write a SQL query that joins Orders and Customers to count completed orders by customer region. Explain how WHERE, JOIN and GROUP BY affect the result.
3. An inventory report shows a sudden 40% decline after a pipeline change. Design a step-by-step investigation using data quality checks, metadata and lineage.

Multiple Choice Questions with Answers

Question 1

Which example is internal data?

A. A national inflation dataset published by a government agency

B. A company's own customer-service interaction records

C. A competitor's annual report

D. A weather forecast purchased from a provider

Correct Answer: B. A company's own customer-service interaction records

Question 2

Which type of data usually has a predefined tabular schema?

A. Structured data

B. Unstructured data

C. Audio data

D. Free-text data

Correct Answer: A. Structured data

Question 3

What is the primary purpose of a data warehouse?

A. To control physical access to office buildings

B. To support integrated reporting and analysis

C. To replace every operational application

D. To store only unprocessed photographs

Correct Answer: B. To support integrated reporting and analysis

Question 4

Which key uniquely identifies a row in a relational table?

A. Foreign key

B. Reference code

C. Primary key

D. Metadata key

Correct Answer: C. Primary key

Question 5

What does metadata describe?

A. Only the financial value of a dataset

B. Data about the meaning, structure or origin of data

C. Only deleted records

D. The speed of an internet connection

Correct Answer: B. Data about the meaning, structure or origin of data

Question 6

Which SQL clause filters groups after aggregation?

A. WHERE

B. ORDER BY

C. FROM

D. HAVING

Correct Answer: D. HAVING

Question 7

What is a major characteristic of a data lake?

A. It stores only highly normalised transaction tables

B. It can store raw data in multiple formats

C. It cannot store unstructured data

D. It is used exclusively for payroll processing

Correct Answer: B. It can store raw data in multiple formats

Question 8

Which pipeline stage checks for invalid formats, missing required fields and duplicates?

A. Validation

B. Publication

C. Visualisation

D. Deletion

Correct Answer: A. Validation

Question 9

What is master data most concerned with?

A. Shared business entities such as customers and products

B. Temporary computer memory

C. The colour scheme of a dashboard

D. Randomly generated test values only

Correct Answer: A. Shared business entities such as customers and products

Question 10

Which SQL expression calculates total gross sales from quantity and unit price?

A. SUM(Quantity + UnitPrice)

B. COUNT(Quantity * UnitPrice)

C. SUM(Quantity * UnitPrice)

D. AVG(Quantity - UnitPrice)

Correct Answer: C. SUM(Quantity * UnitPrice)