Data Quality, Metadata and Master Data Management

Introduction and Context

A data warehouse is valuable only when its data is sufficiently accurate, understandable, consistent, timely and fit for its intended use. Modern organizations often possess many operational systems, cloud applications, spreadsheets, external feeds and event streams. These sources differ in definitions, formats, identifiers, update frequencies and controls. Consequently, integrating data is not merely an extraction-and-loading problem; it is a problem of meaning, trust, accountability and sustained management.

Data quality, metadata management and master data management (MDM) provide complementary responses. Data quality evaluates whether data satisfies requirements. Metadata explains what data means, where it came from and how it may be used. MDM establishes authoritative, governed representations of important business entities such as customers, products, suppliers and locations. Together, these disciplines support reliable analytics, regulatory reporting, operational decisions and artificial intelligence.

Quality is contextual. A telephone number may be irrelevant in a financial risk model but essential in a customer-service process. A product description may be adequate for a marketing report but inadequate for regulatory labeling. Thus, quality should be defined against explicit business requirements rather than treated as an abstract property of a database.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Explain the dimensions and causes of data quality problems.

2. Design a data-profiling and data-quality rule framework.

3. Calculate and interpret common quality measures.

4. Distinguish technical, business and operational metadata.

5. Explain lineage, data catalogs, business glossaries and observability.

6. Compare reference data management with master data management.

7. Design an MDM operating model, including matching, survivorship and stewardship.

8. Evaluate governance, privacy, ethical and regulatory implications.

9. Propose a practical remediation and monitoring strategy.

1. Data Quality as Fitness for Use

Data quality is the degree to which data meets stated requirements for a particular use. The principal dimensions are:

• Completeness: required values are present.
• Validity: values conform to allowed domains, formats or business rules.
• Accuracy: values correctly represent the real-world object or event.
• Consistency: the same fact is represented coherently across records, systems and time.
• Timeliness: data is available and current within the required time window.
• Uniqueness: an entity or event is not represented more than once when duplication is not intended.
• Integrity: relationships and constraints, such as foreign-key relationships, are preserved.
• Traceability: the origin, transformations and responsible parties can be established.

These dimensions overlap but are not interchangeable. A value can be complete but inaccurate, valid but obsolete, or consistent across systems yet consistently wrong. For example, a date of birth may satisfy a date format rule while describing the wrong person.

1.1 Quality Metrics

For a dataset containing N records, a simple completeness rate for an attribute is:

Completeness = number of records with a non-null required value / N × 100

This measure assumes that null or blank values are the relevant indicators of missingness and that all records have equal importance. If records have different business significance, a weighted measure may be more appropriate:

Weighted quality rate = Σ(wᵢqᵢ) / Σwᵢ × 100

where qᵢ is 1 when record i passes the selected criterion and 0 otherwise, and wᵢ is its business weight. Weighted measures must be used transparently because they may hide poor performance among low-volume but high-risk categories.

Validity and conformity are commonly measured as the proportion of values satisfying a rule. Timeliness may be expressed as the proportion of records loaded or updated within a service-level threshold. A quality score that combines dimensions should not be presented as objective unless the weights, thresholds and treatment of failed records are documented.

1.2 Causes of Poor Quality

Common causes include manual entry errors, ambiguous definitions, weak validation, system migrations, incompatible code sets, duplicate customer creation, delayed interfaces, schema changes, incomplete business processes and intentional circumvention of controls. Quality problems are often symptoms of process design rather than isolated database defects. Correcting a warehouse table without correcting the source process may cause the defect to reappear at the next load.

2. Data Profiling and Quality Rules

Data profiling is the systematic examination of data structure, content and relationships. It is normally performed before integration and repeated during operation. Profiling identifies distributions, patterns, anomalies and dependencies that are not apparent from schema definitions.

Typical profiling activities include:

• discovering data types, lengths and null rates;
• calculating distinct-value counts and frequency distributions;
• identifying minimum, maximum, mean, median and outliers for numerical attributes;
• examining patterns in addresses, codes and identifiers;
• detecting duplicate or near-duplicate records;
• testing referential integrity;
• comparing values between source and target systems;
• detecting drift in volume, schema and value distributions.

Profiling must preserve privacy. Analysts should prefer masked, tokenized or aggregated values where personally identifiable information is involved.

2.1 Rule Categories

A quality rule should state the condition, scope, severity, owner, measurement method and response. Important categories include:

• Domain rules: country codes must belong to an approved reference set.
• Format rules: an identifier must match a defined pattern.
• Range rules: a percentage must lie between 0 and 100, when the endpoints are permitted.
• Nullability rules: a required field cannot be missing.
• Cross-field rules: a delivery date cannot precede an order date.
• Cross-record rules: an invoice number should be unique within the relevant legal entity.
• Cross-system rules: the customer status used in the warehouse must reconcile with the approved source.
• Temporal rules: a record must arrive within the agreed processing window.

Rules should distinguish hard failures from warnings. A hard failure may block publication of a regulatory dataset; a warning may permit publication with a documented exception.

2.2 Profiling Example

Suppose a customer table contains 10,000 records. The email field is non-null for 9,600 records, 9,300 values satisfy the organization’s syntax rule, and duplicate analysis identifies 180 records belonging to duplicate customer groups. The basic email completeness rate is 9,600/10,000 = 96%. The syntactic validity rate, measured against all records, is 9,300/10,000 = 93%; measured only among populated emails, it is 9,300/9,600 = 96.875%. Both are legitimate measures but answer different questions. A report must state the denominator.

The duplicate rate also requires a definition. If 180 records are involved in duplicate groups, the affected-record rate is 1.8%. It is not necessarily correct to call 180 the number of duplicate pairs, because a group of three records can generate three pairs but represents only one duplicate group.

3. Data Quality Management and Remediation

A sustainable quality program follows a closed loop:

1. Define: identify critical data elements, intended uses and acceptance thresholds.

2. Discover: profile sources, pipelines and target structures.

3. Assess: measure dimensions and rank defects by business impact.

4. Prevent: add validation, controlled vocabularies and process changes at the point of capture.

5. Correct: standardize, enrich, deduplicate or route exceptions for human review.

6. Monitor: track quality, incidents, drift and remediation performance.

7. Learn: analyze root causes and revise rules, processes and ownership.

Remediation may occur at the source, in an integration layer, in a master-data hub or in a consumption layer. Source correction is generally preferable for recurring defects, but it may be slow or impossible when the source is external or lacks ownership. A warehouse-side correction can protect analytical consumers, but it must retain lineage and avoid concealing the original problem.

A useful incident record contains the affected element, rule, detection time, source, business impact, owner, corrective action, due date and verification evidence. Root-cause analysis techniques include process mapping, dependency analysis and the “five whys.”

4. Metadata: The Context That Makes Data Usable

Metadata is data about data. In a warehouse environment it provides the context required to discover, interpret, operate and govern information.

4.1 Types of Metadata

• Technical metadata: table and column names, data types, schemas, keys, partitions, indexes, pipeline configurations and storage locations.
• Business metadata: definitions, approved terms, ownership, classification, calculation logic, policies and intended use.
• Operational metadata: load times, row counts, job status, failure messages, data-quality results, refresh frequency and retention events.
• Lineage metadata: source systems, transformations, mappings, dependencies and downstream consumers.
• Security metadata: sensitivity classification, access policy, masking requirements and retention category.

A column called `status_cd` has technical metadata but little business meaning. A glossary definition can explain whether “active customer” means a customer with an open account, a customer who purchased during the last twelve months, or a legally active party. Such distinctions prevent semantic inconsistency across reports.

4.2 Metadata Repositories and Catalogs

A data catalog inventories data assets and makes them searchable. A mature catalog typically provides technical harvesting, business descriptions, classifications, ownership, quality indicators, lineage, usage information and certification status. Catalogs may be centralized, federated or hybrid. Centralization improves uniform discovery; federation can preserve local control and reduce the need to move metadata from domain systems.

A catalog is not automatically a governance program. It becomes useful when metadata is maintained, linked to access decisions, reviewed by accountable owners and integrated into engineering workflows.

4.3 Business Glossary

A business glossary is a controlled collection of approved business terms and definitions. Each important term should include a definition, synonyms, prohibited alternatives, owner, steward, scope, related terms, calculation logic and examples. A glossary term should be conceptually distinct from a physical column. One business term may be implemented in several systems, and one physical field may have different meanings in different contexts.

4.4 Data Lineage

Lineage describes how data moves and changes from origin to consumption. Technical lineage may show table and column mappings; business lineage explains the movement of a business concept through processes and reports. Lineage supports impact analysis, audit, troubleshooting, regulatory explanation and trust.

Lineage is difficult when transformations occur in undocumented scripts, notebooks, manual spreadsheets or vendor applications. Automated capture is valuable, but it must be supplemented by human validation, especially for semantic transformations and logic embedded in procedures.

5. Data Observability

Data observability applies monitoring and diagnostic practices to data systems. It extends conventional pipeline monitoring, which may report that a job completed successfully even though it loaded zero rows or produced implausible values.

Important observability signals include:

• freshness and delivery delay;
• volume anomalies;
• schema changes;
• distribution and validity changes;
• missing or delayed upstream dependencies;
• failed quality assertions;
• lineage and usage changes.

Observability should be tied to impact. An alert is useful when it identifies the affected asset, likely cause, severity, owner and expected action. Excessive alerts create fatigue. Baselines should account for seasonality, planned closures and known business events. Statistical anomaly detection can identify unusual deviations, but it does not establish that a deviation is wrong; business interpretation remains necessary.

6. Reference Data and Master Data Management

Reference data consists of controlled values used to classify or describe other data, such as country codes, currencies, product categories, risk ratings and order statuses. Reference-data management controls definitions, versions, mappings, effective dates and ownership. It is particularly important when systems use different codes for the same concept.

Master data represents core business entities shared across processes, including customer, supplier, product, employee, account, location and asset. Master data is usually less event-oriented than transactional data, but it changes over time and may have multiple legitimate views.

MDM is the organizational and technical discipline for creating, governing, reconciling, distributing and maintaining trusted master representations. It is not simply a single database. It includes governance, identity resolution, data models, workflows, policies and integration patterns.

6.1 MDM Styles

• Registry style: maintains cross-references between source records and a common identifier while leaving detailed records in source systems.
• Consolidation style: collects records in a hub to create a trusted analytical view.
• Coexistence style: shares mastered values with source systems while allowing controlled local maintenance.
• Centralized or operational style: the hub becomes the authoritative place for creating and updating master records.

The appropriate style depends on business authority, latency, regulatory requirements, system capability and organizational readiness. A centralized hub is not always the best first step.

6.2 Identity Resolution and Matching

Matching determines whether two records refer to the same real-world entity. Deterministic matching uses exact identifiers or carefully defined combinations of attributes. Probabilistic matching evaluates evidence such as name, address, telephone and email, assigning scores or probabilities. Standardization, transliteration, nickname handling and address normalization can improve matching but may also create false matches.

Two error types are central:

• False positive: two different entities are incorrectly merged.
• False negative: records for the same entity remain separate.

The appropriate trade-off depends on harm. False positives can be particularly serious when financial accounts, medical records or legal identities are merged. High-risk matches should be routed to trained stewards rather than accepted automatically.

6.3 Survivorship and Golden Records

A golden record is a governed representation assembled from one or more source records. Survivorship rules determine which value is retained when sources disagree. Rules may prioritize an authoritative source, the most recent verified value, the value with the highest confidence, or a steward-approved value. There is no universally correct survivorship rule. Source authority may vary by attribute: a tax system may be authoritative for legal name, while a customer portal may be authoritative for preferred contact details.

A mastered value must retain source contributions, timestamps, confidence, match evidence and history. Otherwise, users cannot challenge or explain the result.

7. Governance, Stewardship and Operating Model

Governance assigns decision rights and accountability. Typical roles include:

• Data owner: accountable for a domain or critical element and its business use.
• Data steward: manages definitions, rules, exceptions and day-to-day quality.
• Data custodian: operates platforms, security controls and technical services.
• Data producer: creates or supplies data.
• Data consumer: uses data for decisions, reporting or services.
• Governance council: resolves cross-domain policy and priority conflicts.

Ownership should be explicit for every critical data element. Stewardship is not merely administrative work; it requires domain knowledge, authority to resolve ambiguity and time to investigate defects.

A useful governance artifact is a quality agreement specifying definition, owner, source, refresh expectation, permitted values, quality threshold, escalation path and exception policy. Thresholds should reflect risk. A 99% completeness target may be inadequate for a field required for every regulatory transaction, while a lower threshold may be acceptable for an optional marketing attribute.

8. Privacy, Ethics and Regulation

Data quality and MDM can increase privacy risk by linking records that were previously separate. Identity resolution may expose sensitive relationships, while catalogs and lineage can reveal the existence of restricted datasets. Programs should apply purpose limitation, data minimization, access control, retention rules, masking and auditable use.

Organizations should distinguish correction from unauthorized alteration. A warehouse should not silently overwrite source history merely to improve a report. Corrections should be documented, reversible where appropriate and traceable to an authorized decision.

Automated matching and quality scoring can reproduce bias. Names from minority linguistic groups may be normalized poorly; address data may be less complete in disadvantaged communities; and confidence thresholds may produce unequal false-match rates. Evaluation should therefore include representative data, error analysis and human appeal mechanisms.

Regulatory obligations vary by jurisdiction and sector, but common expectations include demonstrable accuracy, lineage, access control, retention, explainability and correction processes. Legal review is necessary when data is personal, sensitive, cross-border or used for consequential decisions.

9. Applied Case: Retail Customer and Product Data (Illustrative)

Background

A fictional international retailer operates stores, an e-commerce platform and a loyalty application. Its warehouse reports customer revenue and product profitability. Executives observe that revenue differs between the finance dashboard and the marketing dashboard.

Problem

The retailer has three customer identifiers, inconsistent country codes, duplicate product records and delayed store feeds. “Active customer” is defined as a loyalty member in one report and as any customer with a purchase in the last year in another. Product costs are missing for imported items, and a failed nightly feed is not detected because the pipeline reports technical completion.

Analysis

Profiling reveals 4.5% null customer country values, 7% invalid country codes, and duplicate groups affecting 3% of customer records. The product feed has a median arrival time within the service target, but a small number of high-volume stores arrive after the reporting cutoff. Lineage analysis shows that marketing uses a consolidated customer table, whereas finance uses transaction-system identifiers and a separate currency conversion process.

Approach

The retailer establishes a glossary definition for customer, active customer, net sales and product cost. It appoints domain owners and stewards, introduces a controlled country reference table and creates a cross-reference identifier for customer records. Deterministic matching is used where verified loyalty identifiers agree; probabilistic matching is restricted to lower-risk cases and uncertain matches are reviewed manually.

The warehouse introduces quality gates for required identifiers, currency codes, product costs and feed freshness. A failed or unexpectedly small store feed creates an incident even when the technical job succeeds. The catalog records source systems, transformation logic, owners and sensitivity classifications. Finance and marketing agree on a certified semantic layer for shared measures.

Outcome

The organization gains a reconciled customer count, a documented definition of active customer and earlier detection of late feeds. Some discrepancies remain because source systems continue to contain historical defects. These are exposed through lineage and quality dashboards rather than concealed. The retailer therefore improves trust without claiming that a single mastered table has solved every semantic or process problem.

Learning Points

The case demonstrates that quality measurement must be connected to business definitions, that technical success is not data success, and that MDM requires governance as well as matching technology. It also illustrates why different attributes may require different authoritative sources and why unresolved exceptions should remain visible.

10. Contemporary Developments and Best Practices

Cloud data platforms, data products, data mesh architectures and machine-learning services have increased the number of producers and consumers. These developments make federated ownership and machine-readable contracts more important. A data contract can specify schema, semantics, quality expectations, compatibility and change-notification obligations between a producer and consumers.

Generative AI can assist with metadata extraction, glossary suggestions, anomaly explanation, rule generation and natural-language search. It should not be treated as an authority. Generated definitions and inferred lineage require human validation, and sensitive data must not be exposed to inappropriate models. AI-based matching requires bias testing, threshold governance and an explanation of evidence.

Recommended practices are:

1. Begin with critical data elements and high-impact use cases.

2. Define business terms before designing aggregate quality scores.

3. Record denominators, thresholds, exceptions and measurement windows.

4. Prefer prevention at the point of capture to repeated downstream cleansing.

5. Preserve source history, lineage and match evidence.

6. Combine automated checks with accountable human stewardship.

7. Monitor freshness, volume, schema and distributions, not only job status.

8. Treat reference data as governed information with versions and effective dates.

9. Design MDM around decision rights and business value, not technology alone.

10. Review privacy, fairness, security and regulatory consequences continuously.

Chapter Summary

Data quality is fitness for a defined use and includes completeness, validity, accuracy, consistency, timeliness, uniqueness, integrity and traceability. Profiling discovers defects and patterns; quality rules convert business requirements into measurable controls; remediation must address both symptoms and causes. Metrics are meaningful only when their denominators, thresholds and assumptions are explicit.

Metadata provides technical, business, operational, lineage and security context. Catalogs support discovery, glossaries establish shared meaning, and lineage enables impact analysis and accountability. Data observability extends monitoring from pipeline execution to freshness, volume, schema and distribution behavior.

Reference data supplies controlled classification values, while master data represents shared entities such as customers and products. MDM includes governance, identity resolution, survivorship, history and distribution. Registry, consolidation, coexistence and centralized styles involve different trade-offs. Matching must manage false positives and false negatives, and golden records must preserve evidence and provenance.

Effective programs assign owners and stewards, connect quality to risk, protect privacy, test for bias and use technology within a governed operating model. The central principle is that trusted data is produced by reliable processes, explicit meaning, accountable decisions and continuous observation—not by cleansing software alone.

Key Terms

Data quality: Fitness of data for a defined use.

Completeness: The extent to which required values are present.

Validity: Conformity to permitted formats, domains or rules.

Accuracy: Correct representation of the real-world object or event.

Data profiling: Systematic examination of data structure, content and relationships.

Critical data element: A data element whose quality materially affects risk, compliance or business outcomes.

Metadata: Data that describes the meaning, structure, origin or use of other data.

Technical metadata: Metadata describing schemas, types, keys and technical implementation.

Business metadata: Definitions, ownership, policies and business meaning.

Operational metadata: Runtime information such as load status, freshness and row counts.

Data catalog: Searchable inventory of data assets and their metadata.

Business glossary: Governed collection of approved business terms and definitions.

Data lineage: Record of data origins, transformations and downstream uses.

Data observability: Monitoring and diagnosis of data-system behavior and health.

Reference data: Controlled values used to classify or describe other data.

Master data: Shared data about core business entities.

Master data management: Governance and technology for managing trusted master entities.

Golden record: Governed representation assembled from one or more source records.

Survivorship: Rules for selecting values when source records disagree.

Data steward: Person responsible for managing definitions, rules, exceptions and quality.

Review Questions

Short Answer Questions

1. Why can valid data still be inaccurate?

2. Distinguish technical metadata from business metadata.

3. What is the purpose of a business glossary?

4. Differentiate reference data from master data.

5. What is a false positive in entity matching?

Long Answer Questions

1. Explain the major dimensions of data quality and discuss why quality is context-dependent.

2. Describe a complete data-quality management lifecycle from definition through learning.

3. Explain how catalogs, glossaries and lineage support data governance.

4. Compare registry, consolidation, coexistence and centralized MDM styles.

5. Discuss privacy, fairness and regulatory risks in automated identity resolution.

Analytical / Application Questions

1. A dataset has 50,000 records, of which 48,500 contain a required value and 47,000 pass a validity rule. Calculate both the completeness rate and validity rate using all records as the denominator. Explain why the denominator should be reported.

2. Design five quality rules for a customer master and identify the owner, severity and likely remediation location for each.

3. An organization has conflicting customer counts in finance and marketing reports. Propose a profiling, lineage, glossary and MDM investigation plan, including how unresolved exceptions should be governed.

Multiple Choice Questions with Answers

Question 1

Which statement best describes data quality?

A. The number of tables in a warehouse

B. Fitness of data for a defined use

C. The speed of a database query

D. The size of a data catalog

Correct Answer: B. Fitness of data for a defined use

Question 2

Which activity is an example of data profiling?

A. Assigning employee salaries

B. Calculating null rates and distinct-value counts

C. Deleting all duplicate records without review

D. Selecting a cloud vendor

Correct Answer: B. Calculating null rates and distinct-value counts

Question 3

A table contains 10,000 records, and 9,600 have a non-null required value. What is completeness?

A. 9.6%

B. 90.4%

C. 96%

D. 99.6%

Correct Answer: C. 96%

Question 4

Which metadata type records load times and job status?

A. Business metadata

B. Operational metadata

C. Reference metadata

D. Presentation metadata

Correct Answer: B. Operational metadata

Question 5

What is the primary purpose of a business glossary?

A. To store database indexes

B. To define approved business terms consistently

C. To replace all source systems

D. To encrypt personal data

Correct Answer: B. To define approved business terms consistently

Question 6

Which statement about data lineage is correct?

A. It describes only physical storage capacity

B. It records origins, transformations and downstream uses

C. It is limited to duplicate detection

D. It eliminates the need for data owners

Correct Answer: B. It records origins, transformations and downstream uses

Question 7

Which is an example of reference data?

A. A customer’s purchase history

B. A product’s transaction total

C. An approved country-code list

D. A warehouse load log

Correct Answer: C. An approved country-code list

Question 8

A false positive in entity matching occurs when:

A. The same entity remains represented by separate records

B. Two different entities are incorrectly merged

C. A required field is null

D. A source feed arrives late

Correct Answer: B. Two different entities are incorrectly merged

Question 9

Which practice most directly improves detection of a technically successful but empty load?

A. Monitoring only job completion status

B. Monitoring volume and freshness expectations

C. Removing all quality rules

D. Increasing storage capacity

Correct Answer: B. Monitoring volume and freshness expectations

Question 10

Why should a golden record retain source contributions and match evidence?

A. To make the record physically larger

B. To support traceability, challenge and explanation

C. To prevent all future updates

D. To avoid defining business terms

Correct Answer: B. To support traceability, challenge and explanation