Data Quality, Master Data, and Information Integrity

Introduction and Context

Business analytics depends on information that is sufficiently accurate, complete, consistent, valid, timely, and unique for its intended use. A sophisticated analytical model cannot compensate for defective source data. If customer identities are duplicated, product codes differ across systems, transaction dates are incorrect, or important fields are systematically missing, the resulting dashboards and decisions may be misleading even when the calculations are technically correct.

Data quality is therefore not merely a technical concern. It is a business capability, a governance responsibility, and an ethical obligation. Poor-quality information can cause incorrect credit decisions, failed deliveries, inaccurate financial reporting, discriminatory outcomes, regulatory breaches, and loss of public trust. This chapter explains the principal dimensions of data quality, methods for measuring and improving them, the role of master and reference data, and the relationship between information integrity and responsible business conduct.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the major dimensions of data quality and their relationship to business use.
2. Calculate and interpret common data-quality measures.
3. Distinguish master data, reference data, transactional data, and metadata.
4. Design practical data-quality rules, controls, and remediation processes.
5. Explain how poor information quality creates ethical, legal, operational, and strategic risks.
6. Evaluate a data-quality problem and recommend a governance response.

1. Understanding Data Quality

Data quality is the degree to which data is fit for a defined purpose. The phrase “fit for purpose” is important: a data element may be adequate for one use but inadequate for another. A customer’s approximate age may be acceptable for a broad marketing analysis but not for verifying eligibility for a regulated financial product. Similarly, an address that is sufficient for regional sales reporting may be insufficient for delivery routing.

Quality is consequently contextual rather than absolute. Nevertheless, several dimensions are widely used in governance and analytics.

1.1 Accuracy

Accuracy is the extent to which data correctly represents the real-world object, event, or value it is intended to describe. A customer’s date of birth is accurate if it agrees with a reliable source or verified evidence. An incorrect value may be syntactically valid but factually wrong.

Accuracy can be assessed by comparing records with a trusted source, conducting verification, or examining error samples. Accuracy is often difficult to establish because the true value may not be directly observable. A data-quality team should therefore document the source, verification method, and confidence level.

1.2 Completeness

Completeness concerns whether all required data is present. It has at least two meanings:

• Attribute completeness: whether a required field is populated.
• Record or coverage completeness: whether all expected entities or events are represented.

For a field, a simple completeness rate is:

The measure assumes that missingness can be identified consistently and that the denominator represents the records that should contain a value. A blank value may be legitimate in some contexts, so completeness rules must distinguish “not applicable” from “unknown” and from an accidental omission.

1.3 Consistency

Consistency means that the same data agrees across records, systems, formats, and business rules. A customer may be recorded as “United Kingdom” in one system and “UK” in another. These values may be semantically equivalent, but inconsistent coding can interfere with joins, reporting, and automated rules.

Consistency can be assessed within one dataset or across datasets. Examples include matching customer identifiers between a CRM and billing platform, ensuring that order totals equal the sum of line items, and using one approved currency convention in financial reporting.

1.4 Validity

Validity is conformity to an accepted format, domain, range, type, or business rule. A valid email address must satisfy the organization’s chosen structural rule; a valid product status may be limited to “Active,” “Discontinued,” or “Pending.” Validity does not guarantee truth. A correctly formatted telephone number can still belong to the wrong person.

Common validity controls include:

• data-type checks, such as numeric fields containing numbers;
• range checks, such as a percentage lying between 0 and 100;
• pattern checks, such as an identifier following an approved structure;
• domain checks, such as a country code appearing in an authorized list;
• conditional rules, such as requiring a cancellation date when status is “Cancelled.”

1.5 Timeliness and Currency

Timeliness is whether data is available when needed. Currency is whether it reflects the current state of the world. A stock balance updated every 24 hours may be timely for monthly planning but not for real-time replenishment. A customer address may be accurate when collected but no longer current months later.

A useful measure is data latency:

Both terms must use the same time zone and time unit. Negative or implausible latency may indicate clock errors, incorrect timestamps, or processing defects. Timeliness requirements should be defined by business process rather than by a universal target.

1.6 Uniqueness

Uniqueness means that an entity or event is recorded the appropriate number of times. Duplicate customer records can fragment purchasing history, produce inaccurate customer counts, and cause repeated communications. However, identical-looking records are not always duplicates: two people may share a name and address.

A basic duplicate rate is:

The definition of “unintended duplicate” must be established before measurement. Matching may use an exact identifier, deterministic rules, or probabilistic techniques that combine name, address, telephone number, and other attributes.

1.7 Integrity and Conformity

Integrity refers to preservation of data correctness and relationships throughout collection, storage, transformation, and use. Referential integrity, for example, requires a transaction to point to an existing customer or product record. Processing integrity requires that transformations do not silently alter meaning, lose records, or introduce unauthorized changes.

Quality dimensions overlap. A record can be complete and valid but inaccurate; timely but inconsistent; unique but incomplete. Quality assessment should therefore use a multidimensional profile rather than a single undifferentiated score.

2. Measuring Data Quality

Measurement converts general concerns into evidence for prioritization and control. A data-quality metric should identify the population, rule, numerator, denominator, time period, and acceptable threshold. Without these definitions, two teams may report different results while using the same label.

2.1 Rule-Based Measurement

Suppose an organization requires every active customer to have a valid postal code. Let N be the number of active customer records and n_v the number that pass the postal-code rule. The validity rate is:

If 9,400 of 10,000 active records pass, the validity rate is 94%. The error rate is 6%, assuming every record is classified as either passing or failing. The result should not be described as “94% accurate,” because validity and accuracy are different dimensions.

2.2 Weighted Quality Scores

Organizations sometimes combine dimension scores into an overall indicator:

Here, q_i is the score for dimension i, w_i is its weight, and the weights satisfy ∑w_i = 1. The score is meaningful only if the component measures use comparable scales and the weights reflect explicitly agreed priorities. A weighted score can conceal a severe failure in one critical dimension; therefore, threshold or “stop” rules may be needed in addition to an aggregate score.

2.3 Sampling and Statistical Caution

When checking every record is costly, a representative sample can estimate an error rate. Sampling must be designed carefully. A convenience sample may overrepresent easy-to-check records and understate problems. Stratification by source system, region, product, or risk category can reveal concentrated defects.

A reported quality rate should include the population and sampling method. For high-risk uses, automated profiling should be supplemented by manual review and source verification. Metrics are indicators, not proof of truth.

2.4 Data-Quality Service Levels

A data-quality service level specifies the expected condition of data, such as:

• 99.5% of mandatory invoice fields valid each day;
• fewer than 0.5% unintended duplicate supplier records;
• critical operational feeds available within 15 minutes;
• all reference-code changes approved and logged.

Targets should reflect business impact and cost. A 99.9% target may be reasonable for payment instructions but wasteful for a low-risk internal survey field. Threshold breaches should trigger ownership, investigation, and corrective action rather than merely generate a report.

3. Data Domains and Information Architecture

Data quality cannot be managed effectively without understanding the kinds of data being governed.

3.1 Master Data

Master data describes relatively stable, shared business entities used by many processes. Typical domains include:

• customer and citizen;
• product, service, or material;
• supplier or employee;
• location, legal entity, and account.

Master data is not necessarily unchanging. A customer may change address, and a product may be reclassified. Its distinguishing feature is shared importance and reuse across processes. A single trusted customer identifier enables sales, service, billing, and analytics to refer to the same entity.

3.2 Reference Data

Reference data consists of controlled values used to classify, validate, or interpret other data. Examples include country codes, payment-status codes, industry classifications, currencies, tax categories, and units of measure.

Reference data often appears simple, but changes can have broad consequences. Changing a product-status code or country classification may affect historical reports, regulatory submissions, and machine-learning features. Reference-data owners should define permissible values, effective dates, translations, and versioning rules.

3.3 Transactional and Event Data

Transactional data records business events such as sales, invoices, claims, payments, or shipments. It usually changes more frequently than master data and depends on master and reference data for interpretation. A sales transaction may contain a customer identifier, product identifier, currency code, quantity, and date. Defects in these related domains can make the transaction difficult to interpret or reconcile.

3.4 Metadata

Metadata is data about data. It describes meaning, ownership, format, lineage, permissible values, sensitivity, retention, and transformation history. A data catalogue may record that “customer_status” is an approved field, has three permitted values, is owned by Customer Operations, and is classified as personal data.

Metadata is essential to information integrity because analysts need to know not only a value but also its provenance, definition, timing, and limitations.

4. Master Data Management

Master Data Management (MDM) is the organizational and technical discipline of creating, maintaining, distributing, and governing authoritative records for shared business entities. MDM is not simply a software installation. It requires policies, ownership, definitions, workflows, integration, and sustained monitoring.

4.1 Core MDM Patterns

Three common operating patterns are:

1. Registry: systems retain their own records, while a central index links corresponding entities.
2. Consolidation: records are collected into a central hub for reporting and analytical use; source systems remain operational authorities.
3. Centralized or coexistence model: a governed master record is created centrally and distributed to participating systems, with controlled synchronization.

The appropriate pattern depends on regulatory requirements, system architecture, operational risk, and the organization’s ability to change source processes.

4.2 Identity Resolution and the Golden Record

A golden record is a preferred representation of an entity assembled from authoritative and reliable attributes. It is not necessarily a single original record. For example, one system may provide a verified legal name, another a current delivery address, and a third a validated tax identifier.

Identity resolution involves deciding whether two records represent the same entity. Deterministic matching uses exact identifiers or fixed rules. Probabilistic matching assigns evidence-based similarity scores. Both approaches require human review for uncertain or high-impact cases. An erroneous merge can combine two people’s information; an erroneous failure to merge can fragment one person’s history. These are privacy and fairness risks as well as data-quality problems.

4.3 Survivorship and Conflict Resolution

When sources disagree, survivorship rules determine which value is retained. A rule may prioritize a verified regulatory source, the most recently confirmed value, or a designated system of record. Rules should be attribute-specific: the billing system may be authoritative for payment terms, while a customer portal may be authoritative for a self-maintained telephone number.

Every important decision should be traceable. The organization should preserve source values, timestamps, confidence, and the reason for selecting a master value where feasible.

5. A Data-Quality Management Lifecycle

A practical lifecycle links measurement to improvement.

5.1 Define

Start by identifying the business process, critical data elements, intended use, risk, owner, and quality expectations. A data dictionary should define terms such as “active customer,” “net sales,” and “delivery date.” Ambiguous definitions are a major source of inconsistency.

5.2 Profile

Data profiling examines distributions, missing values, duplicates, patterns, outliers, relationships, and changes over time. Profiling may reveal that 20% of records use an obsolete category or that a supposedly unique identifier is repeated.

5.3 Diagnose

Diagnosis distinguishes symptoms from root causes. A missing address may result from a poorly designed form, an optional field, a failed integration, a privacy preference, or an address that is genuinely not applicable. Treating every missing value with imputation can conceal the underlying process failure.

5.4 Remediate

Remediation may include correcting records, merging duplicates, enriching values from trusted sources, redesigning forms, changing interfaces, updating reference data, or correcting transformation logic. Data correction should preserve auditability and avoid overwriting evidence without authorization.

5.5 Prevent and Monitor

Prevention places controls as close as possible to the point of capture. Examples include validation at entry, controlled lists, duplicate warnings, reconciliation, automated tests, and exception queues. Monitoring then tracks quality trends and confirms that a fix addresses the cause rather than merely treating a batch of defective records.

A useful governance process assigns each issue a steward, owner, severity, affected process, root cause, remediation deadline, and escalation path.

6. Ethical, Legal, and Governance Implications

Poor-quality data can harm people unevenly. If an address-matching process fails more often for minority names, affected customers may be denied services or subjected to extra verification. If historical data excludes informal workers, an affordability model may systematically underestimate their ability to repay. Missingness is often socially patterned rather than random.

Organizations should therefore ask:

• Who is represented accurately, and who is represented poorly?
• Could an error lead to denial of access, financial loss, surveillance, or reputational harm?
• Are people able to challenge and correct information about them?
• Is data being reused beyond the purpose for which it was collected?
• Can the organization explain the provenance and limitations of a decision-relevant field?

Privacy principles such as purpose limitation, data minimization, security, retention limitation, and individual rights are closely connected to quality. Collecting excessive data increases exposure and creates more opportunities for error. Conversely, deleting data without a retention rationale may prevent accountability or make correction impossible.

A governance program should include access controls, segregation of duties, audit logs, change approval, incident response, retention rules, and documented accountability. Legal requirements vary by jurisdiction, but organizations should not treat compliance as the only reason to maintain quality. Ethical responsibility extends to foreseeable harms even where a specific legal rule is unclear.

7. Technology and AI Considerations

Modern platforms can automate profiling, entity resolution, lineage capture, anomaly detection, and validation. Machine-learning systems can identify unusual patterns or predict probable duplicates. However, automation does not make quality objective. Models learn from historical decisions and may reproduce biased labels or inaccurate assumptions.

AI-generated or automatically enriched data requires provenance, confidence indicators, human oversight, and correction mechanisms. An inferred attribute should not be presented as a verified fact. Data used to train or evaluate AI systems should be checked for duplication, leakage, outdated representations, inappropriate proxies, and group-specific error rates.

Automated quality rules can also create false positives. A strict pattern may reject a legitimate international address; an anomaly detector may flag a genuine unusual transaction. Controls should support review rather than force unquestioned correction.

8. Worked Example: Assessing a Customer Dataset

An analyst profiles 10,000 active customer records. The organization requires a non-missing customer identifier, a valid country code, and a unique identifier. Results are:

• 9,800 records have non-missing identifiers.
• 9,600 records have valid country codes.
• 150 records are unintended duplicate customer records.

The identifier completeness rate is:

The country-code validity rate is:

The duplicate rate is:

These measures should not be averaged automatically. The business may regard a missing identifier as a critical failure because it prevents reliable linking, while a country-code defect may be less serious for a domestic service. The analyst should investigate overlap among defects: some records may be both incomplete and duplicated. Without that information, adding error counts could double-count records.

A suitable response is to assign the customer-data owner, identify the source processes producing missing identifiers, review the duplicate-matching method, correct high-risk records, and introduce entry and integration controls. The result should be monitored over time rather than treated as a one-off cleansing exercise.

9. Illustrative Case: Northstar Retail’s Customer Master

Background

Northstar Retail operates physical stores, an online shop, a loyalty programme, and a customer-service platform. Each system created customer records independently. Senior management noticed that reported customer counts differed substantially across systems.

Problem

The organization found three major defects. First, customers were often registered with variations in name and address. Second, the online system used country names while the finance system used numeric country codes. Third, the loyalty platform updated addresses only weekly, while delivery operations needed current information. These defects caused duplicate marketing messages, incomplete purchase histories, failed deliveries, and unreliable customer-value analysis.

Analysis

A profiling exercise found that 8% of apparent customers were probable duplicates, 4% lacked a usable country code, and 12% of delivery addresses had not been updated within the service’s stated two-day requirement. Interviews showed that no business owner had defined “customer,” no system was authoritative for each attribute, and staff were rewarded for rapid registration rather than accurate capture.

Approach and Solution

Northstar established a customer-data owner and a network of data stewards. It defined a customer identifier, approved country reference data, and attribute-level authority rules. A matching service linked records using verified identifiers where available and probabilistic matching for uncertain cases. Potential high-impact merges were routed to trained staff.

The company also redesigned registration forms, made critical fields conditionally mandatory, recorded consent and contact preferences separately, and introduced daily exception reports. Historical values were retained with timestamps rather than silently overwritten. Dashboards displayed quality by source system and business process.

Outcome

After three months, the duplicate rate fell substantially, delivery failures declined, and customer-value reports became more reliable. The project did not eliminate all disagreement: some customers used shared household addresses, and some intentionally restricted data sharing. Those cases were treated as legitimate privacy or business conditions rather than automatically “corrected.”

Learning Points

The case illustrates that MDM is a socio-technical change programme. Technology supported matching and synchronization, but definitions, ownership, incentives, privacy controls, and staff judgment were equally important. It also demonstrates why a quality score alone is insufficient: the organization needed to understand causes, affected people, and operational consequences.

10. Challenges and Limitations

Data-quality programmes face several recurring challenges:

• Conflicting definitions: different departments use the same term differently.
• Legacy systems: old platforms may lack identifiers, history, or integration capabilities.
• Cost and prioritization: perfect quality is usually impossible and economically unjustified.
• Dynamic reality: records become outdated as people, products, and relationships change.
• False confidence: high validity rates may conceal inaccurate or biased values.
• Over-cleaning: aggressive standardization can destroy meaningful distinctions.
• Privacy tension: enrichment may improve completeness while violating expectations or legal restrictions.
• Ownership gaps: technical teams may detect defects while business teams control the process that creates them.

Quality should therefore be risk-based. Critical data elements used in payments, safety, legal reporting, eligibility decisions, or essential services deserve stronger controls than low-risk descriptive fields.

11. Best-Practice Framework

A mature organization should:

1. Define data elements, business terms, owners, and intended uses.
2. Classify critical data according to operational, legal, financial, and human impact.
3. Measure quality by explicit dimensions and documented rules.
4. Establish authoritative sources and attribute-level survivorship policies.
5. Use controlled reference data with versioning and effective dates.
6. Apply preventive validation at capture and integration points.
7. Preserve lineage, provenance, correction history, and audit evidence.
8. Investigate root causes rather than repeatedly cleansing symptoms.
9. Provide accessible correction and challenge processes for affected people.
10. Monitor subgroup impacts and ethical risks, not just overall averages.
11. Combine automated controls with human review for ambiguous or high-risk cases.
12. Reassess quality requirements when business purpose, regulation, or technology changes.

Chapter Summary

Data quality is fitness for a defined purpose and must be evaluated across accuracy, completeness, consistency, validity, timeliness, uniqueness, and integrity. These dimensions are related but not interchangeable. Measurement requires explicit populations, rules, denominators, thresholds, and interpretation.

Master data represents shared business entities, while reference data supplies controlled classifications and valid values. Transactional data records events, and metadata explains meaning, ownership, lineage, and limitations. Master Data Management combines governance, processes, identity resolution, authoritative sources, and technology; it is not merely a database project.

Effective quality management follows a lifecycle of definition, profiling, diagnosis, remediation, prevention, and monitoring. Ethical and legal responsibilities arise because information errors can produce unequal and serious harms. Privacy, transparency, correction rights, auditability, and human oversight are central to information integrity. AI can assist with profiling and matching, but automated outputs require provenance, confidence, bias assessment, and review.

Key Terms

Data quality: The degree to which data is fit for a defined purpose.

Accuracy: The extent to which data correctly represents reality.

Completeness: The presence of required values, records, or coverage.

Consistency: Agreement of data across systems, records, and rules.

Validity: Conformity to defined formats, domains, ranges, or business rules.

Timeliness: Availability of data when required for use.

Currency: How well data reflects the current state of the world.

Uniqueness: The absence of unintended duplicate records or events.

Information integrity: Preservation of correctness, meaning, relationships, and traceability.

Master data: Shared, important data describing core business entities.

Reference data: Controlled values used to classify, validate, or interpret data.

Transactional data: Data recording business events such as sales or payments.

Metadata: Data describing the meaning, structure, ownership, lineage, and use of data.

Master Data Management: Governance and technology for creating and maintaining trusted shared entity records.

Golden record: A preferred representation assembled from authoritative information sources.

Identity resolution: Determining whether records refer to the same real-world entity.

Data steward: A person responsible for operational oversight of defined data.

Data lineage: The documented history of data sources, transformations, and destinations.

Data latency: The elapsed time between an event occurring and data becoming available.

Survivorship rule: A rule determining which value is retained when sources disagree.

Review Questions

Short Answer Questions

1. Why is data quality described as fitness for purpose rather than an absolute property?
2. Distinguish accuracy from validity using an example.
3. What is the difference between master data and reference data?
4. Why should duplicate rates be defined carefully before measurement?
5. What role does metadata play in information integrity?

Long Answer Questions

1. Explain the major dimensions of data quality and discuss how they can interact.
2. Describe the data-quality management lifecycle from definition through monitoring.
3. Compare registry, consolidation, and centralized MDM patterns.
4. Explain why poor-quality data creates ethical as well as operational risks.
5. Discuss the benefits and limitations of using AI for entity resolution and data-quality monitoring.

Analytical / Application Questions

1. A dataset contains 25,000 records, of which 24,250 have complete mandatory fields and 500 are confirmed unintended duplicates. Calculate the completeness rate and duplicate rate, then explain why these figures should not be combined into one unqualified quality claim.
2. A bank discovers that its address-matching algorithm fails more often for customers with non-Western names. Analyze the possible data-quality, fairness, privacy, and governance risks, and recommend controls.
3. Design an MDM approach for a university whose admissions, learning, finance, and alumni systems use different student identifiers. Specify ownership, matching, reference data, quality metrics, and correction processes.

Multiple Choice Questions with Answers

Question 1

Which dimension of data quality concerns whether a value conforms to an approved format or domain?

A. Accuracy

B. Validity

C. Timeliness

D. Uniqueness

Correct Answer: B. Validity

Question 2

A dataset contains 8,000 expected values, of which 7,600 are non-missing. What is the completeness rate?

A. 5%

B. 40%

C. 95%

D. 105%

Correct Answer: C. 95%

Question 3

Which statement best describes master data?

A. It records only individual financial transactions

B. It consists exclusively of system log files

C. It describes shared business entities used across processes

D. It is always historical and never changes

Correct Answer: C. It describes shared business entities used across processes

Question 4

What is the principal purpose of reference data?

A. To provide controlled values for classification and validation

B. To replace all transactional records

C. To store only unstructured documents

D. To eliminate the need for data ownership

Correct Answer: A. To provide controlled values for classification and validation

Question 5

If data becomes available 30 minutes after an event occurs, the 30 minutes represents what?

A. Accuracy

B. Data latency

C. Uniqueness

D. Survivorship

Correct Answer: B. Data latency

Question 6

Why can a valid email address still be inaccurate?

A. Validity and accuracy measure different properties

B. Validity always concerns completeness only

C. Accuracy applies only to numerical data

D. Email addresses cannot be validated

Correct Answer: A. Validity and accuracy measure different properties

Question 7

Which MDM activity determines whether two records refer to the same real-world entity?

A. Data retention

B. Identity resolution

C. Reference versioning

D. Latency measurement

Correct Answer: B. Identity resolution

Question 8

Why should a weighted overall quality score be used cautiously?

A. It cannot contain numerical values

B. It may conceal a severe failure in one critical dimension

C. It always produces a score above 100%

D. It measures only metadata

Correct Answer: B. It may conceal a severe failure in one critical dimension

Question 9

Which practice best supports information integrity when correcting a master record?

A. Silently overwrite the old value

B. Delete all conflicting source values

C. Preserve provenance, timestamps, and correction history

D. Allow every employee to change the value without logging

Correct Answer: C. Preserve provenance, timestamps, and correction history

Question 10

What is a major ethical risk of inaccurate data in automated decision-making?

A. It guarantees equal outcomes for all groups

B. It can produce systematic and unequal harm

C. It removes the need for human oversight

D. It affects only storage costs

Correct Answer: B. It can produce systematic and unequal harm