3. Data Preparation and Quality Engineering

Introduction and Context

Data mining rarely fails because an algorithm cannot be executed. It more often fails because the data supplied to the algorithm is incomplete, inconsistent, duplicated, poorly typed, biased, or interpreted without sufficient context. Data preparation is therefore not a clerical prelude to analytics; it is a central analytical activity. Decisions about what constitutes an error, which records should be retained, how missing values should be treated, and whether extreme observations are valid can materially change the conclusions drawn from a dataset.

Data preparation converts heterogeneous, operationally produced data into a form that is suitable for analysis and modelling. Quality engineering extends this activity by treating quality as something that must be specified, measured, tested, monitored, and improved throughout the data lifecycle. The objective is not to create data that is artificially perfect. Rather, it is to create data that is sufficiently accurate, consistent, complete, timely, valid, and fit for a clearly defined purpose.

This chapter presents an integrated workflow for data integration, deduplication, type correction, missing-value treatment, outlier handling, noise reduction, normalization, standardization, and reproducible validation. It also examines the assumptions and trade-offs behind these operations. The discussion is especially relevant to postgraduate data scientists, who must be able to justify preparation decisions to technical teams, managers, auditors, and subject-matter experts.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain the role of data preparation and quality engineering in a data-mining pipeline.

2. Identify and measure common dimensions of data quality.

3. Integrate data from multiple sources while managing schema, semantic, and temporal inconsistencies.

4. Detect and resolve duplicate records using deterministic and probabilistic methods.

5. Correct data types, units, formats, and invalid values.

6. Select appropriate strategies for missing values, outliers, and noisy observations.

7. Distinguish normalization from standardization and choose transformations appropriate to an analytical task.

8. Design reproducible validation rules, data-quality reports, and audit trails.

9. Evaluate the ethical, operational, and modelling consequences of preparation choices.

Data Quality as Fitness for Use

A data-quality problem is not defined only by whether a value looks unusual. Quality is relative to intended use. A customer address may be adequate for regional marketing but inadequate for legal correspondence. A timestamp rounded to the nearest hour may be sufficient for demand forecasting but unsuitable for fraud investigation. Consequently, quality engineering begins by stating the analytical purpose, the unit of analysis, the relevant time period, and the decisions that the data will support.

Several dimensions are particularly important:

• Accuracy concerns correspondence with the real-world entity or event.
• Completeness concerns the presence of required values or records.
• Consistency concerns agreement across fields, systems, and time.
• Validity concerns conformity to permitted formats, ranges, and domain rules.
• Uniqueness concerns the absence of unintended duplicate representations.
• Timeliness concerns whether data is available and current when needed.
• Integrity concerns preservation of relationships, such as valid customer identifiers and referential links.

These dimensions can conflict. A highly complete dataset produced by aggressive imputation may be less accurate than a smaller dataset containing only observed values. A deduplication process may improve uniqueness while accidentally merging two distinct people. A real-time pipeline may improve timeliness while sacrificing the richer verification available in a delayed batch process. Quality engineering therefore requires explicit trade-offs rather than a single universal cleaning rule.

A useful quality statement has the form: “For purpose P, data from source S is acceptable when metric M remains within threshold T during period R.” For example, a credit-risk feature table might require at least 98 per cent valid account identifiers, no future-dated transactions, and less than 1 per cent unresolved duplicate customer records. Such statements convert vague expectations into testable criteria.

A Preparation Workflow That Preserves Meaning

A robust workflow is iterative rather than strictly linear. A practical sequence is:

1. Define purpose and provenance. Document the question, population, observation unit, source systems, extraction dates, and known restrictions.

2. Profile the raw data. Inspect structure, distributions, missingness, uniqueness, ranges, formats, and relationships before changing values.

3. Integrate sources. Align schemas, keys, units, time zones, coding systems, and business meanings.

4. Standardize representation. Correct data types, labels, encodings, units, and formatting conventions.

5. Resolve identity and duplicates. Determine whether multiple rows represent repeated events, repeated extracts, or the same entity.

6. Treat missingness, invalid values, outliers, and noise. Apply methods justified by domain knowledge and analytical purpose.

7. Transform variables. Normalize, standardize, aggregate, encode, or otherwise engineer features without using information unavailable at prediction time.

8. Validate and document. Run automated checks, compare pre- and post-treatment statistics, record decisions, and obtain appropriate review.

9. Version and monitor. Preserve raw data, transformation code, configuration, outputs, and quality reports so that results can be reproduced.

The order is not absolute. For example, type correction may be necessary before profiling numerical distributions, while duplicate resolution may depend on successful key mapping. The important principle is that each operation should have a stated purpose and an observable effect.

Integrating Heterogeneous Data

Integration combines data from databases, application programming interfaces, spreadsheets, sensors, surveys, logs, and external providers. The difficulty is not merely joining tables. It is reconciling different representations of reality.

Schema and semantic alignment

Two fields with the same name may have different meanings, while different names may represent the same concept. “Revenue” could mean invoiced revenue, collected revenue, or revenue excluding tax. “Date” could mean event date, processing date, or reporting date. Before joining sources, create a data dictionary specifying field meaning, type, unit, valid values, source, and temporal interpretation.

Schema alignment includes renaming fields, mapping categories, reconciling data types, and identifying primary and foreign keys. Semantic alignment may require subject-matter consultation. A code such as “C” could mean “closed,” “commercial,” or “cancelled,” depending on the source system. Automated mapping without contextual verification is risky.

Keys and joins

A join is valid only when the joining fields identify compatible entities at compatible levels of detail. Joining a customer-level table to a transaction-level table creates repeated customer attributes, not duplicate customers. Many-to-many joins can silently multiply rows and inflate totals. Analysts should record expected cardinality—one-to-one, one-to-many, or many-to-one—and test whether the observed relationship conforms to it.

When direct keys are unavailable, linkage may use combinations of name, address, telephone number, email, date of birth, or organizational identifiers. These fields require normalization, such as case folding, whitespace removal, postal-code formatting, and Unicode handling. However, normalization should not erase meaningful distinctions. Removing all punctuation from an identifier may make two different codes appear identical.

Temporal and unit consistency

Dates should be stored using an unambiguous representation and interpreted with a known time zone. A transaction recorded at 23:30 in one system may fall on the next calendar day after conversion to another time zone. Measurements should be converted to common units before comparison or modelling. Mixing kilograms and pounds, or dollars and cents, can produce plausible-looking but materially incorrect features.

A good integration process maintains source-specific values where auditability matters and creates harmonized analytical fields separately. The raw value, conversion rule, and resulting value should be traceable.

Deduplication and Entity Resolution

Duplicates arise for different reasons: repeated file extracts, repeated submissions, system retries, multiple records for one real-world entity, or legitimate repeated events. Treating every repeated row as an error is therefore inappropriate. The first question is whether duplication is defined at the row, event, or entity level.

Deterministic deduplication

Deterministic methods use exact or rule-based matches. Examples include an identical transaction identifier, an exact combination of customer identifier and event timestamp, or a repeated file record with the same source sequence number. These methods are transparent and efficient, but they fail when spelling, formatting, or identifiers differ.

A deduplication rule should specify which record is retained and why. Possible policies include retaining the most recent verified record, the record from the authoritative source, the record with the greatest completeness, or an aggregate of repeated measurements. Deleting records without retaining a reason code weakens auditability.

Probabilistic linkage

Probabilistic entity resolution estimates whether two records refer to the same entity based on agreement patterns across fields. Exact agreement on a stable identifier may be strong evidence; partial agreement on a common surname may be weak evidence. Similarity measures can include edit distance, token similarity, phonetic comparison, and geographic distance. Candidate pairs are often generated through blocking, such as comparing only records sharing a postal region, to reduce computational cost.

A practical system uses two thresholds: pairs above the upper threshold are linked automatically, pairs below the lower threshold are treated as non-matches, and intermediate pairs are reviewed or handled conservatively. The thresholds should be calibrated using labelled examples where possible. False merges can be more damaging than missed matches, particularly in healthcare, finance, and identity management.

Type Correction and Representation Standards

Data types express how values should be interpreted and what operations are valid. Common types include integer, decimal, Boolean, categorical, text, date-time, geospatial, and structured or nested values. A column stored as text may contain numbers, but converting it to numeric can reveal invalid symbols, decimal conventions, or overflow issues.

Type correction should distinguish between syntactic validity and semantic validity. “2025-02-30” may match a date pattern but is not a valid calendar date. A negative quantity may be syntactically numeric yet invalid for a sales-quantity field, although it may legitimately represent a return in another context.

Representation standards should address:

• decimal and thousands separators;
• character encoding and Unicode normalization;
• Boolean conventions such as Yes/No, True/False, and 1/0;
• categorical spelling and capitalization;
• date and time formats and time zones;
• units of measurement;
• precision and rounding;
• special missing-value codes such as 999, -1, or “not available.”

Special codes must not be treated as ordinary measurements. If 999 means “not recorded,” including it in an average produces a false result. Convert such codes to an explicit missing state while preserving the original representation in an audit field when necessary.

Missing Data: Diagnosis Before Treatment

Missingness is not a single condition. A value may be absent because a question was not applicable, a sensor failed, a customer refused to answer, a system did not capture the field, or the value was deliberately suppressed. The reason for missingness affects the appropriate response.

A common conceptual framework distinguishes:

• Missing completely at random (MCAR): missingness is unrelated to observed and unobserved values.
• Missing at random (MAR): missingness is related to observed variables, after accounting for them.
• Missing not at random (MNAR): missingness depends on the unobserved value or an unmeasured process.

These categories are assumptions about a data-generating process, not labels that can always be proven from the dataset. Missingness indicators, comparisons between complete and incomplete groups, process knowledge, and sensitivity analyses can provide evidence.

Treatment strategies

Deletion is defensible when missingness is limited, plausibly random, and the remaining sample still represents the target population. Listwise deletion removes a row if any required field is missing; pairwise methods use available values for particular calculations but may produce inconsistent sample bases.

Simple imputation replaces missing values with a statistic such as the mean, median, or mode. The median is often more robust for skewed variables, while the mode may be used for a categorical field. Simple imputation is easy to implement but reduces variability and can distort relationships. It should often be accompanied by a missingness indicator when the fact of missingness may itself carry information.

More advanced approaches include regression imputation, nearest-neighbour imputation, multiple imputation, and model-based methods. Multiple imputation creates several plausible completed datasets, analyses each, and combines estimates to reflect uncertainty. In predictive modelling, imputation parameters must be learned from the training data only and then applied to validation and test data. Otherwise, information leakage can make performance appear better than it will be in deployment.

A missing value should not be imputed merely because an algorithm requires a complete matrix. Sometimes the correct action is to redesign the feature, exclude the variable, collect the information again, or treat “not applicable” as a meaningful category.

Outliers and Noise

An outlier is an observation that is unusually distant from other observations under a specified definition. It may be a recording error, a legitimate rare event, a member of a different population, or evidence of a previously unknown process. An outlier is not automatically bad data.

Detection methods include domain limits, box-plot rules based on the interquartile range, robust z-scores based on the median and median absolute deviation, distance-based methods, density methods, and model-based techniques such as isolation forests. Each method is sensitive to distributional assumptions, sample size, dimensionality, and contamination level.

The correct response depends on diagnosis. A value outside an instrument’s physical range may be corrected or marked invalid. A genuine high-value transaction may be retained, transformed, or modelled separately. Winsorization caps values at selected quantiles and can reduce the influence of extremes, but it changes the data and may conceal meaningful events. Trimming removes observations and can introduce bias if extreme cases are systematically important.

Noise refers to random error or irrelevant variation. In time series, smoothing may use moving averages, exponential smoothing, or robust filters. In categorical data, rare categories may be grouped only when their meanings are sufficiently similar. Aggregation can reduce random variation but may also remove useful patterns. Noise reduction should therefore be evaluated against the intended task, not judged solely by whether distributions look smoother.

Scaling: Normalization and Standardization

Many algorithms are sensitive to feature scale. Distance-based methods, gradient-based optimization, principal component analysis, and regularized models can be dominated by variables with large numerical ranges. Scaling is not universally required for tree-based models, although consistent feature treatment can still assist interpretation and pipeline management.

Min–max normalization maps a variable to a specified interval, commonly [0, 1]:

x' = (x − xmin) / (xmax − xmin).

It preserves the ordering of values but is sensitive to extreme minimum and maximum values. If future data exceeds the training range, transformed values may fall outside the original interval unless a policy is defined.

Standardization centres a variable around its mean and scales it by its standard deviation:

z = (x − μ) / σ.

The result has mean approximately zero and standard deviation approximately one in the fitting sample. Standardization does not make a variable normally distributed and remains sensitive to extreme observations.

Robust scaling uses the median and interquartile range and can be preferable for skewed data. Logarithmic or other monotonic transformations may reduce right skew, but they require attention to zero and negative values. Scaling parameters must be fitted only on the training partition. Applying a scaler fitted to the complete dataset is a form of leakage.

Categorical variables require different treatment. One-hot encoding creates indicator columns, while ordinal encoding is appropriate only when categories have a meaningful order. High-cardinality categories may require grouping, hashing, frequency encoding, or carefully validated target encoding. Target-based methods are particularly vulnerable to leakage and should be fitted within the training folds.

Reproducible Data Validation

Validation is the evidence that prepared data satisfies declared expectations. It should operate at several levels. Field-level checks test type, format, range, and permitted values. Record-level checks test combinations such as minimum date not exceeding maximum date. Relational checks test foreign-key validity, expected cardinality, and reconciliation between totals. Dataset-level checks test row counts, distribution shifts, duplicate rates, and missingness patterns.

Validation rules may be classified as:

• Hard failures, which block publication, such as an invalid primary key or impossible timestamp.
• Warnings, which permit processing but require review, such as an unusual increase in missingness.
• Informational checks, which support monitoring, such as distribution summaries.

A reproducible pipeline should preserve the raw immutable input, transformation code, dependency versions, configuration, validation results, data dictionary, and quality exceptions. Each output should have a lineage link to its inputs. Version control and automated testing make it possible to rerun the same preparation process and determine why results changed.

Data-quality dashboards are useful when they show trends rather than isolated scores. A sudden fall in completeness may indicate an upstream application change. A rise in unique customer counts may reflect genuine growth or a broken identifier mapping. Monitoring should therefore combine metrics with operational context.

An Illustrative Case: Preparing Customer-Service Data for Churn Analysis

Background

The fictional organisation Northstar Connect wants to predict which subscribers are likely to cancel service within the next 60 days. The analytics team receives billing transactions, customer profiles, service tickets, network-usage summaries, and cancellation records from separate systems.

Situation and problem

Initial inspection reveals several problems. Customer identifiers are missing from older service tickets; names and addresses vary in spelling and format; billing dates use local time while usage data uses coordinated universal time; some monthly charges are recorded in cents and others in currency units; “999” appears in a field intended to represent unanswered survey questions; and a small number of customers have unusually high usage values.

A simple merge produces more rows than expected because customer profiles are joined directly to multiple service tickets. Preliminary churn models also show implausibly high performance because the cancellation table contains events recorded after the prediction date.

Analysis

The team first defines the observation unit as one customer-month and the prediction point as the last day of each month. The target is cancellation during the following 60 days. This definition prevents repeated transaction rows from being mistaken for separate customers and establishes a temporal boundary.

A data dictionary identifies each source’s field meaning, unit, time zone, and update process. Billing amounts are converted to a common unit, timestamps are harmonized to UTC and then mapped to the business reporting date, and the survey code 999 is treated as missing. Duplicate billing transactions are identified through transaction identifiers and source sequence numbers.

Entity resolution uses an exact customer identifier where available. For older tickets without identifiers, records are blocked by normalized telephone number and postal code, then compared on name and address similarity. Ambiguous matches are not automatically merged; they are assigned for review or excluded from customer-level features.

Missing survey responses are retained as missing with an indicator because non-response may differ systematically between customer groups. Usage outliers are investigated against network logs. A few are confirmed measurement errors and removed from feature calculations; others represent legitimate heavy users and are retained, with a log-transformed usage feature added.

Features are computed using only records available by the prediction point. Missing-value statistics and scaling parameters are fitted within the training period. Automated tests verify that every customer-month has a valid customer identifier, no feature uses a future event, duplicate rates remain below the agreed threshold, and row counts reconcile with source totals.

Outcome

The revised model has lower apparent validation performance than the original model but performs more consistently in time-based testing. The organisation gains a defensible feature table, a record of excluded and corrected values, and an operational dashboard that identifies quality deterioration before monthly scoring.

Learning points

The case demonstrates that preparation is inseparable from problem definition. Correct joins depend on the unit of analysis; missing-value decisions require process understanding; outliers require investigation; and temporal validation is essential when predictions concern future behaviour. A lower but credible performance estimate is more valuable than an inflated result produced by leakage.

Challenges, Risks, and Common Mistakes

A frequent mistake is cleaning before profiling. Early replacement of unusual values can destroy evidence about source-system failures. Another is applying a single rule to all variables: mean imputation, universal outlier removal, or min–max scaling may be convenient but analytically unjustified.

Other risks include silent many-to-many joins, converting identifiers into numerical measurements, treating missing codes as real values, using future information in features, fitting transformations before train-test separation, and merging records solely because names are similar. Over-cleaning can remove minority cases and rare but important events. Under-cleaning can produce unstable models and misleading reports.

Quality thresholds can also create perverse incentives. If a team is rewarded only for completeness, it may fill missing values without improving accuracy. If a pipeline rejects every anomaly, legitimate changes in the underlying population may be hidden. Thresholds should be reviewed with domain experts and accompanied by exception handling.

Ethical, Governance, and Professional Considerations

Data preparation affects people, not merely columns. Removing records with missing values may disproportionately exclude vulnerable groups. Imputation can assign plausible but incorrect characteristics to individuals. Entity resolution can incorrectly combine records, leading to harmful decisions. Analysts should assess whether preparation choices change representation across protected or vulnerable groups.

Governance requires documented provenance, lawful and proportionate use, access controls, retention rules, and appropriate protection of personal information. Sensitive identifiers should be minimized, pseudonymized where suitable, and separated from analytical features. Audit trails should record who approved rules, when transformations occurred, and how exceptions were handled.

Transparency is especially important when automated quality rules alter data used in high-impact decisions. Stakeholders should be able to understand whether a value was observed, inferred, corrected, capped, or excluded. Professional responsibility includes communicating uncertainty rather than presenting a cleaned dataset as an unquestionable representation of reality.

Technology and Emerging Practice

Modern data platforms increasingly implement preparation as declarative, versioned pipelines. Data contracts specify expected schemas and quality obligations between producers and consumers. Automated profiling, schema-drift detection, lineage systems, and observability platforms can identify changes in completeness, distribution, freshness, and volume.

Machine-learning systems can assist entity matching, anomaly detection, and imputation, but automation does not remove the need for governance. A learned matching model may reproduce historical linkage errors; an anomaly detector may flag minority behaviour as abnormal; and an imputation model may encode relationships that are inappropriate for a particular decision. Human review, sampling, threshold calibration, and drift monitoring remain necessary.

A growing practice is to test data in the same disciplined manner as software. Data unit tests, integration tests, regression checks, and pipeline-level acceptance tests can be executed automatically in continuous integration environments. Synthetic data and privacy-preserving test datasets support development without exposing production records, although they must be validated for representativeness and utility.

Best-Practice Principles

Effective data preparation follows several principles:

1. Define the analytical purpose, unit of analysis, and time boundary before transforming data.

2. Profile raw data and preserve the original before making changes.

3. Treat semantics, units, keys, and temporal meaning as first-class issues.

4. Distinguish invalid, missing, unusual, and genuinely rare values.

5. Fit imputation, scaling, encoding, and feature-selection operations only on appropriate training data.

6. Prefer transparent, justified rules over unexplained automatic correction.

7. Retain reason codes, exception logs, and lineage for material changes.

8. Validate at field, record, relational, and dataset levels.

9. Examine effects on subgroups and on downstream decisions.

10. Monitor quality after deployment because source systems and populations change.

Chapter Summary

Data preparation is a substantive stage of data mining in which raw, heterogeneous observations are made fit for a defined analytical purpose. Quality must be understood through dimensions such as accuracy, completeness, validity, consistency, uniqueness, timeliness, and integrity. Integration requires alignment of schemas, meanings, keys, units, and time. Deduplication must distinguish repeated rows, repeated events, and multiple records for one entity, using deterministic or probabilistic linkage as appropriate.

Type correction prevents invalid interpretation of identifiers, dates, units, and special missing codes. Missing data should be diagnosed before treatment, and imputation must respect uncertainty and prevent leakage. Outliers and noise require investigation rather than automatic deletion. Normalization, standardization, robust scaling, and other transformations should be selected according to the algorithm and distributional properties, with parameters learned only from training data.

Reproducible validation turns quality expectations into tests, reports, thresholds, and monitoring processes. Effective quality engineering preserves provenance, documents assumptions, considers ethical and subgroup effects, and treats data pipelines as versioned software systems. The central lesson is that preparation decisions shape analytical conclusions; therefore, they must be explicit, defensible, and continuously reviewed.

Key Terms

Data preparation: The process of converting raw data into a form suitable for analysis or modelling.

Data quality: The degree to which data is fit for a stated purpose.

Data profiling: Systematic inspection of structure, distributions, missingness, uniqueness, and validity.

Data integration: Combining data from multiple sources while reconciling their representations and meanings.

Schema: The structural description of fields, types, relationships, and constraints in a dataset.

Data lineage: A record of data origins, transformations, dependencies, and outputs.

Deduplication: Identification and treatment of unintended repeated records.

Entity resolution: Determining which records refer to the same real-world entity.

Blocking: Restricting candidate record pairs before detailed comparison in linkage tasks.

Missing completely at random: A missingness condition unrelated to observed and unobserved values.

Missing at random: Missingness related to observed information after relevant variables are considered.

Missing not at random: Missingness dependent on unobserved values or an unmeasured process.

Imputation: Replacing missing values with estimated or otherwise specified values.

Outlier: An observation unusually distant from others under a defined criterion.

Noise: Random error or irrelevant variation that obscures useful patterns.

Normalization: Rescaling values to a specified range or comparable scale.

Standardization: Centring a variable and scaling it by its standard deviation.

Data leakage: Use of information in model development that would not be available at prediction time.

Data contract: An agreed specification of data structure, meaning, quality, and delivery expectations.

Validation rule: A test that checks whether data satisfies a defined condition.

Review Questions

Short Answer Questions

1. Why is data quality described as fitness for use rather than as an absolute property?

2. Distinguish syntactic validity from semantic validity with one example of each.

3. What is the purpose of blocking in probabilistic entity resolution?

4. Why should imputation parameters be fitted only on training data?

5. State one important difference between normalization and standardization.

Long Answer Questions

1. Explain the major dimensions of data quality and discuss how two of them may conflict in practice.

2. Describe a complete workflow for preparing data for a predictive modelling project, including profiling, integration, treatment, validation, and monitoring.

3. Compare deterministic deduplication with probabilistic entity resolution. Discuss their assumptions, strengths, and risks.

4. Explain how missing values and outliers should be diagnosed and treated, including the consequences of inappropriate deletion or imputation.

5. Discuss how reproducibility, data lineage, and automated validation improve the professional reliability of a data-mining pipeline.

Analytical / Application Questions

1. A dataset contains customer IDs, transaction records, and monthly profiles. After joining the tables, the total transaction amount doubles. Diagnose the likely cause and propose tests and corrective actions.

2. A model for predicting hospital readmission has excellent random-split performance but poor performance when tested on later months. Using this chapter, identify preparation and validation issues that may explain the difference.

3. A survey field uses the values 1–5, 99, and blank, where 99 means “question not applicable.” Design a treatment and validation strategy, explaining how your approach could affect fairness and model interpretation.

22. Multiple Choice Questions with Answers

Question 1

A data-quality rule states that a customer-month table must contain one row per customer per month. Which problem is most directly detected when the table contains multiple rows for the same customer-month without a documented reason?

A. A time-zone inconsistency

B. A uniqueness or grain violation

C. A normalization error

D. A categorical encoding error

Correct Answer: B. A uniqueness or grain violation

Question 2

Why can a many-to-many join produce misleading analytical results?

A. It always removes missing values

B. It converts all numeric fields to text

C. It can multiply rows and inflate aggregates

D. It prevents the use of primary keys

Correct Answer: C. It can multiply rows and inflate aggregates

Question 3

A value of 999 in a temperature field actually means “not recorded.” What is the most appropriate first treatment?

A. Include 999 in the mean

B. Convert it to an explicit missing value and document the rule

C. Scale it using min–max normalization

D. Treat it as a legitimate extreme observation

Correct Answer: B. Convert it to an explicit missing value and document the rule

Question 4

Which situation most strongly supports retaining an apparent outlier?

A. The value violates a physical measurement limit

B. The value resulted from a duplicated file extract

C. The value is confirmed as a genuine rare event relevant to the analytical purpose

D. The value causes a model’s accuracy to decrease

Correct Answer: C. The value is confirmed as a genuine rare event relevant to the analytical purpose

Question 5

What is the principal danger of fitting a standardization transformation on the complete dataset before train-test splitting?

A. It changes categorical variables into dates

B. It can leak information from the test set into model development

C. It guarantees that all variables become normally distributed

D. It removes all outliers from the data

Correct Answer: B. It can leak information from the test set into model development

Question 6

Which statement best describes missing at random?

A. Missingness has no relationship to any observed or unobserved value

B. Missingness is related to observed information after relevant variables are considered

C. Missingness always depends directly on the missing value

D. Missingness occurs only in categorical variables

Correct Answer: B. Missingness is related to observed information after relevant variables are considered

Question 7

A company links records using name, address, and telephone similarity. Which policy is safest for pairs with intermediate similarity scores?

A. Merge every pair automatically

B. Delete both records

C. Send them for review or handle them conservatively

D. Replace all fields with population averages

Correct Answer: C. Send them for review or handle them conservatively

Question 8

Which transformation is generally more sensitive to extreme minimum and maximum values?

A. Min–max normalization

B. Median absolute deviation scaling

C. Categorical one-hot encoding

D. Frequency encoding

Correct Answer: A. Min–max normalization

Question 9

Which validation is relational rather than field-level?

A. Checking that a date follows a valid format

B. Checking that a quantity is numeric

C. Checking that every transaction references an existing customer

D. Checking that a category belongs to an allowed list

Correct Answer: C. Checking that every transaction references an existing customer

Question 10

In the Northstar Connect case, why was a time boundary defined for feature construction?

A. To ensure all variables had equal numerical ranges

B. To prevent future cancellation information from entering the prediction features

C. To eliminate the need for entity resolution

D. To convert missing values into zeros

Correct Answer: B. To prevent future cancellation information from entering the prediction features