Data-Centric Machine-Learning Workflow

Introduction and Context

Machine-learning performance is often discussed as if it were primarily a consequence of algorithm selection: a neural network is compared with a gradient-boosting model, hyperparameters are tuned, and the model with the highest validation score is selected. In professional practice, however, many failures originate earlier. The learning objective may be poorly formulated, labels may be inconsistent, samples may not represent deployment conditions, important variables may be unavailable at prediction time, and evaluation data may have leaked into training. These are data-centric failures.

A data-centric machine-learning workflow treats data as an engineered product rather than an incidental input. It connects a business decision to a precisely defined prediction or inference task, establishes valid data-generation and labeling processes, explores data quality and population structure, transforms variables without violating causal or temporal constraints, evaluates models reproducibly, and manages data responsibly throughout its life cycle.

The workflow is iterative. Findings from exploratory analysis may change the problem definition; labeling analysis may reveal that the target is not operationally meaningful; deployment monitoring may expose a population shift requiring new data collection. Thus, machine learning is not a linear sequence from dataset to model, but a controlled cycle of problem formulation, evidence generation, evaluation, deployment, and revision.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Translate a business or organizational objective into a well-defined learning task.

2. Design data-acquisition and labeling strategies appropriate to the decision context.

3. Explore data quality, distributions, relationships, representativeness, and potential leakage.

4. Select and apply suitable preprocessing and feature-engineering methods.

5. Prevent target leakage and evaluate models under realistic temporal and operational conditions.

6. Diagnose and mitigate class imbalance and unequal subgroup performance.

7. Design reproducible experimentation and maintain trustworthy data and model records.

8. Evaluate ethical, privacy, governance, and regulatory implications of data-centric machine learning.

1. From Business Objective to Learning Task

1.1 Decision-first formulation

A business problem is not automatically a machine-learning problem. The first question is: what decision will be improved, by whom, and at what point in the process? A useful formulation identifies:

• the decision-maker or system;
• the unit of prediction, such as a customer, transaction, machine, or clinical encounter;
• the prediction time;
• the outcome or target;
• the action taken after prediction;
• the costs of false positives, false negatives, delays, and abstentions; and
• constraints such as fairness, latency, privacy, and interpretability.

For example, “reduce customer churn” is incomplete. A more precise task is: “For each active subscription on the first day of a month, estimate the probability that it will be cancelled within the next 30 days, using only information available at that time, so that the retention team can contact a limited number of customers.” This formulation defines the prediction unit, horizon, information boundary, action, and capacity constraint.

1.2 Types of learning tasks

In supervised learning, examples consist of input variables and an outcome. Classification predicts a discrete category, such as fraud or no fraud. Regression predicts a numerical quantity, such as demand or repair cost. Ranking orders items according to estimated relevance or risk. Forecasting predicts future observations while respecting temporal order. Structured prediction assigns outputs with internal structure, such as a sequence of labels.

Unsupervised learning does not use a specified target. Clustering may identify groups, dimensionality reduction may create compact representations, and anomaly detection may identify observations that differ from a reference population. These outputs still require an operational interpretation; a cluster is not automatically a meaningful customer segment, and an anomaly is not necessarily an error or fraud case.

Semi-supervised, active, and weakly supervised learning are useful when labels are expensive. In active learning, the system selects informative examples for human labeling. In weak supervision, noisy labeling rules or indirect signals generate provisional labels. These methods can reduce labeling cost but require careful estimation of label quality.

1.3 Success criteria and baselines

A learning task requires measurable success criteria. Technical metrics should be connected to operational value. For a binary classifier, accuracy may be misleading when the positive class is rare. Precision, recall, specificity, calibration, expected cost, or precision at a fixed review capacity may be more appropriate. A baseline is essential: it may be a current manual process, a simple rule, a majority-class predictor, a linear model, or the previous production model.

A model that improves an offline metric but increases review workload, delays service, or produces unacceptable subgroup errors has not necessarily improved the business process. Evaluation should therefore include both model metrics and decision-level outcomes.

2. Data Acquisition and Dataset Design

2.1 Defining the data-generating process

Data should be understood as observations generated by a process. The process includes users, sensors, software systems, collection incentives, missingness mechanisms, and institutional policies. A dataset is therefore a sample from a target population, often with selection effects.

Before collection, specify the target population and deployment population. The target population is the population about which conclusions are intended; the deployment population is the population encountered when the model is used. If a medical model is trained only on data from one hospital but deployed across hospitals with different equipment and patient demographics, the two populations may differ substantially.

A data specification should record source systems, extraction logic, time range, granularity, identifiers, expected update frequency, data ownership, permitted use, and known limitations. Data lineage should allow an analyst to trace a feature from its origin to the training table and final model input.

2.2 Sampling and selection bias

Random sampling is not always possible, but sampling choices must be explicit. Convenience samples can overrepresent accessible users. Historical operational data may reflect previous policies: for example, only applicants previously approved for manual review may have observed outcomes. This creates selective labels.

Sampling bias occurs when the relationship in the sample differs from the relationship in the deployment population. Common mechanisms include coverage bias, non-response, survivorship bias, temporal sampling bias, and feedback loops. A deployed recommender, for instance, influences which items users see and therefore which future interactions are observed.

When sampling probabilities are known, weighted estimation can sometimes correct population imbalance. If an observation from subgroup or stratum h has selection probability p_h, an inverse-probability weight proportional to 1/p_h can be used, provided the selection mechanism is adequately modeled and positivity holds: relevant population units must have a nonzero chance of being observed. Weighting increases variance when some probabilities are very small, so it should be validated rather than applied mechanically.

2.3 Data contracts and quality controls

A data contract specifies expectations between data producers and consumers. It may define schema, types, permissible ranges, nullability, timestamp semantics, uniqueness, referential integrity, and acceptable freshness. Automated checks should run when data are ingested and before training.

Useful quality dimensions include:

• completeness: whether required values are present;
• validity: whether values conform to domain rules;
• consistency: whether related fields agree;
• uniqueness: whether records are duplicated unexpectedly;
• timeliness: whether data arrive within the required interval; and
• accuracy: whether values represent the real-world quantity adequately.

Data quality is contextual. A missing occupation field may be harmless for one task but critical for another. A timestamp recorded only to the nearest day may be adequate for monthly forecasting but inadequate for fraud detection.

3. Labeling and Ground Truth

3.1 Labels are operational definitions

A label is not simply a fact waiting to be downloaded. It is an operational decision about what counts as an outcome, when it is observed, and how ambiguity is handled. For “default,” one organization may use a payment 30 days overdue, another 90 days overdue, and a third a negotiated loss. These labels describe different phenomena.

A labeling protocol should define the inclusion unit, annotation instructions, positive and negative criteria, time window, abstention or “uncertain” category, escalation process, and version. Annotators should receive representative examples and difficult edge cases. Instructions should distinguish absence of evidence from evidence of absence.

3.2 Label quality and agreement

Human annotation involves ambiguity and disagreement. Agreement can be summarized with raw agreement, but raw agreement may appear high when one category dominates. Cohen’s kappa for two annotators is commonly defined as

κ = (p_o − p_e) / (1 − p_e),

where p_o is observed agreement and p_e is the agreement expected by chance under the annotators’ marginal category frequencies. Kappa is interpretable only with an appropriate annotation design and can be affected by prevalence and biased marginals. It should not be treated as a universal measure of truth.

For more than two annotators, alternatives such as Fleiss’ kappa or Krippendorff’s alpha may be considered. In many applied projects, disagreement analysis is more useful than a single coefficient. Disagreement can reveal ambiguous guidelines, subgroup-specific difficulty, or a genuinely continuous phenomenon being forced into binary categories.

3.3 Noisy, delayed, and censored labels

Labels may be delayed. In churn prediction, customers who have not yet reached the prediction horizon cannot be confidently labeled as non-churners. Treating all recent cases as negative creates right-censoring bias. In survival analysis, censoring methods can be more appropriate than ordinary classification.

Labels may also be influenced by previous model decisions. If a fraud investigation was initiated only for transactions flagged by an earlier system, uninvestigated transactions do not necessarily represent negatives. Such selective observation requires investigation design, random audits, or statistical methods for learning with missing outcomes.

4. Exploratory Data Analysis as Risk Discovery

Exploratory data analysis (EDA) is not merely visualization before modeling. It is a structured search for errors, unusual population structure, leakage, unstable relationships, and assumptions that affect modeling decisions.

4.1 Univariate and multivariate examination

Univariate analysis examines counts, frequencies, central tendency, dispersion, quantiles, and missingness. For a numerical variable, the mean can be strongly affected by extreme values; the median and interquartile range may provide a more robust description. For categorical variables, rare levels may require consolidation or domain review rather than automatic removal.

Multivariate analysis examines relationships among variables and with the target. Scatterplots, grouped summaries, correlation measures, contingency tables, and conditional distributions can reveal nonlinear patterns and interactions. Correlation is not causation, and a weak marginal relationship does not rule out a useful conditional relationship.

EDA should be segmented by time, geography, source system, and relevant protected or operational groups. An overall missingness rate can conceal a critical subgroup problem. For a variable X, missingness can be represented by an indicator M_X equal to 1 when X is missing and 0 otherwise. If M_X is associated with the target or with a subgroup, missingness may carry process information and should be analyzed carefully rather than silently imputed.

4.2 Outliers and anomalies

An outlier may indicate measurement error, a valid rare event, a data-entry problem, or a distributional change. Removing outliers solely because they reduce model performance is dangerous. Investigators should trace unusual values to source systems and domain meaning. Robust transformations, winsorization, or specialized models may be preferable to deletion.

4.3 Distribution shift

Training and deployment data can differ because of changing behavior, policy, equipment, or market conditions. Shift can occur in covariates, labels, or the relationship between inputs and outcomes. A model should therefore be tested on temporally later and operationally distinct data where possible. Monitoring should compare feature distributions, missingness, prediction distributions, and eventual outcomes.

5. Preprocessing and Feature Engineering

Preprocessing converts raw data into representations suitable for learning while preserving the meaning of the task and the boundaries of valid information.

5.1 Numerical variables

Scaling is important for distance-based methods, regularized linear models, and neural networks, although tree-based models are usually less sensitive to monotonic scaling. Standardization transforms a value x using

z = (x − μ) / σ,

where μ is the training-set mean and σ is the training-set standard deviation. The transformation assumes σ is nonzero and must be fitted on training data only. Applying a training-derived transformation to validation and test data prevents information from those sets entering the fitting process.

Robust scaling may use the median and interquartile range when distributions contain extreme values. Logarithmic transformations can reduce right skew for nonnegative variables, but the transformation must match the domain: zero values may require a justified transformation such as log(1+x), and negative values require a different approach.

5.2 Categorical, textual, and temporal variables

One-hot encoding represents categories with indicator variables and is suitable for nominal categories with manageable cardinality. Ordinal encoding should be used only when category order is meaningful. High-cardinality features require careful treatment because target encoding can leak outcomes if computed without out-of-fold or training-only procedures.

Text representations may include token counts, term-frequency/inverse-document-frequency features, embeddings, or domain-specific language models. Temporal features should respect their semantics. Extracting hour, day of week, season, or elapsed time may be useful, but future information must not be incorporated. Cyclical variables such as hour of day can be represented using sine and cosine functions when continuity between the end and beginning of a cycle is meaningful:

x_sin = sin(2πt/P), x_cos = cos(2πt/P),

where t is the position in the cycle and P is its period in the same units.

5.3 Feature engineering principles

Good features encode stable, available, and decision-relevant information. Examples include rolling transaction counts, time since last service, ratios with meaningful denominators, and domain-based interactions. Every engineered feature should have a definition, data source, time window, aggregation rule, missing-value policy, and owner.

Feature engineering can introduce spurious proxies for protected characteristics or institutional practices. A postal code may be predictive but may also reproduce historical segregation or unequal access. Predictive usefulness must therefore be assessed alongside fairness, legitimacy, and deployment consequences.

6. Leakage Prevention and Valid Evaluation

6.1 What leakage means

Data leakage occurs when information unavailable at the prediction time influences training or evaluation. Target leakage uses a variable caused by, or recorded after, the outcome. Train-test contamination occurs when information from validation or test data affects preprocessing, feature selection, imputation, threshold selection, or hyperparameter tuning.

Examples include using a discharge diagnosis to predict admission risk, computing a customer’s lifetime average using transactions after the prediction date, imputing missing values using the full dataset, and repeatedly selecting models based on the test set.

6.2 Temporal and grouped splitting

Random splitting is inappropriate when observations are temporally dependent or when multiple records belong to the same entity. A customer appearing in both training and test sets can make performance appear high because the model recognizes customer-specific patterns. Grouped splitting assigns all records from an entity to one partition.

For forecasting and many operational tasks, training must precede validation and test periods. A rolling-origin design can train on an initial period, validate on a later period, then expand or move the training window. The design should reflect how the model will actually be retrained.

A sound pipeline fits every learned preprocessing step only on the training portion of each fold. Feature selection, dimensionality reduction, resampling, and threshold tuning must also occur inside the relevant training process.

6.3 Evaluation metrics and calibration

For binary classification, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. Precision is TP/(TP+FP), when the denominator is nonzero; recall or sensitivity is TP/(TP+FN); and specificity is TN/(TN+FP). These metrics answer different questions and should be selected according to the decision context.

A probability model is calibrated when among cases assigned probability p, the observed event frequency is approximately p. Discrimination and calibration are distinct: a model can rank cases well while producing probabilities that are systematically too high or too low. Calibration curves, reliability diagrams, and proper scoring rules such as log loss or the Brier score can complement ranking metrics.

For binary outcomes y_i in {0,1} and predicted probabilities q_i, the Brier score is

B = (1/n) Σ from i=1 to n (q_i − y_i)^2.

Lower values are better, subject to the same dataset and outcome definition. The score combines calibration and refinement and should not be interpreted without considering event prevalence.

7. Class Imbalance and Decision Thresholds

Class imbalance means that classes occur at unequal frequencies. It is not inherently a defect; it becomes a problem when the evaluation metric, training objective, or decision rule fails to reflect the costs and information needs of the application.

Possible responses include collecting more minority examples, using class-weighted losses, under- or over-sampling within training folds, anomaly-detection methods, threshold adjustment, and choosing metrics such as precision-recall curves. Synthetic sampling must be applied only to training data. Generating synthetic examples before splitting can place near-duplicates across partitions and create leakage.

The default threshold of 0.5 is not universally correct. Suppose a positive decision has benefit B when correct, cost C_FP for a false positive, and cost C_FN for a false negative. The threshold should be derived from the decision utility and predicted probability, subject to assumptions about calibrated probabilities and constant costs. In practice, costs may vary by case, capacity may be limited, and the model may be allowed to abstain. Threshold selection should therefore be validated using operational simulations rather than chosen from a generic rule.

Performance must be reported by relevant subgroups. A single aggregate score can conceal large differences in recall, false-positive rate, calibration, or access to human review. Fairness criteria can conflict mathematically when base rates differ, so the appropriate criterion must be connected to the domain, legal framework, and harm analysis rather than selected as a purely technical preference.

8. Reproducible Experimentation

Reproducibility means that an experiment can be reconstructed, while repeatability commonly refers to obtaining consistent results under the same conditions. A reproducible experiment records data versions, code revision, feature definitions, split logic, random seeds, environment specifications, hyperparameters, metrics, and artifacts.

A mature workflow separates three datasets or roles: training data for fitting, validation data or cross-validation for selection, and a final test set reserved for an unbiased estimate after major decisions are complete. In small datasets, nested cross-validation can reduce selection bias. In the outer loop, a model is evaluated; in the inner loop, hyperparameters are selected. This is computationally expensive but methodologically stronger when model selection is substantial.

Experiment tracking should include failed runs and rationale for decisions, not only the best result. Version control for datasets is especially important because a query that appears unchanged may produce different records as source systems are corrected. Data and model cards can document intended use, limitations, subgroup performance, training data, and known risks.

Statistical uncertainty matters. A difference in validation scores may reflect sampling variation. Confidence intervals, bootstrap procedures, repeated splits, or paired comparisons can help quantify uncertainty, although dependence and temporal structure must be respected. Practical significance should be distinguished from statistical significance.

9. Responsible Data Management and Governance

Data-centric machine learning creates obligations beyond predictive performance. Data collection should have a legitimate purpose, a defined retention period, access controls, and appropriate security. Personally identifiable information should be minimized, protected, and processed under applicable law and organizational policy. De-identification reduces risk but does not guarantee anonymity, particularly when datasets can be linked.

Consent, notice, purpose limitation, and secondary use require careful consideration. Sensitive attributes may need to be collected for fairness auditing even when they are excluded from prediction. Not collecting protected-group information does not eliminate discrimination; it may make disparities harder to detect.

Governance should assign ownership for data quality, labeling, model approval, monitoring, incident response, and retirement. Automated decisions that affect individuals may require explanations, human review, appeal mechanisms, and documentation of lawful basis. Security threats include unauthorized access, data poisoning, membership inference, and reconstruction attacks.

Responsible practice also examines representational harm, allocation harm, privacy harm, and automation bias. A model should not be deployed merely because it is technically feasible. The organization must be able to monitor outcomes, correct errors, and suspend the system when conditions change.

10. Illustrative Case: Predicting Hospital Readmission

Background

A regional hospital wishes to reduce avoidable readmissions within 30 days of discharge. It proposes a model that identifies patients who may benefit from follow-up support.

Problem formulation

The unit is a hospital admission. The prediction time is the planned discharge decision. The outcome is an unplanned readmission within 30 days, defined using linked hospital records. The action is referral to a care-coordination team with limited weekly capacity. A useful evaluation must therefore include recall among referred patients, precision at the available capacity, calibration, subgroup performance, and evidence that intervention improves outcomes.

Data and labeling analysis

Candidate sources include admission information, laboratory results, prior utilization, medication records, discharge planning data, and demographic information. The team discovers that a post-discharge billing code is highly predictive but is created after the prediction time. It is removed as leakage. Recent discharges cannot yet have a complete 30-day outcome window, so they are excluded from the initial supervised dataset or handled with an appropriate censoring design.

Two clinicians independently label a sample of cases to assess ambiguity in “avoidable” readmission. They find disagreement because some readmissions are clinically necessary. The team consequently separates the measurable outcome, any-readmission, from the more subjective outcome, avoidable-readmission, and documents that the first model predicts occurrence rather than preventability.

Exploration and modeling

EDA shows that missing laboratory values are more common in patients transferred from smaller facilities. Missingness is retained as a process indicator only after clinical review, and the model is tested for subgroup effects. A time-based split is used: earlier admissions for training, a later period for validation, and the most recent period for final testing. All imputers, encoders, feature selection, and class weighting are fitted inside the training process.

Because the care-coordination team can contact only 100 patients per week, the operational metric is recall among the top 100 predicted-risk patients, supplemented by calibration and subgroup analysis. The model is compared with a simple rule based on prior admissions. A calibrated model improves identification over the rule, but performance is lower for patients transferred from smaller facilities. Investigation reveals missing data and different follow-up practices rather than an obvious biological difference.

Deployment and outcome evaluation

The hospital conducts a prospective pilot. Predictions are displayed with an explanation of major contributing factors, but staff retain authority to override the recommendation. The pilot measures not only readmissions but also intervention uptake, staff workload, false reassurance, subgroup access to services, and patient outcomes. If the model identifies risk without increasing effective support, the system has limited value.

Learning points

The case illustrates that defining the outcome, prediction time, intervention capacity, and evaluation population is as important as choosing an algorithm. Leakage prevention, time-aware validation, label analysis, subgroup monitoring, and prospective evaluation are essential. Most importantly, the model predicts risk; it does not establish that an intervention will be effective or that a readmission was preventable.

11. Contemporary Developments and Best Practices

Modern foundation models and automated feature-learning systems can reduce manual representation design, but they do not remove the need for high-quality data, valid labels, leakage controls, and governance. Synthetic data may support testing or privacy-preserving development, yet synthetic records can reproduce bias, omit rare cases, or create unrealistic dependencies. Human-in-the-loop systems increasingly combine model predictions with expert review, active labeling, and abstention.

Recommended practice includes:

1. Begin with the decision, action, and prediction-time information boundary.

2. Maintain a data and label specification with versioned definitions.

3. Profile data by time, source, subgroup, and deployment-relevant segment.

4. Build automated data-quality tests and lineage records.

5. Use temporal or grouped splits whenever the application requires them.

6. Fit preprocessing, resampling, feature selection, and threshold decisions without test-set contamination.

7. Report calibration, uncertainty, subgroup performance, and operational capacity—not only one headline metric.

8. Treat labels as potentially noisy, delayed, selective, or policy-dependent.

9. Track experiments, data versions, code, environments, and rejected alternatives.

10. Establish monitoring, human oversight, privacy controls, and a process for model retirement.

Challenges and Limitations

Data-centric workflows cannot eliminate uncertainty. Some outcomes are inherently ambiguous, delayed, or affected by interventions. Correcting historical bias may require policy and institutional change rather than reweighting alone. Fairness metrics can conflict, and subgroup sample sizes may be too small for precise estimates. Privacy protection may reduce useful signal, while collecting sensitive attributes creates additional governance obligations. Distribution shift can invalidate a model despite excellent historical performance. Finally, causal questions—such as whether an intervention will change an outcome—cannot generally be answered by predictive accuracy alone.

Chapter Summary

A reliable machine-learning system begins with a precise decision-oriented formulation rather than an algorithm. The practitioner must define the prediction unit, time of prediction, outcome, action, constraints, and success criteria. Data acquisition requires attention to population definition, sampling, selection effects, lineage, and quality contracts. Labels are operational constructs that may be ambiguous, noisy, delayed, censored, or selectively observed.

Exploratory analysis is a risk-discovery activity covering distributions, missingness, outliers, relationships, subgroup differences, and distribution shift. Preprocessing and feature engineering must preserve semantic validity and be fitted without access to validation or test information. Leakage prevention requires temporal or grouped splits when appropriate and careful control of every learned transformation. Imbalance should be addressed through data, modeling, metrics, and decision thresholds aligned with actual costs and capacity.

Reproducible experimentation depends on versioned data, code, environments, splits, metrics, and decisions. Responsible data management adds privacy, security, fairness, transparency, human oversight, monitoring, and governance. The central principle is that model quality is inseparable from data quality, task validity, evaluation design, and the consequences of deployment.

Key Terms

Data-centric machine learning: An approach that treats data quality, composition, labeling, and governance as primary determinants of system performance.

Prediction unit: The entity or event for which a prediction is generated.

Prediction time: The precise point at which a prediction is made and information availability is bounded.

Target population: The population about which conclusions are intended.

Data contract: A formal specification of expected data schema, quality, timing, and semantics.

Selection bias: Systematic distortion caused by how observations enter a dataset.

Label leakage: Use of information unavailable at prediction time or caused by the outcome.

Label noise: Incorrect, inconsistent, ambiguous, or unreliable target values.

Censoring: Incomplete observation of an outcome because the relevant observation window has not ended or is otherwise unavailable.

Feature engineering: Construction or transformation of variables to represent useful domain information.

Standardization: Transformation using the training mean and standard deviation, commonly z = (x − μ)/σ.

Temporal split: A train-validation-test division that respects chronological order.

Grouped split: A division that keeps records from the same entity within one partition.

Class imbalance: Unequal prevalence of outcome classes.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Distribution shift: Change in data distributions or relationships between development and deployment.

Reproducibility: Ability to reconstruct an experiment from recorded data, code, settings, and environment.

Data lineage: Traceability of data from source through transformations to model input.

Data poisoning: Deliberate or accidental manipulation of training data to compromise a model.

Human-in-the-loop: A system design in which people review, authorize, correct, or override model outputs.

Review Questions

Short Answer Questions

1. Why must a business objective be translated into a decision-oriented learning task?

2. Distinguish target population from deployment population.

3. What is target leakage? Give one example.

4. Why should preprocessing transformations be fitted only on training data?

5. What does calibration mean for a probabilistic classifier?

Long Answer Questions

1. Explain how data acquisition, labeling, exploratory analysis, and governance interact in a data-centric workflow.

2. Compare random, temporal, and grouped data splits, and identify situations in which each is appropriate.

3. Discuss the causes of label noise, delayed labels, and selective labels, and propose mitigation strategies.

4. Explain how class imbalance should be addressed through metrics, sampling, weighting, and threshold selection.

5. Develop a reproducibility and responsible-management plan for a machine-learning system used in a high-impact domain.

Analytical / Application Questions

1. A fraud model is trained on transactions from January to June and tested by randomly splitting all records. Several customers appear in both sets, and a “case closure date” feature is included. Identify the validity threats and redesign the evaluation protocol.

2. A hospital can review only 50 high-risk cases per day. Propose an evaluation framework that accounts for capacity, calibration, subgroup performance, and the distinction between predicting risk and improving outcomes.

3. A customer-churn dataset has many recent customers whose 30-day outcome is not yet observable. Explain why labeling them as non-churners is problematic and design a more valid data and evaluation strategy.

Multiple Choice Questions with Answers

Question 1

Which statement best defines a decision-oriented learning task?

A. It specifies only the algorithm and hyperparameters to be used.

B. It defines the prediction unit, prediction time, outcome, action, and constraints.

C. It maximizes accuracy regardless of operational consequences.

D. It uses every variable available in the organization’s databases.

Correct Answer: B. It defines the prediction unit, prediction time, outcome, action, and constraints.

Question 2

What is the main purpose of a data contract?

A. To guarantee that a model will be unbiased.

B. To replace exploratory data analysis.

C. To specify expected schema, quality, timing, and semantics of data.

D. To select the best classification threshold.

Correct Answer: C. To specify expected schema, quality, timing, and semantics of data.

Question 3

Which example is most clearly target leakage?

A. Using a patient’s age at admission to predict readmission.

B. Using a transaction amount recorded at purchase time to predict fraud.

C. Using a discharge diagnosis created after discharge to predict admission risk.

D. Using a training-set median to impute a missing training value.

Correct Answer: C. Using a discharge diagnosis created after discharge to predict admission risk.

Question 4

Why are grouped splits useful?

A. They ensure every class has exactly the same frequency.

B. They prevent records from the same entity appearing in both training and test partitions.

C. They automatically correct label noise.

D. They eliminate the need for a validation set.

Correct Answer: B. They prevent records from the same entity appearing in both training and test partitions.

Question 5

In standardization z = (x − μ)/σ, what should μ and σ normally represent?

A. The mean and standard deviation calculated from the test set.

B. The global population parameters, even when unavailable.

C. The mean and standard deviation fitted on the training data.

D. The target mean and target standard deviation.

Correct Answer: C. The mean and standard deviation fitted on the training data.

Question 6

Which metric directly measures the proportion of predicted positives that are actually positive?

A. Recall

B. Specificity

C. Precision

D. Accuracy

Correct Answer: C. Precision

Question 7

What is the primary risk of applying oversampling before a train-test split?

A. The minority class becomes impossible to model.

B. Synthetic or duplicated information can cross partition boundaries.

C. The test set becomes temporally ordered.

D. Calibration is guaranteed to improve.

Correct Answer: B. Synthetic or duplicated information can cross partition boundaries.

Question 8

A calibrated model is one for which:

A. All predictions are either zero or one.

B. Predicted probabilities correspond approximately to observed event frequencies.

C. The model has the highest possible accuracy.

D. Every subgroup has identical base rates.

Correct Answer: B. Predicted probabilities correspond approximately to observed event frequencies.

Question 9

Which practice most directly supports reproducible experimentation?

A. Reporting only the best-performing run.

B. Changing the test set whenever results are disappointing.

C. Recording data versions, code revision, split logic, settings, and environment.

D. Using a more complex model than the baseline.

Correct Answer: C. Recording data versions, code revision, split logic, settings, and environment.

Question 10

Why may collecting protected-group information be useful even when it is excluded from prediction?

A. It guarantees that discrimination cannot occur.

B. It enables subgroup performance and fairness auditing.

C. It removes the need for privacy controls.

D. It ensures that all groups have identical outcomes.

Correct Answer: B. It enables subgroup performance and fairness auditing.