Analytics Lifecycle and Problem Formulation

Introduction and Context

Data analytics creates value not merely by applying algorithms to data, but by connecting evidence to decisions. Many analytical initiatives fail even when the statistical model is technically competent. Common causes include an unclear business problem, poorly defined outcomes, unsuitable data, ambiguous success criteria, weak stakeholder alignment, and a mismatch between the model’s output and the decision process.

Problem formulation is therefore a central professional skill in data analytics. It converts an organizational concern—such as rising customer churn, delayed deliveries, excessive energy consumption, or credit losses—into a structured analytical problem. The formulation must specify what is to be predicted, explained, optimized, monitored, or decided; for whom; under which constraints; and according to which measures of success.

This chapter presents the analytics lifecycle, with particular emphasis on CRISP-DM (Cross-Industry Standard Process for Data Mining), complementary lifecycle models, stakeholder analysis, hypothesis development, metrics, constraints, and success criteria. It also examines the difference between business success and model performance, the role of iteration, and ethical and governance considerations.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the purpose and phases of the analytics lifecycle.

2. Apply CRISP-DM and compare it with related frameworks.

3. Translate an organizational challenge into a measurable analytical question.

4. Distinguish business objectives, analytical objectives, hypotheses, metrics, and constraints.

5. Identify stakeholders and reconcile competing requirements.

6. Define appropriate technical, operational, financial, and ethical success criteria.

7. Recognize common problem-formulation failures and improve analytical briefs.

8. Evaluate an analytics initiative through a realistic applied case.

1. From Organizational Challenge to Analytical Problem

An organizational challenge is usually expressed in broad and non-technical language. For example:

• “Customer attrition is too high.”
• “The hospital needs to reduce readmissions.”
• “Our deliveries are unreliable.”
• “The marketing budget is not producing sufficient returns.”

These statements identify a concern, but they do not yet define an analytical problem. A well-formulated analytical problem specifies a decision and the information required to support it.

A useful transformation is:

Challenge → Decision → Analytical question → Data requirement → Action → Success measure

For example:

• Challenge: Customer attrition is increasing.
• Decision: Which customers should receive retention interventions?
• Analytical question: Which active customers are likely to cancel within the next 60 days?
• Data requirement: Historical customer status, usage, complaints, contracts, and interventions.
• Action: Offer an appropriate intervention to selected customers.
• Success measure: Incremental retained margin, subject to contact-cost and fairness constraints.

The final formulation is stronger than “build a churn model” because it identifies the time horizon, target population, decision, action, and economic outcome.

1.1 Descriptive, Diagnostic, Predictive, Prescriptive, and Causal Questions

Analytics questions may serve different purposes:

Descriptive questions ask what has happened. Examples include monthly revenue by region or the distribution of support-ticket categories.

Diagnostic questions ask why an outcome occurred. Techniques may include segmentation, drill-down analysis, process analysis, and carefully designed observational studies.

Predictive questions estimate an unknown or future outcome, such as the probability of loan default within twelve months.

Prescriptive questions recommend an action, allocation, or policy, often subject to constraints such as capacity, budget, or service requirements.

Causal questions ask what would happen if an intervention changed. For example, “What is the incremental effect of a free-trial extension on paid conversion?” Prediction and causation are not equivalent. A variable may predict an outcome without being an effective intervention target.

The analytical method must follow the question. A dashboard may answer a descriptive question but cannot, by itself, establish that a policy caused a change. Similarly, a highly accurate predictor may not identify which action will improve outcomes.

1.2 Unit of Analysis, Population, Outcome, and Horizon

Four elements are especially important in formulation:

• Unit of analysis: the entity represented by one analytical record, such as a customer, transaction, machine, patient episode, or store-day.
• Target population: the entities to which the analysis applies.
• Outcome or target variable: the event, value, or state to be measured or predicted.
• Time horizon: the period over which the outcome is observed or predicted.

Ambiguity in these elements produces invalid conclusions. A customer-level model cannot automatically support account-level decisions if several individuals share an account. A machine-failure model must distinguish failure within seven days from failure within one year. A hospital readmission measure must specify whether the index admission, planned readmissions, and transfers are included.

2. The Analytics Lifecycle

The analytics lifecycle is a structured sequence of activities through which an organization frames, develops, deploys, and evaluates an analytical solution. It is a lifecycle rather than a one-way pipeline because later findings often require earlier assumptions to be revised.

A generalized lifecycle contains the following activities:

1. Problem framing: clarify the decision, stakeholders, objectives, scope, and constraints.

2. Data understanding: identify sources, definitions, quality issues, provenance, and access conditions.

3. Data preparation: integrate, clean, transform, label, and document data.

4. Exploration and analysis: investigate patterns, distributions, relationships, segments, and anomalies.

5. Model development: select methods, construct baselines, train models, and assess uncertainty.

6. Evaluation: determine whether results satisfy technical, business, operational, ethical, and regulatory requirements.

7. Deployment and adoption: embed outputs into systems, workflows, and decisions.

8. Monitoring and learning: detect drift, performance degradation, process changes, and unintended effects.

The lifecycle has feedback loops. For example, data exploration may reveal that the proposed target is measured inconsistently, causing a return to problem framing. Evaluation may show that a model is accurate but unusable because predictions arrive after the decision deadline. Monitoring may reveal population drift and initiate retraining or reformulation.

3. CRISP-DM and Related Frameworks

3.1 CRISP-DM

CRISP-DM is a widely used process model for data-mining and analytics projects. Its six phases are:

1. Business understanding

2. Data understanding

3. Data preparation

4. Modeling

5. Evaluation

6. Deployment

The phases are iterative rather than strictly sequential.

#### Business Understanding

This phase translates organizational goals into a project objective and an analytical objective. It identifies stakeholders, resources, assumptions, risks, constraints, and criteria for success. The output should be a decision-oriented analytical brief rather than a vague request for “insights.”

#### Data Understanding

The analyst inventories available data, investigates how it was generated, examines quality, and determines whether it represents the target population. Important questions include: Who collected the data? For what original purpose? What do missing values mean? Are records duplicated? Are labels delayed or revised? Could the data contain leakage from the future?

#### Data Preparation

This phase includes selecting relevant records and variables, joining sources, correcting errors, managing missingness, engineering features, and constructing labels. Preparation is often the most labor-intensive phase. It must be reproducible and documented because seemingly minor choices can alter the population and outcome definition.

#### Modeling

The analyst selects appropriate methods, establishes baseline comparisons, trains models, tunes parameters, and records assumptions. Model choice should reflect the objective, data structure, interpretability requirements, operational environment, and cost of errors—not only predictive accuracy.

#### Evaluation

Evaluation asks two different questions: Is the model or analysis technically credible? And does it support the intended organizational decision? Evaluation should consider generalization, calibration, subgroup performance, robustness, interpretability, costs, and consequences.

#### Deployment

Deployment may involve a production scoring service, dashboard, report, decision rule, workflow change, or policy. It includes user training, documentation, access control, monitoring, maintenance, and an explicit owner. A model that is never used has no operational value, regardless of its validation score.

3.2 Related Frameworks

KDD (Knowledge Discovery in Databases) emphasizes the broader process of discovering knowledge, including selection, preprocessing, transformation, data mining, and interpretation/evaluation. It is useful when the objective is pattern discovery rather than a clearly specified operational deployment.

SEMMA, associated with SAS, organizes work around Sample, Explore, Modify, Model, and Assess. It is practical for analytical modeling but places less explicit emphasis on business framing and deployment governance than CRISP-DM.

TDSP (Team Data Science Process) provides a collaborative and engineering-oriented lifecycle, including business understanding, data acquisition and understanding, modeling, deployment, and customer acceptance. It is useful in modern team and cloud environments.

Agile analytics applies iterative delivery, short feedback cycles, backlog prioritization, and frequent stakeholder review. It complements rather than replaces analytical rigor. Rapid iteration must not become an excuse for undocumented definitions or uncontrolled experimentation.

MLOps extends the lifecycle into production by emphasizing versioning, automated testing, deployment pipelines, observability, monitoring, retraining, and governance. MLOps is particularly relevant when models operate continuously or influence high-impact decisions.

CRISP-DM remains valuable as a conceptual map, but it should be supplemented by contemporary practices for causal inference, data governance, software engineering, responsible AI, and operational monitoring.

4. Formulating Objectives, Hypotheses, and Questions

4.1 Business and Analytical Objectives

A business objective describes the organizational result sought. It should identify the decision owner, time frame, and intended value. For example: “Reduce avoidable delivery failures in the urban parcel network during the next two quarters without reducing on-time performance for other routes.”

An analytical objective describes the analytical contribution: “Estimate the probability of a delivery failure before route dispatch and identify operational factors associated with elevated risk.”

The analytical objective must not be mistaken for the business objective. “Achieve an area under the ROC curve of 0.90” is a technical target, not a business result.

A strong objective is specific, measurable, achievable with available authority and resources, relevant to a decision, and time-bounded. However, a rigid SMART formulation should not conceal uncertainty. Objectives should also state assumptions and specify what is outside scope.

4.2 Analytical Question Templates

Useful templates include:

• Descriptive: What happened, where, when, and for whom?
• Diagnostic: Which factors are associated with differences in the outcome?
• Predictive: What is the probability or expected value of an outcome for a specified future period?
• Prescriptive: Which feasible action should be selected to improve the objective?
• Causal: What is the effect of intervention A compared with intervention B for a defined population?

A precise predictive question might be: “Using information available at 18:00 on each day, which active subscriptions have a probability above a chosen threshold of cancellation during the following 30 days?” The availability time prevents future information from entering the prediction.

4.3 Hypotheses

A hypothesis is a testable proposition about a relationship, difference, mechanism, or prediction. Examples are:

• H1: Customers experiencing two or more unresolved service incidents in the previous 30 days have a higher cancellation rate than comparable customers without such incidents.
• H2: A reminder delivered seven days before renewal increases completed renewals relative to no reminder.
• H3: A route-level congestion indicator improves failure prediction beyond a baseline using only historical route averages.

Hypotheses should specify the population, exposure or intervention, outcome, comparison, and time window. Exploratory findings may generate hypotheses, but they should not be presented as confirmatory evidence without an appropriate design.

4.4 Baselines and Counterfactual Thinking

Every model should be compared with a meaningful baseline. A baseline may be a majority-class prediction, historical average, current business rule, last-period value, or simple linear/logistic model. Without a baseline, complexity cannot be justified.

For intervention questions, the relevant comparison is often counterfactual: what would have happened to the same or comparable units without the intervention? Randomized experiments provide a strong basis when feasible. In observational settings, analysts must address confounding, selection bias, measurement differences, and temporal changes; predictive associations alone do not establish intervention effects.

5. Metrics, Constraints, and Success Criteria

5.1 Metric Selection

A metric operationalizes an objective. The choice depends on the decision, outcome type, error costs, and data-generating process.

For binary classification, a confusion matrix contains true positives, false positives, true negatives, and false negatives. Common measures include:

\[

\text{Precision} = \frac{TP}{TP+FP}

\]

\[

\text{Recall} = \frac{TP}{TP+FN}

\]

Precision is the proportion of predicted positives that are actually positive; recall is the proportion of actual positives detected. These definitions assume that the positive class and classification threshold have been specified.

Accuracy is

\[

\text{Accuracy} = \frac{TP+TN}{TP+TN+FP+FN},

\]

but it can be misleading with severe class imbalance. For example, predicting “no fraud” for every transaction may produce high accuracy while detecting no fraud. The F1 score is the harmonic mean of precision and recall:

\[

F_1 = \frac{2PR}{P+R},

\]

where \(P\) is precision and \(R\) is recall; it is defined when \(P+R>0\). It gives equal weight to precision and recall and may not reflect business costs when false positives and false negatives have different consequences.

For probabilistic predictions, calibration matters: among cases assigned a probability of 0.7, approximately 70% should experience the event over the stated horizon, subject to sampling variation. Discrimination measures such as the area under the ROC curve assess ranking ability, but do not guarantee calibration or useful decisions.

For regression, mean absolute error is

\[

MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|,

\]

and root mean squared error is

\[

RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}.

\]

Here \(y_i\) is the observed value, \(\hat{y}_i\) is the prediction, and \(n\) is the number of evaluated observations. MAE is expressed in the outcome’s units. RMSE also uses those units but penalizes larger errors more strongly. Both assume that the observations and evaluation procedure are appropriate for the intended use.

For business decisions, metrics may include incremental profit, avoided loss, service-level attainment, time saved, adoption, or return on investment. Financial metrics should include all material costs and should distinguish correlation from incremental impact.

5.2 Constraints

Constraints define what the solution must not violate. Typical constraints include:

• privacy, security, and data-residency requirements;
• legal or regulatory obligations;
• maximum intervention budget;
• staffing, inventory, or processing capacity;
• latency and availability requirements;
• interpretability or auditability requirements;
• minimum performance for protected or vulnerable groups;
• restrictions on using sensitive or proxy variables.

A solution that maximizes a primary metric while violating a hard constraint is not successful. Constraints should be documented before evaluation where possible, rather than introduced selectively after results are known.

5.3 Technical, Business, Operational, and Ethical Success Criteria

Success criteria should operate at several levels:

1. Technical: performance on a pre-specified validation design, stability, calibration, uncertainty, and subgroup behavior.

2. Business: measurable improvement in the target decision or outcome.

3. Operational: acceptable latency, reliability, workflow integration, user adoption, and maintenance cost.

4. Ethical and governance: lawful processing, privacy protection, explainability appropriate to the context, contestability, non-discrimination, and accountability.

These criteria should be stated with thresholds and measurement procedures. For example: “During a three-month controlled pilot, the intervention must increase retained contribution margin per contacted customer, maintain contact rates within budget, and show no material deterioration in service outcomes for monitored customer groups.”

6. Stakeholders and Governance

Stakeholders include the sponsor, decision owner, domain experts, data owners, analysts, engineers, operational users, compliance and legal teams, affected individuals, and sometimes external regulators. Their interests may conflict. A marketing team may prioritize recall, while a contact center may prioritize precision because its capacity is limited. Customers may value privacy even when additional data could improve prediction.

A stakeholder register should record each stakeholder’s role, decisions, information needs, influence, risks, and approval responsibilities. A responsibility matrix can clarify who is responsible for performing work, accountable for decisions, consulted for expertise, and informed of outcomes.

Governance should address data lineage, access, retention, quality ownership, model versioning, approval gates, incident response, and decommissioning. High-impact applications require stronger documentation and human oversight. Analysts must avoid using sensitive data merely because it is available, and must assess whether apparently neutral variables act as proxies for protected characteristics.

Privacy principles include purpose limitation, data minimization, access restriction, secure processing, retention control, and transparency. Depending on jurisdiction and context, laws such as the General Data Protection Regulation or sector-specific rules may impose requirements concerning lawful basis, automated decision-making, rights of access, and impact assessment. Legal compliance is necessary but not sufficient: ethical analysis also considers dignity, autonomy, power asymmetry, and foreseeable harm.

7. Worked Example: Formulating a Subscription-Retention Problem

Consider a fictional streaming service, StreamCo. Its monthly cancellation rate has increased, and executives request “an AI churn model.” This is an illustrative case, not a report of a real organization.

Background and Initial Problem

StreamCo has 2 million active subscriptions. The retention team contacts customers manually, but capacity is limited. The executive concern is not simply prediction accuracy; it is whether targeted interventions can profitably reduce avoidable cancellations.

Problem Formulation

The decision is which eligible customers, if any, should receive a retention offer during the next campaign cycle. The unit of analysis is an active subscription at the campaign decision date. The target population excludes accounts already scheduled for closure, accounts under investigation for fraud, and customers in jurisdictions where the proposed offer is unavailable.

The predictive question is: “For each eligible active subscription, what is the probability of cancellation within the next 30 days using information available at the decision date?” The prescriptive question is: “Which customers should receive which offer, given contact capacity and offer cost?” The causal question is: “What is the incremental effect of each offer on continuation relative to no offer?”

Hypotheses

The team proposes that recent unresolved complaints, reduced viewing activity, payment difficulty, and repeated service interruptions are associated with near-term cancellation. It also proposes that a discount will not have the same incremental effect for every customer. These are hypotheses requiring analysis and, for intervention effects, controlled testing.

Data and Leakage Risks

Candidate data includes subscription history, viewing summaries, support interactions, payment events, service availability, and prior offers. The team discovers that a “cancellation-risk flag” was sometimes updated after a cancellation request had been initiated. Including it would create target leakage because it would use information unavailable at the decision time. The variable is removed from predictive features, although it may remain useful for audit analysis.

The team also finds that complaint categories changed after a support-system migration. This threatens comparability over time. The data owner documents the change, and the evaluation is stratified by system period.

Baseline and Evaluation Design

The baseline is the existing rule: contact customers with two or more complaints in the preceding month. A temporal split is used: earlier months for training, a later month for validation, and the most recent period for a holdout test. A random split alone could overstate performance because records from the same customer and adjacent time periods are dependent.

The model is evaluated using calibration, precision at the contact-center capacity, recall, subgroup performance, and incremental financial outcomes. Suppose the center can contact only 100,000 customers. Ranking all eligible customers by predicted risk and selecting the top 100,000 is operationally meaningful; overall accuracy is not.

Intervention Evaluation

A high-risk customer is not necessarily persuadable. StreamCo conducts a randomized pilot among eligible customers, assigning a discount, a service-support message, or no offer. The primary outcome is continuation at 30 days. Secondary outcomes include margin after discount, complaint volume, and opt-out rates. Randomization supports comparison of average outcomes across assigned groups, assuming implementation is faithful and outcome measurement is consistent.

Outcome and Learning Points

The pilot shows that the predictive model identifies likely cancellers more effectively than the old rule, but the service-support message produces greater incremental margin than the discount for customers with unresolved technical complaints. The model is therefore integrated with an offer-selection policy rather than used to distribute one universal discount.

This case demonstrates several principles: define the decision before choosing a model; prevent temporal leakage; use a capacity-based evaluation; distinguish risk from persuadability; test interventions; and measure incremental value rather than relying on predictive performance alone.

8. Common Failure Modes and Best Practices

Failure Modes

Solution-first thinking occurs when a request for a dashboard, neural network, or chatbot is accepted before the decision problem is understood.

Metric substitution occurs when an easily optimized measure replaces the real objective, such as maximizing clicks instead of profitable conversions.

Target ambiguity occurs when different teams use different definitions of churn, defect, active customer, or successful delivery.

Data availability bias occurs when analysts formulate questions around convenient data rather than important decisions.

Leakage and temporal contamination occur when information created after the prediction point enters training or evaluation.

Ignoring adoption occurs when outputs are not integrated into work practices or users do not trust them.

Overclaiming causality occurs when observational associations are interpreted as intervention effects.

Silent scope expansion occurs when the project population, outcomes, or use cases change without formal review.

Best Practices

Begin with a one-page analytical brief containing the decision, owner, population, unit of analysis, outcome, horizon, action, assumptions, constraints, metrics, and risks. Establish a glossary and data dictionary before model development. Create a simple baseline. Record the information-availability cutoff. Use temporal or grouped validation when the deployment setting requires it. Predefine primary and secondary metrics. Evaluate subgroup performance and unintended effects. Involve end users early. Treat deployment, monitoring, and retirement as part of the original project rather than postscript activities.

A mature team also maintains an assumption log. Each assumption should have an owner, rationale, validation plan, and consequence if false. This makes uncertainty visible and supports responsible decision-making.

9. Contemporary Developments

Modern analytics lifecycles increasingly include real-time data, automated feature pipelines, foundation models, synthetic data, and AI-assisted development. These technologies can accelerate exploration and implementation, but they do not remove the need for problem formulation. Generative systems may produce plausible but unsupported explanations, leak confidential information, or amplify historical bias. Their use should include access controls, provenance checks, human review, evaluation against task-specific criteria, and clear accountability.

Data and concept drift make monitoring essential. Drift may arise from changing customer behavior, policy changes, new sensors, economic conditions, or alterations in data collection. A model can retain good average performance while becoming unreliable for a particular subgroup or newly emerging situation. Monitoring should therefore cover input quality, missingness, prediction distributions, calibration, outcomes, subgroup behavior, latency, intervention rates, and user feedback.

Increasingly, organizations are moving from isolated models toward decision intelligence: coordinated systems that combine data, models, rules, human judgment, optimization, and governance. This reinforces the central lesson of the chapter: analytics value is produced by a complete decision system, not by an algorithm in isolation.

Chapter Summary

The analytics lifecycle connects organizational objectives with data, analysis, decisions, and measurable outcomes. CRISP-DM provides six iterative phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Related approaches such as KDD, TDSP, Agile analytics, and MLOps add complementary perspectives on discovery, collaboration, iteration, engineering, and operations.

Effective problem formulation defines the decision, stakeholders, unit of analysis, target population, outcome, time horizon, action, hypotheses, assumptions, and constraints. It distinguishes descriptive, diagnostic, predictive, prescriptive, and causal questions. Technical metrics such as precision, recall, MAE, and RMSE must be selected according to decision consequences and should not replace business success measures. Baselines, temporal validity, calibration, subgroup analysis, and leakage prevention are essential.

A complete success definition includes technical, business, operational, ethical, and governance criteria. Deployment requires adoption, monitoring, accountability, and maintenance. The StreamCo case illustrates why predicting risk is different from identifying effective interventions and why incremental outcomes matter. At postgraduate level, analytics should be understood as an iterative sociotechnical process involving models, data, people, institutions, and decisions.

Key Terms

Analytics lifecycle: Iterative process connecting problem definition, data, analysis, deployment, and monitoring.

CRISP-DM: Cross-Industry Standard Process for Data Mining, consisting of six iterative phases.

Business objective: Desired organizational result or decision outcome.

Analytical objective: Specific analytical contribution intended to support a business objective.

Unit of analysis: Entity represented by one analytical observation.

Target population: Entities to which an analysis or model is intended to apply.

Outcome variable: Measured result or event of interest.

Time horizon: Defined period over which an outcome is observed or predicted.

Hypothesis: Testable proposition about a relationship, difference, or effect.

Baseline: Simple or existing method used as a reference for comparison.

Data leakage: Use of information unavailable at the intended prediction or decision time.

Precision: Proportion of predicted positive cases that are actually positive.

Recall: Proportion of actual positive cases detected by a classifier.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Constraint: Condition or limit that a solution must satisfy.

Causal effect: Difference in outcomes attributable to an intervention under a defined comparison.

Drift: Change over time in data, relationships, populations, or model behavior.

MLOps: Practices for deploying, operating, monitoring, and governing machine-learning systems.

Stakeholder: Person or group affected by, responsible for, or interested in an analytics initiative.

Success criterion: Predefined condition used to judge whether an analytics initiative has achieved its objectives.

Review Questions

Short Answer Questions

1. What is the difference between a business objective and an analytical objective?

2. Define the unit of analysis and explain why it matters.

3. List the six phases of CRISP-DM.

4. Why can accuracy be misleading for an imbalanced classification problem?

5. What is data leakage, and how can a temporal cutoff help prevent it?

Long Answer Questions

1. Explain how an organizational challenge can be transformed into a decision-oriented analytical question. Use an example.

2. Compare CRISP-DM with MLOps and explain why both perspectives may be needed in a production analytics initiative.

3. Discuss the distinction between predictive and causal questions, including the role of counterfactual thinking.

4. Explain how technical, business, operational, and ethical success criteria should be combined.

5. Critically evaluate the role of stakeholders, governance, and documentation in the analytics lifecycle.

Analytical / Application Questions

1. A bank asks for a model to “identify risky customers.” Formulate a complete analytical brief, including population, outcome, horizon, decision, hypotheses, metrics, constraints, and success criteria.

2. A hospital model predicts readmission with high recall but performs poorly for an important patient subgroup. Analyze whether deployment should proceed and propose an evaluation and remediation plan.

3. A retailer has limited call-center capacity and wants to target likely churners with discounts. Design an evaluation strategy that distinguishes cancellation risk from the incremental effect of the discount.

Multiple Choice Questions with Answers

Question 1

Which CRISP-DM phase primarily translates organizational goals into an analytical objective and success criteria?

A. Business understanding

B. Data preparation

C. Modeling

D. Deployment

Correct Answer: A. Business understanding

Question 2

What is the main purpose of defining an information-availability cutoff?

A. To increase the number of variables

B. To prevent future information from entering a prediction

C. To guarantee model fairness

D. To eliminate missing values

Correct Answer: B. To prevent future information from entering a prediction

Question 3

Which question is causal rather than predictive?

A. Which customers are likely to cancel next month?

B. How many orders were delayed last quarter?

C. What is the effect of a reminder on renewal compared with no reminder?

D. Which regions have the highest average sales?

Correct Answer: C. What is the effect of a reminder on renewal compared with no reminder?

Question 4

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0.50

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 5

Which metric is expressed in the same units as the predicted outcome and uses absolute errors?

A. MAE

B. Accuracy

C. Recall

D. ROC AUC

Correct Answer: A. MAE

Question 6

Why is a baseline important in model evaluation?

A. It guarantees a causal interpretation

B. It provides a reference against which added complexity can be judged

C. It removes the need for validation data

D. It always produces the best operational result

Correct Answer: B. It provides a reference against which added complexity can be judged

Question 7

Which item is an operational constraint?

A. A legal prohibition on using a variable

B. A requirement that predictions arrive within two seconds

C. A hypothesis about customer behavior

D. A definition of the target population

Correct Answer: B. A requirement that predictions arrive within two seconds

Question 8

What does calibration of predicted probabilities assess?

A. Whether predictions are computationally fast

B. Whether predicted probabilities correspond to observed event frequencies

C. Whether all features are normally distributed

D. Whether a model has no false positives

Correct Answer: B. Whether predicted probabilities correspond to observed event frequencies

Question 9

In the StreamCo case, why was a randomized pilot needed?

A. To determine whether risk prediction was possible

B. To estimate the incremental effect of retention interventions

C. To replace all data preparation

D. To eliminate the need for a baseline

Correct Answer: B. To estimate the incremental effect of retention interventions

Question 10

Which practice is most directly associated with monitoring a deployed model over time?

A. Checking for data and concept drift

B. Removing the target variable

C. Using only training-set performance

D. Avoiding stakeholder feedback

Correct Answer: A. Checking for data and concept drift