Problem Framing and the Data Science Lifecycle

Introduction

Data science projects rarely fail because a statistical algorithm is unavailable. They more often fail because the wrong problem is solved, the intended outcome is unclear, stakeholders disagree about success, or a technically accurate model cannot be used in practice. Problem framing is therefore the bridge between organizational purpose and analytical work.

Problem framing translates a broad organizational concern into a precise, evidence-based decision problem. It identifies who needs to decide, what action may follow, which population and time period are relevant, what information is available, and how success will be judged. The data science lifecycle then provides a disciplined way to move from that framing through data, modelling, evaluation, deployment, and learning.

This chapter uses CRISP-DM as its central framework, while comparing it with related lifecycles such as the KDD process, SEMMA, and the Microsoft Team Data Science Process. These frameworks are not rigid linear recipes. They are iterative management and technical models that make assumptions, dependencies, and decision points visible.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish an organizational problem from an analytical question;
• identify stakeholders, decisions, constraints, assumptions, and risks;
• formulate hypotheses and operational definitions;
• explain and apply the CRISP-DM lifecycle;
• compare CRISP-DM with related data science processes;
• select appropriate outcome, model, and operational metrics;
• define deliverables and success criteria;
• recognize ethical, governance, and deployment issues; and
• produce a concise project charter for a data science initiative.

From Organizational Concern to Analytical Problem

An organizational concern is usually expressed in broad language: “customer churn is increasing,” “fraud losses are too high,” or “the hospital needs to reduce waiting times.” Such statements are important but not yet suitable for analysis. They do not specify the decision, the population, the time frame, the relevant outcome, or the action available to the organization.

A useful transformation has four levels:

1. Organizational objective: the value the organization seeks, such as reducing avoidable costs.
2. Decision problem: the choice that must be improved, such as which customers should receive retention support.
3. Analytical question: a question answerable with data, such as whether a customer is likely to cancel within 30 days.
4. Data product or deliverable: the usable output, such as a ranked daily list with recommended actions.

The analytical question must not be confused with the business objective. Predicting churn is not itself valuable; it is valuable only if the organization can intervene effectively and the intervention improves outcomes.

A practical problem statement can follow this pattern:

For [decision-maker], determine whether/how [analytical task] for [population] during [time period], so that [decision] can improve [organizational outcome], subject to [constraints].

For example: “For the customer-care team, estimate which active mobile subscribers are at high risk of cancelling within the next 30 days, so that limited retention offers can be targeted, subject to privacy, budget, and contact-capacity constraints.”

Stakeholders and decision rights

Stakeholders include people who sponsor the work, supply domain knowledge, own data, use the output, monitor performance, or experience its consequences. A stakeholder map should identify interests and decision rights rather than merely list job titles.

A project should have one accountable owner for the decision, even when many people contribute. Without clear decision rights, projects often produce technically interesting outputs that nobody is authorized or motivated to use.

Hypotheses, Assumptions, and Constraints

A hypothesis is a proposition that can be examined with evidence. It should specify a population, exposure or predictor, outcome, and time relation where relevant. For example: “Among active subscribers, customers with repeated service failures in the previous 60 days have a higher probability of cancellation in the following 30 days.” This is more useful than “service quality affects churn,” because it defines a testable time window and population.

Hypotheses may be:

• Descriptive: what has happened or how groups differ;
• Predictive: what is likely to happen; or
• Causal: whether changing an intervention would change an outcome.

A predictive model can identify correlation without proving that an intervention causes improvement. Confusing prediction with causation is a major source of poor recommendations.

Assumptions are conditions accepted for planning or analysis. Examples include the assumption that customer identifiers can be joined reliably, that historical labels represent future events, or that a proposed intervention is available to users. Each important assumption should have an owner, a way to test it, and a consequence if it fails.

Constraints may involve:

• data access, quality, granularity, or latency;
• budget, staffing, and computing capacity;
• legal requirements and retention rules;
• fairness, privacy, and security;
• response-time or availability requirements;
• intervention capacity; and
• organizational readiness.

A constraint is not merely a technical inconvenience. For example, if staff can contact only 500 customers daily, a model ranking 20,000 customers does not constitute a complete solution.

Defining Success and Metrics

Success must be defined at several levels because model quality alone does not establish organizational value.

For binary classification, if TP denotes true positives, TN true negatives, FP false positives, and FN false negatives, then:

Precision is the proportion of predicted positives that are actually positive. It is applicable when the positive prediction is operationally meaningful. Recall is:

Recall is the proportion of actual positives detected. Both measures require a defined classification threshold and a labelled outcome. They should not be interpreted without considering class prevalence, costs, and consequences.

Accuracy is:

Accuracy can be misleading when classes are imbalanced. If only 2% of transactions are fraudulent, a model that always predicts “not fraud” has 98% accuracy but detects no fraud. Metric selection must follow the decision context.

A business case may use expected net value. If an intervention is applied to a set of cases, a simplified estimate is:

This is a decision-support expression rather than a universal accounting identity. Its terms must be estimated in consistent monetary units, and the analysis should distinguish model association from incremental causal impact. A randomized trial or carefully designed quasi-experiment is needed to estimate whether the intervention itself changes outcomes.

Success criteria should be measurable and time-bound. For example: “Within three months of pilot deployment, the retention team will use the ranked list on at least 90% of working days; among eligible contacted customers, cancellation will fall by at least 5% relative to a pre-specified control group; no monitored protected group will exceed the approved error-rate disparity threshold.”

CRISP-DM: The Core Lifecycle

CRISP-DM—Cross-Industry Standard Process for Data Mining—organizes work into six phases:

1. business understanding;
2. data understanding;
3. data preparation;
4. modelling;
5. evaluation; and
6. deployment.

The phases are iterative. Evaluation may reveal that the original business question is inappropriate; data preparation may expose a measurement problem; deployment may reveal that users cannot act on the output.

Figure: CRISP-DM is a cycle of feedback rather than a one-way sequence.

Business understanding

This phase establishes objectives, stakeholders, scope, constraints, risks, and criteria for success. Outputs include a project charter, stakeholder map, initial hypothesis, decision description, and feasibility assessment. A strong charter states what is out of scope; uncontrolled scope expansion is a common cause of delay.

Data understanding

The team identifies available sources, examines structure and quality, and checks whether the data can represent the target population. Activities include profiling missingness, duplicates, unusual values, label definitions, time coverage, sampling bias, and join quality. Data dictionaries should record field meaning, unit, owner, update frequency, and permitted use.

Data preparation

Preparation includes selection, integration, cleaning, transformation, feature construction, and dataset splitting. It must be reproducible. A critical safeguard is preventing leakage: information used to create a feature must have been available at the time the prediction would actually be made. For example, a “cancellation reason” recorded after a customer cancels cannot be used to predict that cancellation.

Training, validation, and test data should respect the data-generating process. For time-dependent problems, a chronological split is usually more realistic than random splitting. The test set should remain untouched until final evaluation.

Modelling

The team selects a baseline and one or more appropriate models. A baseline may be a simple rule, historical average, or existing process. It establishes whether complexity produces meaningful improvement. Model choice should consider interpretability, calibration, latency, maintenance, fairness, and cost—not only predictive performance.

Evaluation

Evaluation asks two separate questions: “Is the model technically credible?” and “Would using it improve the decision?” Evaluation should examine performance on an appropriate holdout set, robustness across relevant groups and periods, error costs, calibration, and operational feasibility. If the model performs well but no effective action follows, the project should be reframed rather than declared successful.

Deployment

Deployment may mean a dashboard, report, API, workflow integration, batch score, or human decision aid. It includes documentation, access control, monitoring, feedback mechanisms, incident procedures, and a plan for retraining or retirement. A deployment is incomplete if users cannot understand the output or if nobody owns the ongoing service.

Related Lifecycles

The KDD process emphasizes selecting data, preprocessing, transforming data, applying data-mining methods, and interpreting discovered patterns. SEMMA—Sample, Explore, Modify, Model, Assess—was associated with SAS and focuses strongly on analytical workflow. The Microsoft Team Data Science Process adds project roles, work items, source control, infrastructure, and iterative delivery. Agile and DataOps practices contribute short feedback cycles, versioning, automated testing, and reliable data pipelines.

These frameworks overlap but differ in emphasis. CRISP-DM is valuable for cross-functional framing and communication; KDD emphasizes knowledge discovery; SEMMA emphasizes modelling workflow; and Team Data Science Process emphasizes team execution. In practice, an organization may use CRISP-DM for governance and stage gates, Agile for planning, and DataOps or MLOps for deployment and monitoring.

Worked Example: Framing a Churn Project

A subscription company reports rising cancellations. The initial request is “build a churn model.” A disciplined framing process asks:

• What decision will change? The retention team will select customers for outreach.
• Who is eligible? Active monthly subscribers with at least 30 days of history.
• What is the outcome? Cancellation within the 30 days after the scoring date.
• What is the prediction time? Each customer is scored daily at midnight.
• What action is available? A phone call or approved offer, limited to 500 contacts per day.
• What is the baseline? Current rule-based selection.
• What constraints apply? Consent, privacy, contact frequency, and fairness review.

The hypothesis is that recent service failures, reduced usage, unresolved complaints, and payment friction are associated with near-term cancellation. These variables are candidates, not automatically valid causes. Post-cancellation information must be excluded.

Suppose a pilot produces TP = 360, FP = 240, FN = 140, and TN = 9,260 on a labelled evaluation set. Then:

Thus, 60% of flagged customers cancelled in the evaluation period. Recall is:

Thus, the model identified 72% of observed cancellations. These figures do not prove that outreach will reduce cancellations. The company should conduct a controlled pilot in which eligible customers are randomly assigned, where ethically and operationally permissible, to outreach or control. The primary business metric should be the difference in cancellation rates between groups, alongside cost, customer experience, and subgroup monitoring.

Deliverables include a signed charter, data dictionary, reproducible dataset pipeline, model card, evaluation report, user workflow, pilot plan, monitoring dashboard, and decision log. The model should be rejected or redesigned if the intervention is ineffective, the data are not lawfully usable, or performance degrades materially for a protected group.

Ethics, Governance, and Responsible Use

Problem framing determines whether a potentially harmful system is built. Teams should ask whether the objective is legitimate, whether affected people can reasonably expect the data use, and whether the decision can be challenged. Data minimization means collecting and using only what is necessary. Purpose limitation means not silently repurposing data for incompatible objectives.

Important controls include:

• lawful and documented data access;
• privacy-preserving identifiers and least-privilege access;
• retention and deletion rules;
• provenance and lineage records;
• bias and subgroup performance assessment;
• explainable communication appropriate to the audience;
• human review for high-impact decisions; and
• monitoring for drift, misuse, and unexpected harm.

Fairness cannot be reduced to one universal metric. Different fairness criteria may conflict, particularly when group base rates differ. The appropriate analysis depends on law, context, harms, and the decision mechanism. A technically accurate model may still be unacceptable if it reinforces historical exclusion or denies meaningful recourse.

Generative AI can assist with documentation, code suggestions, data-dictionary drafts, and stakeholder question generation, but it can hallucinate definitions, expose confidential information, or reproduce bias. Human verification, secure environments, version control, and approval procedures remain necessary.

Challenges and Best Practices

Common challenges include vague objectives, proxy targets, inaccessible data, leakage, changing definitions, class imbalance, overreliance on a single metric, and deployment without ownership. Further difficulties arise when organizational incentives reward a model launch rather than improved outcomes.

Best practices are to:

1. begin with the decision and available intervention;
2. write a measurable problem statement and explicit exclusions;
3. establish a baseline before selecting complex models;
4. document assumptions, definitions, provenance, and limitations;
5. design evaluation before modelling;
6. separate exploratory analysis from confirmatory claims;
7. use time-aware validation when the future is the target;
8. include domain, engineering, governance, and user perspectives;
9. pilot with monitoring and a rollback plan; and
10. treat deployment as an ongoing service, not a final presentation.

Chapter Summary

Problem framing converts organizational concerns into decisions, analytical questions, and usable deliverables. Effective framing identifies stakeholders, hypotheses, assumptions, constraints, populations, time windows, interventions, and success criteria. CRISP-DM provides six iterative phases: business understanding, data understanding, data preparation, modelling, evaluation, and deployment. Related processes emphasize discovery, modelling, team execution, or operational delivery, and can be combined appropriately.

Metrics must be selected for the decision: precision, recall, accuracy, business value, operational reliability, and responsible-use measures answer different questions. Prediction does not establish causation, and model performance does not guarantee organizational benefit. Ethical governance, privacy, fairness, documentation, monitoring, and human accountability must be designed from the beginning.

Key Terms

Problem framing: Converting an organizational concern into a precise decision and analytical task.

Stakeholder: A person or group that influences, uses, owns, or is affected by a project.

Decision problem: The organizational choice that analysis is intended to improve.

Analytical question: A data-answerable question derived from a decision problem.

Hypothesis: A testable proposition about relationships or outcomes.

Assumption: A condition accepted for planning or analysis.

Constraint: A limitation on data, resources, legality, timing, or action.

Success criterion: A measurable condition indicating acceptable achievement.

CRISP-DM: An iterative lifecycle for data-mining and data-science projects.

Data leakage: Use of information unavailable at the time a real prediction is made.

Baseline: A simple reference method against which a model is compared.

Precision: The proportion of predicted positives that are actual positives.

Recall: The proportion of actual positives identified by a system.

Calibration: Agreement between predicted probabilities and observed frequencies.

Deployment: Making an analytical output available for operational use.

Drift: Change in data patterns, relationships, or outcomes over time.

Data lineage: A record of data origin, transformation, movement, and use.

Model card: Documentation describing a model’s purpose, performance, limitations, and risks.

Review Questions

Short Answer Questions

1. Why is “build a churn model” an incomplete problem statement?
2. Distinguish a predictive hypothesis from a causal hypothesis.
3. Name the six CRISP-DM phases.
4. What is data leakage, and why is it harmful?
5. Why can accuracy be misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain how an organizational objective can be transformed into a decision problem, analytical question, and data product.
2. Compare CRISP-DM, KDD, SEMMA, and an operational process such as MLOps or DataOps.
3. Discuss how stakeholders, assumptions, constraints, and decision rights affect project feasibility.
4. Explain why model evaluation should include technical, business, operational, and responsible-use criteria.
5. Describe the governance controls required before deploying a model that influences customer treatment.

Analytical / Application Questions

1. A hospital asks for “an AI system to reduce waiting times.” Frame the problem by specifying the decision, population, outcome, time window, intervention, constraints, and success metrics.
2. Design an evaluation plan for a fraud model when false negatives cost more than false positives and only 1% of transactions are fraudulent.
3. A model has excellent test performance but staff do not use its recommendations. Diagnose the lifecycle failures and propose a revised deployment and evaluation plan.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an analytical question from an organizational objective?

A. An analytical question specifies what can be answered with available evidence

B. An analytical question is always more important than the organizational objective

C. An analytical question eliminates the need for stakeholder involvement

D. An analytical question describes only the project budget

Correct Answer: A. An analytical question specifies what can be answered with available evidence

Question 2

Which CRISP-DM phase primarily establishes objectives, stakeholders, constraints, and business success criteria?

A. Data preparation

B. Business understanding

C. Modelling

D. Deployment

Correct Answer: B. Business understanding

Question 3

A feature records a customer’s cancellation reason after the customer has already cancelled. Using it to predict cancellation is an example of:

A. Underfitting

B. Class balancing

C. Data leakage

D. Calibration

Correct Answer: C. Data leakage

Question 4

If TP = 80 and FP = 20, what is precision?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 5

Which measure is the proportion of actual positive cases detected?

A. Recall

B. Precision

C. Accuracy

D. Specificity

Correct Answer: A. Recall

Question 6

Why is a baseline useful in a modelling project?

A. It guarantees the final model will be unbiased

B. It provides a reference for judging whether added complexity helps

C. It replaces the need for evaluation data

D. It prevents all forms of data leakage

Correct Answer: B. It provides a reference for judging whether added complexity helps

Question 7

Which statement about prediction and causation is correct?

A. A predictive association automatically proves an intervention will work

B. A highly accurate model always identifies causal factors

C. Prediction can be useful without establishing that changing a predictor causes the outcome

D. Causal analysis never requires an outcome measure

Correct Answer: C. Prediction can be useful without establishing that changing a predictor causes the outcome

Question 8

Which is the most appropriate deployment deliverable for a model used by frontline staff?

A. Only the source code

B. A workflow, documentation, access controls, and monitoring plan

C. A list of algorithm names

D. An unvalidated training dataset

Correct Answer: B. A workflow, documentation, access controls, and monitoring plan

Question 9

A project team should normally use chronological validation rather than random splitting when:

A. The prediction concerns future events and temporal patterns matter

B. There are no timestamps anywhere in the data

C. The classes are perfectly balanced

D. The model is a simple baseline

Correct Answer: A. The prediction concerns future events and temporal patterns matter

Question 10

Which metric category measures whether a deployed service responds quickly and remains available?

A. Business value

B. Model discrimination

C. Operational performance

D. Causal effect

Correct Answer: C. Operational performance