Analytical Problem Framing and Workflow

Introduction

Data analysis rarely fails because an analyst cannot calculate an average or create a chart. It more often fails because the organisational problem was poorly framed. A vague request such as “analyse customer behaviour” does not identify the decision to be made, the population affected, the time period, the success criteria, or the consequences of being wrong. Problem framing converts organisational concerns into questions that can be answered with evidence.

This chapter presents a practical foundation for translating business and organisational problems into measurable analytical tasks. It introduces stakeholders, objectives, hypotheses, metrics, key performance indicators, constraints, assumptions, deliverables and decision consequences. It also explains CRISP-DM, a widely used workflow for data-mining and analytics projects, and shows how analytical work should proceed iteratively rather than as a simple sequence of technical tasks.

A well-framed analytical project connects five elements:

1. an organisational decision or problem;
2. measurable analytical questions;
3. appropriate data and methods;
4. interpretable findings;
5. an action whose consequences can be evaluated.

The analyst’s role is therefore both technical and interpretive. Technical competence is necessary, but so are communication, domain understanding, ethical judgement and the ability to recognise when a metric does not represent the underlying objective.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish an organisational problem from an analytical problem;
• identify stakeholders, decision owners and affected groups;
• translate objectives into measurable questions and hypotheses;
• distinguish metrics, measures, indicators and KPIs;
• define populations, units of analysis, time horizons and outcomes;
• apply the CRISP-DM workflow to a practical project;
• document assumptions, constraints, risks and deliverables;
• assess how analytical errors affect decisions;
• recognise ethical, privacy and governance requirements;
• evaluate an analytical brief for clarity, feasibility and usefulness.

1. From Organisational Concern to Analytical Problem

An organisational concern is a situation perceived as undesirable or requiring improvement. Examples include rising customer complaints, delayed deliveries, high employee turnover or declining use of a digital service. An analytical problem is a carefully specified question about that concern that can be investigated using data.

Consider the statement: “Sales are falling.” It is incomplete. Sales may be falling in one region, for one product category, or only during a particular season. The decline may be caused by price changes, stock-outs, competitor activity, recording errors or a change in the customer base. Before selecting a model, the analyst should ask:

• What exactly is declining: revenue, units, margin or market share?
• Compared with which baseline?
• Over what period?
• For which customers, products or locations?
• Who must decide what to do?
• What action is possible if a cause is identified?

A useful problem statement has the form:

The organisation needs to decide [decision] because [observed concern]. Analysis will examine [population, outcome and drivers] during [time period] so that [decision owner] can [intended action], subject to [important constraints].

For example:

The operations team needs to decide whether to change delivery staffing because the proportion of late deliveries has increased. Analysis will examine orders delivered in the last twelve months by region, delivery partner and service type so that the operations manager can allocate capacity, subject to budget and data-privacy constraints.

This statement is not yet a complete research design, but it establishes direction and prevents analysis from becoming an unfocused search for interesting patterns.

1.1 Descriptive, diagnostic, predictive and prescriptive questions

Analytical questions differ in purpose.

These categories are related but not interchangeable. A predictive model may identify customers at high risk without establishing that a proposed intervention will prevent cancellation. Association is not proof of causation. Similarly, a dashboard describing a problem does not automatically prescribe a solution.

1.2 Unit of analysis and population

The unit of analysis is the entity represented by one analytical record. It may be a transaction, customer, employee, day, machine reading, hospital visit or delivery. Confusing units can produce invalid conclusions. For example, if one customer makes many purchases, a transaction-level average may give frequent purchasers greater influence than occasional purchasers.

The population is the full set of entities to which the conclusion is intended to apply. A sample is the subset actually observed. Analysts should specify inclusion and exclusion criteria, observation period and geographical or organisational scope. They should also ask whether the available data represent the target population. A model trained on urban customers may not generalise to rural customers.

1.3 Time and comparison

Most organisational questions require a comparison. Common baselines include:

• previous period, such as last month;
• same period in a previous year;
• a budget or target;
• a control or comparison group;
• an industry benchmark;
• a pre-intervention period.

The comparison must be meaningful. Comparing December with November may confuse seasonal effects with genuine change. Comparing two departments with different case mixes may also be misleading. Time boundaries, lag effects and seasonality should be documented before analysis begins.

2. Stakeholders, Decisions and Consequences

Stakeholders are people or groups who influence, use, provide, govern or are affected by an analytical project. A stakeholder map commonly includes:

• Decision owner: has authority to act on the result.
• Subject-matter expert: understands the operational context.
• Data owner: controls access, definitions and data quality.
• Analyst or data scientist: designs and performs the analysis.
• Technical team: provides systems, pipelines or deployment support.
• Governance, legal or privacy specialists: assess compliance and risk.
• Affected individuals or communities: experience consequences, whether or not they commission the project.

The most senior stakeholder is not always the decision owner. A director may sponsor an investigation, while an operations manager decides staffing. Identifying the actual decision owner clarifies what form the deliverable should take and when it is needed.

A stakeholder interview should establish:

1. What decision is currently difficult?
2. What evidence is already used?
3. What action could follow from each possible result?
4. What would count as success?
5. What constraints cannot be violated?
6. Who may be harmed by an incorrect or unfair result?

Decision consequences should be explicit. A false positive may cause unnecessary investigation, denied access or wasted resources. A false negative may allow fraud, equipment failure or unsafe conditions to continue. The acceptable error depends on the context, not merely on model accuracy.

3. Questions, Hypotheses and Operational Definitions

3.1 From objective to question

A broad objective such as “improve retention” can be decomposed into measurable questions:

• What is the current retention rate?
• How does retention vary by customer segment?
• Which observable behaviours precede cancellation?
• Can cancellation risk be estimated early enough to support intervention?
• Does a particular retention intervention improve outcomes?

Each question should identify an outcome, population, time period and, where relevant, comparison.

3.2 Hypotheses

A hypothesis is a testable proposition. It should be sufficiently specific that evidence could support or contradict it. For example:

Customers who experience more than two unresolved support cases in a quarter have a higher probability of cancellation in the following quarter than customers with no unresolved cases.

A hypothesis is not a conclusion. It guides measurement and analysis. Analysts should avoid changing the hypothesis after seeing results without documenting the change, because repeated searching can produce apparently strong patterns by chance.

For a quantitative comparison, an analyst may define a null hypothesis and an alternative hypothesis. For example, H₀: p₁ = p₂ states that two population proportions are equal, while H₁: p₁ ≠ p₂ states that they differ. Here, p₁ and p₂ are the relevant population proportions. The appropriate test depends on study design, sampling, independence, measurement scale and distributional assumptions. Statistical significance alone does not establish practical importance or causality.

3.3 Operational definitions

An operational definition specifies how an abstract concept will be measured. “Customer loyalty” might be represented by repeat purchase within 90 days, subscription renewal, or share of wallet. Each definition has different implications.

A good operational definition states:

• the variable name;
• its meaning;
• calculation rule;
• unit and scale;
• source system;
• time window;
• missing-value treatment;
• inclusion and exclusion rules.

For example, a monthly cancellation rate can be defined as:

The numerator and denominator must refer to compatible populations. If customers who joined during the month are included in the numerator but not the denominator, the rate is distorted. The result is dimensionless and is often reported as a percentage by multiplying the ratio by 100.

4. Measures, Metrics and KPIs

A measure is an observed value, such as delivery time in minutes. A metric is a defined quantitative measure used to assess performance or behaviour, such as median delivery time. An indicator signals a condition or trend, and may be quantitative or qualitative. A key performance indicator is a metric formally linked to an important organisational objective and used to monitor progress or support decisions.

Not every metric is a KPI. A dashboard may contain dozens of diagnostic measures, while only a few KPIs represent strategic priorities. A KPI should have a clear owner, target or reference point, reporting frequency, data definition and action associated with movement.

4.1 Useful metric categories

• Outcome metrics: measure the result of interest, such as profit or retention.
• Output metrics: measure activity produced, such as cases processed.
• Input metrics: measure resources used, such as staff hours.
• Leading indicators: may signal future outcomes, such as unresolved complaints.
• Lagging indicators: record outcomes after they occur, such as annual churn.
• Quality metrics: assess accuracy, completeness or reliability.
• Equity metrics: compare performance or impact across relevant groups.

A metric can be mathematically correct yet operationally poor. If staff are rewarded only for the number of cases closed, they may close easy cases and neglect complex ones. This is a form of metric gaming or Goodhart’s law: when a measure becomes a target, it may cease to be a good measure.

4.2 Metric specification

A metric specification should include:

The distinction between a target and a forecast is important. A target is a desired level; a forecast is an estimate of a future level based on evidence and assumptions.

5. Core Quantitative Measures

Only use a formula when its definition and assumptions are understood. For a set of n observations x₁, x₂, …, x_n, the arithmetic mean is:

The mean is appropriate for quantitative data when the observations are meaningfully averaged and extreme values do not make it misleading. The median, the middle ordered observation, is often more representative for skewed measures such as income or response time.

A proportion is:

The denominator defines the population at risk. A rate may additionally include a time unit, such as incidents per 1,000 person-months. Rates from different populations should not be compared without considering exposure and case mix.

For a binary classifier, common performance measures include:

Precision answers: among cases predicted positive, how many were actually positive? Recall answers: among actual positive cases, how many were detected? These measures apply when positive and negative classes are defined and a classification threshold has been selected. They can vary with prevalence and threshold; therefore, reporting accuracy alone is often inadequate for imbalanced problems.

Worked example: defining a service KPI

A service centre received 500 eligible cases in one week. Of these, 460 met the response-time standard. The on-time response rate is:

If 20 cases were excluded because they were not eligible under the documented policy, the denominator must be 500 only if 500 already represents eligible cases. If 520 total cases were received and 20 were ineligible, the same result is obtained from 460 ÷ 500. The analyst must not silently use 460 ÷ 520, because that answers a different question.

6. The CRISP-DM Workflow

CRISP-DM means Cross-Industry Standard Process for Data Mining. It is a practical framework with six interacting phases:

1. business understanding;
2. data understanding;
3. data preparation;
4. modelling;
5. evaluation;
6. deployment.

The phases are iterative. Findings from data understanding may change the business question; evaluation may reveal that a model does not support the decision; deployment may expose data or workflow problems.

Figure: CRISP-DM is an iterative workflow rather than a one-way pipeline.

6.1 Business understanding

Define the organisational objective, decision, success criteria, resources, risks and project scope. Produce a problem statement, stakeholder map, analytical questions and initial plan. A technical success criterion, such as low prediction error, must be connected to a business success criterion, such as fewer missed appointments without unacceptable additional cost.

6.2 Data understanding

Identify available sources, inspect structure and quality, and assess whether the data represent the intended population. Typical activities include examining distributions, missingness, duplicates, inconsistent categories, outliers, time coverage and possible leakage. Data leakage occurs when information unavailable at decision time is used to predict the outcome, producing unrealistically strong performance.

6.3 Data preparation

Select records and variables, join sources, correct formats, handle missing values, create features and document transformations. Preparation should be reproducible. Imputation, scaling and feature selection should be learned from training data where appropriate, rather than from the complete dataset, to avoid contaminating evaluation.

6.4 Modelling

Select a method suitable for the question, data type and decision. A simple baseline is valuable: it provides a reference against which more complex models can be judged. Modelling choices should consider interpretability, operational cost, fairness, robustness and maintainability, not only predictive performance.

6.5 Evaluation

Evaluate both analytical validity and decision usefulness. Check performance on data not used to fit the model, examine errors across important groups, test sensitivity to assumptions and compare with the baseline. Evaluation should answer: Is the result credible? Does it answer the original question? Can the organisation act on it?

6.6 Deployment

Deployment may mean a dashboard, report, scheduled data product, decision-support tool or integrated predictive service. Define ownership, refresh frequency, monitoring, escalation and retirement conditions. A deployed model can deteriorate when customer behaviour, policies, data collection or external conditions change. This is often called data or concept drift.

7. Constraints, Assumptions, Deliverables and Risks

Constraints limit what can be done. They may concern time, budget, computing capacity, data access, privacy, regulation, staffing, latency or interpretability. Constraints should be recorded rather than discovered at the end.

Assumptions are propositions treated as reasonable for the analysis. Examples include assuming timestamps use one time zone, records are independent, a missing value means “not recorded” rather than “zero,” or a historical relationship will remain sufficiently stable. Assumptions should be tested where possible and accompanied by consequences if they fail.

Typical deliverables include:

• analytical brief and stakeholder map;
• data inventory and data-quality report;
• reproducible preparation workflow;
• exploratory analysis and visualisations;
• model or statistical analysis, if justified;
• evaluation report;
• decision dashboard or recommendation;
• technical documentation and handover;
• monitoring and review plan.

A risk register can record the risk, likelihood, impact, mitigation and owner. For example, a missing outcome label may threaten model evaluation; mitigation may involve a manual audit sample or a revised measurement process.

8. Case Study: Reducing Missed Outpatient Appointments

The following is an illustrative case, not a report of a specific real organisation.

Background

A community health service reports that missed appointments are increasing. Missed appointments reduce service capacity and may delay care. The service manager asks an analytics team to “build a model to stop no-shows.”

Problem framing

The decision is not simply whether a model can predict non-attendance. The service must decide which patients, if any, should receive reminders or additional support, while protecting privacy and avoiding unfair treatment. The analytical objective becomes:

Estimate the probability of non-attendance for appointments at least 48 hours before the appointment, and assess whether targeted reminders can reduce missed appointments without reducing access or disproportionately burdening a protected group.

The unit of analysis is an appointment, not a patient, because one patient may have several appointments. The outcome is whether the appointment was missed. The prediction time is fixed at 48 hours before the appointment to prevent leakage from later events.

Stakeholders and constraints

Stakeholders include the service manager, appointment schedulers, clinicians, patients, data-protection personnel and the analytics team. Constraints include limited reminder capacity, sensitive health information, incomplete contact details and the requirement that a risk score must not be used to deny care.

Metrics

Primary outcome: missed-appointment proportion among eligible appointments.

Operational metrics: reminder delivery rate, staff workload and cost per reminder.

Equity metrics: outcome and intervention rates across relevant demographic or access groups, interpreted cautiously and with appropriate governance.

Technical metrics: recall at the available reminder capacity, precision, calibration and performance on a temporally later test set.

A simple baseline might send reminders to all eligible appointments. A model should only be adopted if it improves the decision relative to this baseline, not merely if it produces a high accuracy score.

Analysis and approach

The team first audits appointment records, contact information, cancellation codes and timestamps. It discovers that some appointments have a reminder status recorded after the appointment time. That field is excluded from prediction because it would leak future information. The team also finds that missing contact information is concentrated in certain areas; this is treated as an operational issue requiring service improvement, not merely as a variable for prediction.

A transparent baseline model and a more flexible model are compared using a later time period. Threshold selection reflects the number of reminders staff can send. The team examines false negatives and false positives, calibration, subgroup performance and whether the model is stable during seasonal periods.

Outcome

The analysis indicates that targeted reminders could prioritise appointments with higher estimated risk, but the evidence does not prove that risk prediction itself causes fewer missed appointments. A small pilot is therefore designed. Eligible appointments are randomly assigned, where ethically and operationally appropriate, to standard practice or targeted reminder intervention. The pilot measures attendance, successful contact, workload and subgroup effects.

Learning points

1. The real problem concerned intervention capacity, not prediction alone.
2. The prediction time had to be specified to prevent leakage.
3. A baseline and pilot were needed to assess practical value.
4. Patient impact and privacy were central design requirements.
5. Missing contact information revealed a service problem that a model could not solve.

9. Ethics, Governance and Responsible Use

Analytical work can reproduce or amplify existing inequalities. Important governance questions include:

• Is there a legitimate purpose and lawful basis for using the data?
• Were individuals informed where required?
• Is data collection proportionate to the purpose?
• Are sensitive attributes being used appropriately for auditing or decision-making?
• Could a proxy variable reproduce discrimination?
• Can affected people understand or challenge a consequential decision?
• Are retention, access control and deletion rules defined?

Privacy protection includes data minimisation, access restriction, secure storage, pseudonymisation where appropriate and careful handling of small groups. Pseudonymisation reduces direct identification but does not make data anonymous in every context.

Fairness is not a single numerical property. Different fairness measures can conflict when group base rates differ. The appropriate evaluation depends on the decision, legal context, harms and available remedies. Analysts should involve domain and governance specialists rather than treating fairness as a final technical adjustment.

Explainability should match the audience and consequence. A manager may need the main drivers and limitations; a technical team may need feature definitions and validation details; an affected person may need a clear explanation of the decision and a route for review. Human oversight should be meaningful rather than a nominal approval step.

10. Contemporary Practice and Best Practices

Modern analytics increasingly combines structured databases, cloud platforms, automated pipelines, machine learning and generative AI assistants. These technologies can accelerate coding, documentation and exploration, but they do not remove the need for problem framing. AI-generated code or text must be checked for correctness, privacy risks, hidden assumptions, licensing concerns and unsupported claims.

Best practices include:

• begin with a decision and action, not a preferred algorithm;
• write a one-page analytical brief before extensive modelling;
• define the unit of analysis, outcome, prediction time and denominator;
• maintain a data dictionary and decision log;
• establish a baseline before adopting complexity;
• separate exploratory findings from confirmatory claims;
• evaluate errors and impacts across relevant groups;
• document assumptions, exclusions and limitations;
• design deployment, monitoring and ownership early;
• communicate uncertainty and avoid overstating causality;
• obtain appropriate privacy, legal and domain review.

Chapter Summary

Analytical problem framing translates an organisational concern into a decision-oriented question that can be answered with evidence. Effective framing identifies the decision owner, affected stakeholders, population, unit of analysis, outcome, time horizon, comparison and possible actions. Descriptive, diagnostic, predictive and prescriptive questions serve different purposes and should not be conflated.

Hypotheses make proposed relationships testable, while operational definitions make abstract concepts measurable. Metrics quantify performance; KPIs connect selected metrics to important objectives, owners, targets and actions. Denominators, baselines and time windows must be defined carefully because apparently simple rates can be misleading.

CRISP-DM provides an iterative structure consisting of business understanding, data understanding, data preparation, modelling, evaluation and deployment. A credible project also documents constraints, assumptions, deliverables, risks and decision consequences. Technical performance is only one component of success. Privacy, fairness, interpretability, security, monitoring and the possibility of harm must be considered throughout the workflow.

Key Terms

Analytical problem: A specified question that can be investigated using data to support a decision.

Assumption: A proposition treated as reasonable for the purposes of an analysis.

Baseline: A simple reference method or performance level used for comparison.

CRISP-DM: An iterative framework for data-mining projects comprising six connected phases.

Data leakage: Use of information unavailable at decision time, causing unrealistically strong evaluation.

Decision owner: The person or group with authority to act on an analytical result.

Deliverable: A planned output of an analytical project, such as a report, model or dashboard.

Hypothesis: A specific, testable proposition about a relationship or difference.

Indicator: A measure that signals a condition, trend or level of performance.

Key performance indicator: A metric formally linked to an important organisational objective.

Metric: A defined quantitative measure used to assess performance or behaviour.

Operational definition: A precise rule describing how a concept is measured.

Population: The complete set of entities to which a conclusion is intended to apply.

Precision: Among predicted positive cases, the proportion that is actually positive.

Recall: Among actual positive cases, the proportion correctly identified.

Stakeholder: A person or group that influences, uses, governs or is affected by analysis.

Unit of analysis: The entity represented by one record in an analysis.

Validity: The extent to which an analysis measures or answers what it claims to measure or answer.

Review Questions

Short Answer Questions

1. What is the difference between an organisational problem and an analytical problem?
2. Why must the unit of analysis be specified before analysing data?
3. Distinguish a metric from a KPI.
4. What is data leakage, and why does it damage model evaluation?
5. Name the six phases of CRISP-DM.

Long Answer Questions

1. Explain how a vague objective such as “improve customer retention” can be converted into measurable analytical questions.
2. Discuss the roles of decision owners, data owners, subject-matter experts and affected individuals in an analytical project.
3. Explain why a technically accurate model may still fail to deliver organisational value.
4. Compare descriptive, diagnostic, predictive and prescriptive analytics, giving an example of each.
5. Describe how privacy, fairness and human oversight should be incorporated into the analytical workflow.

Analytical / Application Questions

1. A delivery company reports that late deliveries have increased. Write a complete analytical problem statement, identify the unit of analysis and propose three metrics with definitions.
2. A classifier identifies customers likely to cancel a subscription. Explain which information is needed to choose between precision and recall, and describe one business consequence of each type of error.
3. Design a CRISP-DM plan for analysing employee absenteeism. Include stakeholders, hypotheses, data risks, deliverables, deployment considerations and at least one ethical concern.

Multiple Choice Questions with Answers

Question 1

Which statement best describes an analytical problem?

A. A general concern that performance is unsatisfactory

B. A specified question that can be investigated with data to support a decision

C. A preferred machine-learning algorithm

D. A dashboard containing many charts

Correct Answer: B. A specified question that can be investigated with data to support a decision

Question 2

What is the main purpose of identifying the unit of analysis?

A. To select the colour scheme for a dashboard

B. To determine which programming language must be used

C. To define what entity each analytical record represents

D. To guarantee that data contain no missing values

Correct Answer: C. To define what entity each analytical record represents

Question 3

Which question is primarily predictive?

A. How many orders were late last month?

B. Why were orders late in the northern region?

C. Which orders are likely to be late tomorrow?

D. Which staffing policy should management adopt?

Correct Answer: C. Which orders are likely to be late tomorrow?

Question 4

A service centre handles 460 on-time cases out of 500 eligible cases. What is the on-time rate?

A. 46%

B. 82%

C. 92%

D. 96%

Correct Answer: C. 92%

Question 5

Which statement about a KPI is correct?

A. Every recorded variable is automatically a KPI

B. A KPI is linked to an important objective and supports performance monitoring or decisions

C. A KPI can never have a target

D. A KPI must always be a predictive model

Correct Answer: B. A KPI is linked to an important objective and supports performance monitoring or decisions

Question 6

Which CRISP-DM phase focuses on inspecting data quality, structure and coverage?

A. Business understanding

B. Data understanding

C. Modelling

D. Deployment

Correct Answer: B. Data understanding

Question 7

What is data leakage?

A. Loss of data caused by a hardware failure

B. Use of information that would not be available when the prediction is made

C. A method for encrypting sensitive data

D. A process for removing duplicate records

Correct Answer: B. Use of information that would not be available when the prediction is made

Question 8

If false negatives would allow dangerous equipment failures to remain undetected, which measure may be especially important?

A. Recall

B. File size

C. Number of dashboard pages

D. Mean of identifier values

Correct Answer: A. Recall

Question 9

Why is a baseline useful in modelling?

A. It guarantees that a complex model is fair

B. It provides a reference against which additional complexity can be judged

C. It eliminates the need for evaluation data

D. It proves that a relationship is causal

Correct Answer: B. It provides a reference against which additional complexity can be judged

Question 10

Which is an appropriate governance practice for a high-impact analytical system?

A. Hide limitations from decision makers

B. Use every available variable regardless of purpose

C. Provide oversight, document assumptions and enable review of consequential decisions

D. Deploy the model without monitoring after launch

Correct Answer: C. Provide oversight, document assumptions and enable review of consequential decisions