Analytical Method Selection and Model Development

Introduction

A capstone project in business analytics is not simply an exercise in applying software to a dataset. It is an evidence-based investigation in which a business problem is translated into an analytical question, an appropriate method is selected, a model is developed and evaluated, and the results are converted into a defensible decision. The quality of the conclusion depends as much on method selection and research design as on computational accuracy.

Business questions differ in purpose. A manager may want to know what happened, why it happened, what is likely to happen next, whether an intervention caused an outcome, or what action should be taken. These questions correspond broadly to descriptive, diagnostic, predictive, causal and prescriptive analytics. The categories overlap, but they should not be confused. A highly accurate predictive model does not necessarily explain causes, and a statistically significant association does not automatically justify a management intervention.

This chapter presents a structured approach to selecting analytical methods and developing models suitable for a bachelor-level capstone dissertation. It emphasizes business context, theory, assumptions, evidence, interpretability, validation, ethics and implementation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish descriptive, diagnostic, predictive, causal and prescriptive analytical objectives.
2. Translate a business problem into an analytical question and estimand.
3. Select methods using purpose, data structure, theory, assumptions and decision context.
4. Develop and evaluate statistical, machine-learning and optimization models.
5. Interpret model outputs without overstating evidence or causality.
6. Identify risks involving bias, privacy, fairness, leakage, reproducibility and governance.
7. Justify a method choice in a capstone dissertation.

1. From Business Problem to Analytical Question

A business problem is usually stated in operational language: “Customer churn is increasing,” “inventory costs are too high,” or “the campaign appears ineffective.” Such statements identify a concern, not yet a research design. The analyst should clarify five elements:

• Decision maker: Who will use the result?
• Decision: What action could be changed?
• Outcome: What measurable result matters?
• Unit of analysis: Customer, transaction, store, employee, product or time period?
• Time horizon: Historical explanation, next month’s forecast or a multi-year intervention?

A useful analytical question specifies the population, outcome, predictors or intervention, time period and intended use. For example: “Among active subscription customers, which observable characteristics predict cancellation within the next 30 days, and how can retention resources be prioritized?” This is more precise than “Why are customers leaving?”

1.1 The estimand and the prediction target

An estimand is the quantity the analysis seeks to learn. In a predictive study it may be the probability that customer i churns:

Here, Y_i is the binary churn outcome and X_i is the information available at the prediction time. The conditioning information is essential: variables observed after cancellation cannot legitimately be used to predict it.

A causal study may instead seek an average treatment effect:

where Y(1) is the potential outcome if a unit receives treatment, Y(0) is the potential outcome without treatment, and E denotes an expected value. This quantity requires stronger assumptions than prediction, including a credible treatment comparison and control of relevant confounding.

2. Five Analytical Purposes

2.1 Descriptive analytics

Descriptive analytics summarizes observed data. It answers questions such as:

• What were monthly sales by region?
• Which products generated the most revenue?
• How did service response time vary?

Typical methods include frequency tables, averages, medians, percentiles, ratios, cross-tabulations, dashboards and time-series plots. The arithmetic mean is:

where xᵢ is an observation and n is the number of observations. The mean is useful for reasonably symmetric data but can be distorted by extreme values; the median may better represent skewed income, transaction-value or waiting-time distributions.

Descriptive results establish the data context but do not by themselves explain causes or predict future outcomes.

2.2 Diagnostic analytics

Diagnostic analytics investigates explanations for observed patterns. It may use segmentation, drill-down analysis, correlation, hypothesis testing, regression, process analysis or root-cause analysis. For two quantitative variables, Pearson’s correlation is commonly written as:

The coefficient ranges from −1 to +1 and measures linear association. It does not establish that changes in x cause changes in y. Correlation can result from confounding, reverse causality, common trends or chance.

2.3 Predictive analytics

Predictive analytics estimates unknown or future outcomes using information available at the time of prediction. Classification predicts categories, such as churn versus retention; regression predicts numerical values, such as demand; and time-series methods predict ordered observations over time.

A multiple linear regression model is:

Here, yᵢ is the outcome, xᵢⱼ are predictors, β₀ is the intercept, βⱼ are partial slope coefficients and εᵢ is the unexplained error. The model is appropriate when a reasonably linear conditional mean is plausible and the outcome is continuous. Independence, suitable handling of variance, and attention to influential observations are also required for reliable inference.

For binary outcomes, logistic regression models the log-odds:

The odds ratio associated with a one-unit increase in xⱼ, holding other variables constant, is e^βⱼ. Logistic regression predicts probabilities between zero and one and is often more interpretable than a complex model. Its coefficients should not be described as causal unless the research design supports causal interpretation.

Common predictive algorithms include decision trees, random forests, gradient boosting, k-nearest neighbours, support-vector machines and neural networks. Increasing complexity may improve predictive performance but reduce transparency, increase computational requirements and make governance more difficult.

2.4 Causal analytics

Causal analytics asks what would happen if an action, policy or treatment changed. Examples include the effect of a price reduction on demand or the effect of a training programme on productivity. Causal claims require a counterfactual: the treated unit’s outcome must be compared with what would have happened to that same unit without treatment.

A randomized experiment supports causal inference when treatment is randomly assigned, implementation is credible, outcomes are measured consistently and attrition is limited. In observational data, approaches may include matching, regression adjustment, instrumental variables, regression discontinuity and difference-in-differences. Each depends on assumptions that should be explicitly stated.

For difference-in-differences, a simplified two-group, two-period estimator is:

The subscripts T and C denote treatment and comparison groups. A central assumption is parallel trends: without treatment, the groups would have experienced comparable changes over time. Pre-treatment trend evidence should therefore be examined. A before-and-after change in one treated group alone is not sufficient evidence of an intervention effect.

2.5 Prescriptive analytics

Prescriptive analytics recommends actions subject to constraints. It combines forecasts, objectives, decision variables and operational rules. Linear programming has the general form:

Here, xⱼ are decision variables, cⱼ are objective coefficients, aᵢⱼ are resource-use coefficients and bᵢ are available capacities. Units must be consistent: for example, labour hours consumed cannot be compared directly with a monetary budget unless constraints are expressed in their appropriate units.

Prescriptive models are only as useful as their objective function, constraints and inputs. An “optimal” solution may be inappropriate if demand forecasts are unreliable, service quality is omitted, or the model ignores fairness and implementation feasibility.

3. Selecting a Method Systematically

Method selection should be justified rather than based on familiarity or software availability. A practical decision sequence is:

1. Define the decision and analytical purpose.
2. Identify the outcome type: continuous, binary, categorical, count, time-to-event or spatial.
3. Establish the unit of analysis and time structure.
4. Examine data quality, sample size, missingness and measurement validity.
5. Use business theory to identify plausible predictors, mechanisms and constraints.
6. State assumptions and assess whether the data can support them.
7. Compare a transparent baseline with more complex alternatives.
8. Validate using a design that reflects deployment conditions.
9. Select the method using both statistical performance and business consequences.

3.1 Baselines and comparison models

A baseline provides a meaningful reference. For a continuous outcome, a mean predictor may be a baseline; for classification, the majority class or a calibrated historical rate may be appropriate. A complex model should demonstrate improvement over this baseline on unseen data. A simpler model may be preferred when its performance is similar and its explanation is substantially clearer.

3.2 Performance metrics

For a regression model, mean absolute error is:

where yᵢ is the observed value and ŷᵢ is the prediction. MAE is expressed in the outcome’s units and is therefore easy to communicate. Root mean squared error is:

RMSE penalizes large errors more heavily than MAE. Neither metric is universally superior; the choice should reflect the cost of errors.

For binary classification, a confusion matrix produces true positives, true negatives, false positives and false negatives. Accuracy is:

Precision is TP/(TP+FP), while recall is TP/(TP+FN). Accuracy can be misleading with imbalanced classes. If missing a high-risk customer is costly, recall may matter more; if intervention resources are scarce, precision may matter more. Threshold selection should be connected to costs and capacity rather than automatically set to 0.5.

For probability predictions, calibration asks whether cases assigned a probability of 0.70 experience the event approximately 70% of the time. A model may rank cases well but be poorly calibrated, which is dangerous when probabilities guide resource allocation.

4. Model Development Workflow

4.1 Data preparation and feature design

Data preparation includes defining inclusion criteria, removing duplicates, checking ranges and units, documenting missing values and creating an analysis-ready dataset. Every transformation should be reproducible. Missing values should not be silently converted to zero because zero may have a substantive meaning.

Feature engineering can improve performance, but it must preserve temporal and conceptual validity. For example, a customer’s number of support contacts in the previous 30 days may be a legitimate predictor of future churn; a “cancellation reason” recorded after churn is leakage.

Categorical variables may be represented using indicator variables. If a variable has g categories, one category is normally omitted as a reference in a regression model to avoid perfect linear dependence. Numeric variables may require scaling for distance-based or gradient-based algorithms, although tree-based methods generally do not require standardization.

4.2 Splitting and validation

A random training-test split is inappropriate when observations are time-dependent and future information could enter the training sample. Time-series forecasting should use chronological validation, such as training on earlier periods and testing on later periods. Repeated cross-validation can estimate performance variability, but preprocessing and feature selection must occur inside each training fold to avoid leakage.

The final test set should be used sparingly, ideally after model development is complete. Reporting only the best of many tested models without accounting for selection creates an overly optimistic estimate.

4.3 Assumptions and diagnostics

For linear regression, important considerations include:

• approximate linearity of the conditional mean;
• independence or an explicitly modelled dependence structure;
• absence of severe multicollinearity;
• appropriate treatment of heteroscedasticity;
• influential observations and outliers;
• valid and consistently measured variables.

The coefficient of determination is:

It represents the reduction in squared error relative to predicting the sample mean, under the specified sample and model. A high R² does not prove causality, guarantee future performance or imply that the model is useful for decision-making.

4.4 Interpretability

Interpretability operates at several levels. A coefficient may describe an adjusted association; a feature-importance score may describe contribution to prediction; and a partial-dependence or local explanation may describe model behaviour under specified conditions. These are not interchangeable with causal effects.

Use domain language, units and uncertainty. Instead of writing “marketing caused sales to increase,” a defensible report might state: “Holding the included variables constant, the fitted model associates a one-unit increase in campaign exposure with an estimated increase of …; the observational design does not establish causation.” Confidence intervals, prediction intervals or bootstrap intervals should be reported when inference is intended.

5. Worked Example: Selecting a Churn Model

Suppose a subscription business has 12 months of customer records. The business wants to target retention offers before the next billing cycle. The outcome is cancellation within 30 days. Available predictors include tenure, plan type, monthly fee, recent service contacts and days since last login.

The question is predictive, not automatically causal: which customers are at elevated risk using information available before the billing cycle? A logistic regression is a defensible starting point because the outcome is binary, managers can inspect coefficient direction and probabilities can support prioritization. A decision tree may be added to capture non-linear relationships, while a random forest may be evaluated as a performance benchmark.

The analyst should:

1. Define the prediction date for every customer.
2. Exclude post-cancellation variables.
3. Use earlier months for training and a later month for testing.
4. Report recall, precision, calibration and expected intervention capacity.
5. Compare with a simple historical-rate baseline.
6. Examine performance by plan type and customer segment.
7. Select a threshold consistent with the number of offers available.

If 1,000 customers are eligible but only 100 offers can be issued, the top 100 predicted risks may be selected. This operational rule is not equivalent to maximizing accuracy. The business should compare the expected value of retained customers, offer cost, customer experience and possible unfair targeting.

6. Illustrative Case: Regional Retail Demand Planning

Background. A fictional retailer experiences frequent stockouts in fast-moving products while holding excess inventory in slower stores. Managers request “an AI forecasting solution.”

Problem definition. The actual decision is how much of each product to replenish at each store each week. The project therefore contains both a predictive component—forecasting demand—and a prescriptive component—allocating inventory under capacity and service constraints.

Analysis. The analyst identifies store-product-week as the unit of analysis and examines promotions, holidays, price, local events, inventory availability and historical sales. A crucial measurement issue is censored demand: observed sales may be low because the product was unavailable. Treating all low sales as low demand would bias the forecast.

A seasonal naïve forecast is used as a baseline. A regression model with calendar and promotion variables is compared with a gradient-boosting model. Chronological validation is used. Forecast accuracy is assessed with MAE, while business evaluation also considers stockout rate and excess inventory. The model with the lowest error is not automatically selected if its predictions are poorly calibrated during promotions or cannot be explained to replenishment managers.

Approach and solution. The selected forecasting model produces expected demand and uncertainty information. A replenishment optimization model then considers inventory on hand, delivery capacity, minimum order quantities and a service-level target. The objective includes purchase and holding costs, with a penalty for estimated stockouts. Managers review exceptions rather than accepting every recommendation automatically.

Outcome. In this illustrative case, a pilot is evaluated against comparable stores and a pre-specified period. The evaluation reports changes in stockouts, waste, inventory value and customer availability, not merely model accuracy. The pilot also includes a rollback rule when data quality or forecast error exceeds a defined threshold.

Learning points. The request for “AI” concealed a multi-stage decision problem. Forecasting, causal evaluation and optimization answer different questions. Data-generating processes, operational constraints and human oversight are as important as algorithm selection.

7. Ethical, Governance and Regulatory Considerations

Analytical models can reproduce historical discrimination, create privacy risks or make decisions that affected individuals cannot contest. A capstone project should document the lawful and ethical basis for data use, data minimization, access controls, retention, anonymization or pseudonymization, and secure handling of sensitive attributes.

Fairness should be examined in context. Possible assessments include differences in selection rates, false-positive rates or calibration across relevant groups. No single fairness metric is universally correct; metrics may conflict, and group comparisons can be unstable for small samples. The report should explain which harms are material and how they are mitigated.

Governance practices include a data dictionary, model card, version control, reproducible code, approval records, assumptions log, validation results and monitoring plan. Monitoring should detect performance drift, changes in population composition, data pipeline failures and disparate impacts. Generative AI may assist with coding or documentation, but outputs require verification, confidential data should not be exposed, and the analyst remains responsible for methodological decisions.

8. Common Failure Modes and Best Practices

Common failures include choosing a method before defining the question, confusing prediction with explanation, using post-outcome information, reporting accuracy for imbalanced data, ignoring the baseline, tuning on the test set, treating missingness as harmless, and presenting automated recommendations without constraints or human review.

Best practice is to maintain an analysis decision record containing the question, estimand, data sources, exclusions, transformations, candidate methods, assumptions, validation design, metrics, limitations and decision rule. A strong dissertation also separates exploratory findings from confirmatory claims, reports negative or inconclusive results, and explains what additional evidence would change the conclusion.

Chapter Summary

Analytical method selection begins with the business decision and the type of evidence required. Descriptive methods summarize what happened; diagnostic methods investigate patterns; predictive methods estimate future or unknown outcomes; causal methods estimate intervention effects under counterfactual assumptions; and prescriptive methods recommend actions subject to objectives and constraints.

A defensible model uses a clearly defined unit of analysis, outcome, information time and estimand. Method choice should reflect theory, data structure, assumptions, interpretability, validation requirements and the cost of errors. Baselines, leakage prevention, appropriate metrics, chronological validation where necessary and transparent reporting are central to model development. Complex algorithms are not automatically better. Ethical governance, fairness, privacy, reproducibility and monitoring must be integrated into the capstone rather than added after modelling.

Key Terms

Analytical question: A precise question linking a business decision to measurable evidence.

Baseline: A simple reference method against which a model is evaluated.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Causal effect: The change in an outcome attributable to an intervention relative to a counterfactual.

Classification: Prediction of a discrete category or class.

Confounding: Distortion of an association because a third variable relates to both exposure and outcome.

Cross-validation: Repeated partitioning of data for model training and evaluation.

Data leakage: Use of information unavailable at the time a prediction or decision would be made.

Descriptive analytics: Analysis that summarizes observed data.

Diagnostic analytics: Analysis that investigates reasons for observed patterns.

Estimand: The population quantity a study seeks to estimate.

Feature engineering: Creating or transforming predictors for analytical use.

Interpretability: The extent to which model behaviour can be understood and communicated.

Prescriptive analytics: Analysis that recommends actions under objectives and constraints.

Predictive analytics: Estimation of unknown or future outcomes.

Regularization: A technique that penalizes model complexity to reduce overfitting.

Regression: Modelling of relationships involving a numerical or conditional-mean outcome.

Validation: Evaluation of a model’s performance using data or procedures that reflect intended use.

Review Questions

Short Answer Questions

1. What distinguishes predictive analytics from causal analytics?
2. Why is a baseline important in model evaluation?
3. Define data leakage and give one business example.
4. Why can accuracy be misleading for an imbalanced classification problem?
5. What is the purpose of chronological validation in forecasting?

Long Answer Questions

1. Explain a structured process for selecting an analytical method for a capstone project.
2. Compare descriptive, diagnostic, predictive, causal and prescriptive analytics, including suitable applications for each.
3. Discuss the assumptions and interpretation of multiple linear regression.
4. Explain why model complexity should be balanced against interpretability and governance.
5. Describe how ethical, privacy and fairness considerations should be incorporated into model development.

Analytical / Application Questions

1. A bank wants to identify customers likely to default and decide whom to contact. Propose an analytical workflow, including outcome definition, candidate models, validation design, metrics and ethical checks.
2. A retailer observes that stores receiving a new layout have higher sales than other stores. Explain why this is not automatically a causal conclusion and propose a stronger evaluation design.
3. A logistics company has a demand forecast and limited vehicle capacity. Explain how a prescriptive model could convert forecasts into delivery recommendations and identify assumptions that should be tested.

Multiple Choice Questions with Answers

Question 1

Which analytical purpose primarily answers the question “What is likely to happen?”

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: C. Predictive analytics

Question 2

Which issue occurs when a churn model uses a cancellation reason recorded after the customer has already cancelled?

A. Underfitting

B. Data leakage

C. Heteroscedasticity

D. Calibration

Correct Answer: B. Data leakage

Question 3

What does the average treatment effect compare?

A. Two prediction algorithms

B. Observed outcomes before and after sampling

C. Expected outcomes with and without treatment

D. Training error and test error

Correct Answer: C. Expected outcomes with and without treatment

Question 4

Which metric is expressed in the same units as a continuous outcome?

A. Mean absolute error

B. Accuracy

C. Recall

D. R-squared

Correct Answer: A. Mean absolute error

Question 5

Why may accuracy be unsuitable for a highly imbalanced classification problem?

A. It cannot be calculated for binary outcomes

B. It may appear high while the minority class is poorly identified

C. It always penalizes false positives more than false negatives

D. It measures calibration directly

Correct Answer: B. It may appear high while the minority class is poorly identified

Question 6

What is a central assumption of a basic difference-in-differences design?

A. Perfect prediction

B. Parallel trends absent treatment

C. Equal sample sizes in all periods

D. A normally distributed outcome

Correct Answer: B. Parallel trends absent treatment

Question 7

In linear programming, what does a decision variable represent?

A. A quantity the model is allowed to choose

B. The unexplained error term

C. A fixed historical observation

D. A validation metric

Correct Answer: A. A quantity the model is allowed to choose

Question 8

Which validation approach is generally most appropriate when predicting future time periods?

A. Randomly mixing all observations before splitting

B. Using only the largest observations

C. Chronological training and testing

D. Evaluating on the training data

Correct Answer: C. Chronological training and testing

Question 9

A logistic regression coefficient βⱼ is commonly transformed into which quantity for interpretation?

A. βⱼ²

B. e^{βⱼ}

C. 1 − βⱼ

D. √βⱼ

Correct Answer: B. e^{βⱼ}

Question 10

What should be done with a complex model whose performance is only marginally better than a transparent baseline?

A. Automatically choose the complex model

B. Ignore business consequences

C. Consider whether the small gain justifies added complexity

D. Evaluate it only on training data

Correct Answer: C. Consider whether the small gain justifies added complexity