Applied Machine Learning for Complex Data

Introduction and Context

Machine learning becomes substantially more demanding when data are sequential, linguistic, relational, or strongly dependent on context. In these settings, the central challenge is not merely selecting an algorithm. It is constructing a valid learning problem: defining the prediction target, representing the information available at decision time, validating the model under realistic conditions, and interpreting performance in relation to operational consequences.

This chapter examines four important application areas: time-series forecasting, text classification, recommendation systems, and other domain-specific prediction problems. Although the data types differ, they share several principles:

1. The prediction task must be defined precisely.
2. Features must represent only information legitimately available when a prediction is made.
3. Training, validation, and test data must reflect deployment conditions.
4. Evaluation metrics must match the business or social decision being supported.
5. Model performance must be monitored after deployment because data-generating processes change.

A model can achieve excellent apparent accuracy and still be unusable if it learns from the future, from duplicate users, from information created after the decision, or from a target-derived feature. Such errors are forms of leakage and are among the most serious risks in applied machine learning.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish forecasting, classification, ranking, and recommendation problems;
• prepare time-series data without temporal leakage;
• select and evaluate forecasting models using appropriate baselines and metrics;
• represent documents for text classification and understand the role of class imbalance;
• explain the difference between content-based, collaborative, and hybrid recommendation;
• design group-aware, time-aware, and context-aware validation strategies;
• identify common sources of leakage and distribution shift;
• assess practical, ethical, and governance issues in complex-data applications.

1. Framing Complex-Data Problems

1.1 Prediction target and decision time

A supervised learning problem consists of observations, predictors, a target, and a decision context. Let x_i denote the features for case i, and let y_i denote its target. A model estimates a function:

Here, ŷ_i is the prediction, f is the model, and θ represents learned parameters. This formulation is incomplete unless the time at which the prediction is made is specified. A feature such as “number of complaints in the following month” may be highly predictive, but it is invalid if the model is intended to operate at the start of the current month.

The decision-time question is therefore: What was known, or could legitimately have been known, when the prediction would have been generated? Every feature should be traceable to that information boundary.

1.2 Common complex-data task types

The same algorithm can serve different tasks, but the target definition and validation design must change. For example, predicting whether a customer will purchase is classification; ordering products by likely relevance is ranking; and deciding which products to display is recommendation.

2. Time-Series Forecasting

A time series is a sequence of observations indexed by time. The ordering is meaningful, and observations are often dependent. Let y_t be the value at time t. A forecasting model estimates one or more future values, such as y_t+1 or y_t+h, where h is the forecast horizon.

2.1 Components and dependence

A series may contain:

• Trend: a persistent long-term increase or decrease;
• Seasonality: a repeating pattern associated with calendar or operational cycles;
• Cyclic variation: longer, less regular fluctuations;
• Noise: unpredictable variation;
• Autocorrelation: dependence between observations at different time lags;
• External effects: explanatory variables such as price, weather, holidays, or promotions.

A simple additive decomposition is:

where T_t is trend, S_t is seasonal effect, and R_t is the remainder. This decomposition is useful when the seasonal amplitude is approximately constant. If seasonal variation grows with the level, a multiplicative formulation may be more appropriate:

The multiplicative model requires suitable positive-valued data. A logarithmic transformation can sometimes convert multiplicative relationships into additive ones, but the transformation and its effect on interpretation must be documented.

2.2 Forecasting baselines

A complex model should be compared with simple baselines. Important baselines include:

• Naïve forecast: ŷ_t+1 = y_t;
• Seasonal naïve forecast: use the observation from the corresponding previous season, such as ŷ_t = y_t-m, where m is the seasonal period;
• Historical mean: use the average of past observations when a stable mean is plausible.

If a sophisticated model cannot outperform an appropriate naïve baseline on a realistic test period, its additional complexity is not justified.

2.3 Common forecasting models

Moving average. A simple smoothed estimate is:

Here, k is the window length. The method is easy to explain but may respond slowly to changing trends.

Exponential smoothing. Simple exponential smoothing updates a level estimate:

where ℓ_t is the estimated level and α satisfies 0 < α ≤ 1. A larger α gives more weight to recent observations. Extensions add trend and seasonality.

Autoregression. An autoregressive model of order p is:

where c is an intercept, φ_j are coefficients, and ε_t is an error term. The formulation assumes that the relationship between the current value and selected past values is sufficiently stable. It is not automatically appropriate for non-stationary or strongly changing series.

Regression with external variables. Demand may be predicted from price, promotion, weather, and calendar variables. Such models are useful only when the external variables are available, or can themselves be forecast, at prediction time.

Tree-based and neural models. Random forests, gradient boosting, recurrent neural networks, temporal convolutional networks, and transformer-based models can represent nonlinear effects and multiple covariates. They often require more data, more careful tuning, and stronger monitoring than classical approaches. Greater complexity does not remove the need for leakage-safe validation.

2.4 Forecasting metrics

For actual values y_i and predictions ŷ_i, with n observations, mean absolute error is:

MAE has the same units as the target and treats all absolute errors linearly. Root mean squared error is:

RMSE penalizes large errors more heavily than MAE. Mean absolute percentage error is often written as:

MAPE is undefined when y_i = 0 and can become unstable when actual values are close to zero. It should therefore not be used automatically for intermittent or low-volume demand. A scaled measure such as mean absolute scaled error compares model error with a naïve benchmark and is often more suitable for comparing series of different scales.

2.5 Forecast validation

Randomly shuffling time-series observations is usually invalid because it allows training data to contain information from after the validation period. Better approaches include:

• Chronological holdout: train on an earlier period and test on a later period;
• Rolling-origin evaluation: repeatedly train on an expanding or moving historical window and forecast the next period;
• Blocked time-series cross-validation: use several ordered training-validation blocks.

A rolling-origin design estimates how the model behaves when repeatedly deployed. Any preprocessing operation—scaling, imputation, feature selection, and target encoding—must be fitted using only the training portion of each split.

3. Text Classification

Text classification assigns documents, messages, or other language units to categories. Examples include sentiment analysis, spam detection, support-ticket routing, and content moderation.

3.1 Text representation

Raw text must be converted into numerical features. Common approaches are:

• Bag of words: counts or binary indicators for vocabulary terms;
• n-grams: sequences of one or more adjacent tokens;
• TF–IDF: weights terms according to local frequency and rarity across documents;
• Embeddings: dense vectors intended to represent semantic or contextual similarity;
• Transformer representations: context-sensitive representations generated by pretrained language models.

For term t in document d, a common TF–IDF definition is:

Here, tf(t,d) is the term frequency, N is the number of documents, and df(t) is the number of documents containing the term. Smoothing and normalization variants are frequently used. The formula is meaningful only when the vocabulary and document-frequency statistics are learned from the training data; computing them over the full dataset leaks information from validation or test documents.

Linear models such as logistic regression and linear support vector machines often perform strongly with TF–IDF features, particularly when the dataset is moderate in size. Neural language models can capture word order and context but require careful computational, licensing, privacy, and evaluation decisions.

3.2 Classification metrics and imbalance

Accuracy is:

where TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives. Accuracy can be misleading when one class dominates. Precision and recall are:

The F1 score is the harmonic mean of precision and recall:

The appropriate metric depends on the consequence of errors. In safety screening, missing a positive case may be more serious than creating an additional review; in automated customer routing, excessive false alarms may reduce operational efficiency. Thresholds should be selected using validation data and an explicit cost or service objective.

3.3 Text-specific risks

Text data may contain duplicated documents, repeated templates, author-specific clues, quoted replies, timestamps, and personally identifiable information. If documents from the same user appear in both training and test sets, a model may learn the user’s writing style rather than the intended generalizable pattern. Group-based splitting by author, customer, conversation, or organization may be necessary.

Preprocessing must also be appropriate to the task. Removing negation words can damage sentiment classification; aggressive stemming may reduce interpretability; and translation or spelling normalization can alter protected-group language patterns. Human review and subgroup evaluation are important for high-impact uses.

4. Recommendation and Ranking

Recommendation systems help select or order items for users. They differ from ordinary classification because the system often chooses a small ranked list from a very large item catalogue, and because user exposure affects the data subsequently observed.

4.1 Main recommendation paradigms

Content-based recommendation represents users and items using item attributes. An item is recommended when its attributes match a user’s historical preferences. This approach can recommend new items if their content is known, but may be narrow and dependent on high-quality metadata.

Collaborative filtering learns from interactions among users and items. In explicit feedback, users provide ratings; in implicit feedback, clicks, views, purchases, or skips are observed. Implicit feedback is not equivalent to a negative rating: absence of a click may mean lack of exposure rather than dislike.

Matrix factorization represents an interaction matrix with lower-dimensional user and item vectors. A predicted preference can be written as:

where μ is the global mean, b_u and b_i are user and item biases, and p_u and q_i are latent vectors. The inner product p_uᵀq_i measures compatibility. This model is a useful conceptual foundation, but real systems must address new users, new items, changing preferences, catalogue constraints, and feedback bias.

Hybrid systems combine content, collaborative, contextual, and business signals. They are often more robust because they can use content for cold-start items and interaction history for established items.

4.2 Ranking evaluation

Offline recommendation evaluation commonly uses a held-out interaction and asks whether it appears among the top k recommendations. Recall at k is:

Normalized discounted cumulative gain gives greater weight to relevant items near the top. For relevance grades [[MATH rel_i]] at rank [[MATH i]], one common form is:

NDCG divides DCG by the ideal DCG for the same case, producing a value between 0 and 1 when relevance values are nonnegative. Offline metrics do not fully measure satisfaction, diversity, fairness, revenue, or long-term retention. Online experiments and safety monitoring may therefore be required.

4.3 Recommendation leakage and exposure bias

A random interaction split can leak future behaviour into training. A time-based split is usually safer. If a user’s later purchase is used to train a model that is evaluated on an earlier recommendation opportunity, the evaluation is invalid.

Exposure bias arises because the system observes feedback mainly for items it chose to display. A displayed item has a greater opportunity to be clicked than an undisplayed item. Evaluation should record exposure, account for position effects where possible, and avoid interpreting missing interaction as definite rejection.

5. Domain-Aware Validation and Leakage Prevention

5.1 Types of leakage

• Temporal leakage: future observations or future-derived aggregates enter training.
• Target leakage: a predictor is created using the target or a post-outcome event.
• Train-test contamination: preprocessing is fitted on all data before splitting.
• Duplicate leakage: near-identical records appear across splits.
• Group leakage: records from the same person, household, patient, machine, or conversation appear in both training and test data.
• Selection leakage: the evaluation sample is formed using information unavailable at deployment.
• Context leakage: a variable encodes the decision already made, such as an approval code when predicting approval.

5.2 Choosing the split unit

The correct split unit depends on how the model will be used:

A useful workflow is:

1. Define the deployment event and prediction timestamp.
2. Freeze the information boundary.
3. Identify entity groups and repeated records.
4. Create training, validation, and test sets according to deployment.
5. Fit all learned transformations within each training fold.
6. Compare with a simple baseline.
7. Inspect errors by time, group, geography, class, and operating condition.
8. Document assumptions and monitor post-deployment drift.

5.3 Distribution shift

A deployed model may face a different distribution from its training data. Covariate shift occurs when the feature distribution changes; prior-probability shift occurs when class frequencies change; concept drift occurs when the relationship between features and target changes.

Monitoring should include data quality, missingness, feature ranges, prediction distributions, calibration, subgroup performance, and delayed outcome metrics. Retraining should not be automatic without checking whether labels remain comparable and whether the underlying process has changed.

6. Worked Example: Leakage-Safe Retail Demand Forecasting

Consider a retailer forecasting daily demand for each store and product. The business needs tomorrow’s forecast at 18:00 today to plan replenishment.

Problem definition

The target is demand on day t+1. Candidate predictors include recent sales, day of week, holiday indicators, price, planned promotion, and weather forecast available at 18:00. The following day’s actual weather and the final end-of-day promotion outcome are not available at decision time and must not be used.

Feature construction

A seven-day lag is defined as y_t−7, and a recent mean may be:

The exclusion of y_t may be necessary if day-t sales are not complete at 18:00. The feature window must match the operational timestamp rather than the calendar date alone.

Validation design

Suppose data cover January through December. A valid design could train on January–August, validate on September, and test on October–December. A stronger design uses rolling origins: train through April and validate in May, train through May and validate in June, and so on. Store-product groups should remain identifiable so that the model is tested on the same deployment population, while a separate cold-start test may evaluate new products.

Model comparison

Assume the test-period results are:

The third result appears impressive but is invalid because it uses information unavailable at prediction time. It illustrates why a lower error is not evidence of a better deployable model. The valid gradient-boosting model reduces MAE by 4.3 units relative to the seasonal baseline, but management should also examine stockout costs, overstock costs, performance during promotions, and errors for low-volume products.

Learning points

The case demonstrates that feature timestamps, realistic validation, baseline comparison, and operational metrics are inseparable. A technically sophisticated model can fail if its inputs cannot be produced reliably at the required time.

7. Other Domain Applications

Healthcare

Patient records contain repeated measurements, multiple providers, and delayed outcomes. Splitting individual visits randomly can cause patient leakage. Patient-level grouping, chronological evaluation, calibration, missingness analysis, and external validation are essential. A high-risk prediction should support clinical judgement rather than silently replace it.

Finance and fraud detection

Fraud labels are delayed and often incomplete. Class imbalance, changing attacker behaviour, and intervention effects complicate evaluation. Precision at a review capacity, detection delay, false-positive burden, and monetary loss may be more useful than accuracy. Features must be based on transaction information available before or during the decision.

Industrial monitoring

Sensor data are autocorrelated and often generated by machines operating under different regimes. Random row-level splitting can produce overly optimistic results because adjacent observations are nearly duplicates. Splits by machine, production run, or time period may be required. Maintenance actions can themselves alter future sensor patterns, creating feedback loops.

8. Professional, Ethical, and Governance Considerations

Complex-data systems affect people and organizations, so technical performance is only one evaluation dimension.

• Privacy: text, transactions, location, and health data may contain identifiable or sensitive information. Collection, retention, access, and reuse require lawful and ethical controls.
• Fairness: performance should be examined across relevant demographic and operational groups. Aggregate accuracy can conceal unequal error rates.
• Transparency: users and decision-makers need understandable descriptions of inputs, limitations, confidence, and recourse procedures.
• Human oversight: high-impact decisions require escalation paths and the ability to challenge or correct outputs.
• Security: models and feature pipelines may be attacked through data poisoning, prompt or text manipulation, or extraction of sensitive information.
• Reproducibility: maintain versioned datasets, code, feature definitions, model parameters, evaluation splits, and approval records.
• Sustainability: model complexity should be justified by measurable benefit, considering computation, latency, and maintenance cost.

Regulatory obligations vary by jurisdiction and application. A responsible professional therefore treats legal review, data governance, documentation, and impact assessment as part of model development rather than as post-deployment paperwork.

9. Contemporary Developments and Best Practices

Foundation models and pretrained language models have expanded the possibilities of text analysis, retrieval, summarization, and multimodal prediction. However, pretrained systems may encode bias, retain sensitive information, or perform poorly on specialized language. Domain adaptation should be evaluated against a simple task-specific baseline, and generated outputs require factuality and safety checks.

Modern recommendation systems increasingly combine retrieval, ranking, contextual signals, and representation learning. In forecasting, probabilistic and global models can learn across many related series. These developments do not change the fundamentals: define the information boundary, validate under deployment conditions, compare against baselines, and monitor change.

A practical best-practice checklist is:

1. Write a one-sentence deployment definition.
2. Record the prediction time and forecast horizon.
3. Create a data dictionary with source timestamps and ownership.
4. Select a split that reflects the real unit of generalization.
5. Build a naïve or business-rule baseline.
6. Fit preprocessing only within training data.
7. Select metrics based on consequences, not convenience.
8. Evaluate important subgroups and operating conditions.
9. Test robustness to missing, delayed, and shifted inputs.
10. Document limitations and establish monitoring before release.

Chapter Summary

Applied machine learning for complex data requires more than algorithm selection. Time-series forecasting must respect temporal order, compare against naïve baselines, and use rolling or chronological evaluation. Text classification depends on appropriate representations, class-sensitive metrics, and protection against author, duplicate, and preprocessing leakage. Recommendation systems combine content, collaborative, and contextual information, but must account for cold start, exposure bias, temporal behaviour, and ranking quality.

Domain-aware validation is the unifying framework. The split must reflect the intended unit of generalization, and every feature must be available at the decision time. Leakage, distribution shift, feedback loops, fairness, privacy, and monitoring determine whether a model remains trustworthy outside the laboratory. Professional practice therefore combines statistical evaluation with operational, ethical, and governance analysis.

Key Terms

Decision time: The point at which a prediction is generated and the information boundary is fixed.

Forecast horizon: The distance between the prediction time and the future target time.

Temporal leakage: Use of future information in training or evaluation.

Target leakage: A predictor contains information caused by, or derived from, the target outcome.

Autocorrelation: Dependence between observations in a time series at different lags.

Seasonality: A repeating pattern associated with a regular calendar or operational cycle.

Naïve forecast: A simple forecast based on a recent or corresponding past observation.

Rolling-origin evaluation: Repeated forecasting evaluation using progressively later training cutoffs.

TF–IDF: A text weighting method combining term frequency with inverse document frequency.

Embedding: A numerical vector representation intended to capture similarity or meaning.

Precision: The proportion of predicted positive cases that are actually positive.

Recall: The proportion of actual positive cases that are detected.

Collaborative filtering: Recommendation based on patterns of interaction among users and items.

Cold start: Difficulty making recommendations for a new user or item with little interaction history.

Matrix factorization: A method representing users and items with latent vectors.

Exposure bias: Distortion caused because feedback is observed mainly for displayed items.

Group leakage: Leakage caused by placing related entities in both training and test data.

Concept drift: Change over time in the relationship between predictors and the target.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Data shift: A change between training and deployment data distributions or relationships.

Review Questions

Short Answer Questions

1. Why is random shuffling usually inappropriate for time-series validation?
2. State one situation in which recall is more important than precision.
3. What is the difference between content-based and collaborative recommendation?
4. Define target leakage and give one example.
5. Why can accuracy be misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain how a retail demand-forecasting system should be designed from target definition through post-deployment monitoring.
2. Compare bag-of-words, TF–IDF, embeddings, and transformer representations for text classification.
3. Explain the main sources of bias and leakage in recommendation systems and propose controls.
4. Describe chronological, group-based, and rolling-origin validation, including when each is appropriate.
5. Discuss why technical performance, ethical governance, and operational feasibility must be evaluated together.

Analytical / Application Questions

1. A hospital randomly splits patient visits into training and test sets. Explain the likely leakage risk and design a better evaluation procedure.
2. A fraud model has 99.5% accuracy but detects only 10% of fraudulent transactions. Which metrics and operational analyses should be added, and why?
3. A recommendation engine is evaluated by randomly hiding purchases from a user’s entire history. Identify the temporal problem and propose a more realistic offline test.

Multiple Choice Questions with Answers

Question 1

Which validation method is generally most appropriate for forecasting future demand?

A. Random row-level shuffling

B. Chronological or rolling-origin validation

C. Validation using only the largest class

D. Validation after combining training and test data

Correct Answer: B. Chronological or rolling-origin validation

Question 2

In a forecasting model, which feature is most likely to create temporal leakage when predicting tomorrow’s demand at the end of today?

A. Day of week

B. Yesterday’s demand

C. Tomorrow’s actual weather

D. A planned holiday indicator

Correct Answer: C. Tomorrow’s actual weather

Question 3

Which metric has the same units as the forecast target?

A. MAE

B. MAPE

C. Accuracy

D. F1 score

Correct Answer: A. MAE

Question 4

Why can accuracy be misleading in a highly imbalanced classification problem?

A. It cannot be calculated for binary targets

B. A model can obtain high accuracy by mostly predicting the majority class

C. It always gives more weight to false positives

D. It requires continuous predictions

Correct Answer: B. A model can obtain high accuracy by mostly predicting the majority class

Question 5

What does TF–IDF generally do?

A. Weights common terms more heavily than rare terms across all documents

B. Represents every document with exactly one feature

C. Emphasizes terms frequent in a document but relatively rare across documents

D. Uses future labels to create text features

Correct Answer: C. Emphasizes terms frequent in a document but relatively rare across documents

Question 6

Which recommendation approach can use item descriptions to recommend a new item with no interaction history?

A. Content-based recommendation

B. Pure collaborative filtering based only on historical interactions

C. Random recommendation

D. User deletion

Correct Answer: A. Content-based recommendation

Question 7

What does cold start refer to in recommendation systems?

A. A server temperature problem

B. The difficulty of recommending for a new user or item with little history

C. A model with too many seasonal variables

D. A ranking metric with a value of zero

Correct Answer: B. The difficulty of recommending for a new user or item with little history

Question 8

Which split is most appropriate when the intended model must generalize to entirely new patients?

A. A random split of visits from the same patients

B. A group-based split by patient

C. A split based only on alphabetical order

D. A split after using all patients to fit preprocessing

Correct Answer: B. A group-based split by patient

Question 9

What is exposure bias in recommendation?

A. The tendency for displayed items to receive more opportunities for interaction

B. The use of too few item features

C. A model’s inability to calculate precision

D. A seasonal pattern in sales data

Correct Answer: A. The tendency for displayed items to receive more opportunities for interaction

Question 10

Which practice helps prevent train-test contamination during feature scaling?

A. Fit the scaler on the complete dataset before splitting

B. Fit the scaler separately using each training fold

C. Use the test labels to select the scale

D. Scale only the target after evaluation

Correct Answer: B. Fit the scaler separately using each training fold