Big Data Analytics, Machine Learning and AI

Introduction

Big data becomes valuable only when organisations can convert large, diverse and rapidly changing datasets into reliable decisions. Big data analytics provides the methods for describing what has happened, diagnosing why it happened, predicting what is likely to happen, and prescribing what should be done. Machine learning extends these capabilities by learning patterns from data, while artificial intelligence (AI) encompasses systems that perform tasks associated with human cognitive abilities, such as perception, language understanding, reasoning and generation.

The scale of big data changes the engineering and analytical problem. A model that works on a laptop may fail when data are distributed across thousands of files, arrive continuously, contain inconsistent records, or change after deployment. Consequently, analytics must be treated as an end-to-end lifecycle involving business understanding, data engineering, feature construction, model development, evaluation, deployment, monitoring, governance and responsible use.

Learning Objectives

After completing this chapter, students should be able to:

• distinguish descriptive, diagnostic, predictive and prescriptive analytics;
• design scalable feature-engineering and machine-learning workflows;
• select appropriate supervised, unsupervised and deep-learning methods;
• evaluate models using metrics suited to the decision context;
• identify overfitting, data leakage, bias, drift and explainability problems;
• assess responsible and secure uses of generative AI in organisations.

1. The Four Types of Analytics

Analytics can be classified according to the question being answered. The categories overlap in practice, but the distinction helps organisations define objectives and evidence requirements.

Descriptive analytics uses counts, sums, averages, rates and distributions. For observations x₁, x₂, ..., x_n, the arithmetic mean is:

Here, n is the number of observations and xᵢ is observation i. The mean is appropriate for numeric data when extreme values do not dominate interpretation. The median is often more representative for skewed quantities such as income or transaction value. A rate must also preserve its denominator: a 5% failure rate means 5 failures per 100 relevant opportunities, not simply 5 failures.

Diagnostic analytics compares segments, time periods and processes. Correlation can identify linear association, but it does not establish causation. A confounding variable may influence both apparently related variables. Controlled experiments, natural experiments, process knowledge and causal methods are needed for stronger causal claims.

Predictive analytics estimates an unknown outcome from available information. A probability prediction is not the same as a decision: a fraud score of 0.7 requires a threshold, cost model and operational response. Prescriptive analytics combines predictions with constraints, objectives and possible actions. A simple decision rule selects the action with the greatest expected utility:

a is a candidate action, Y is the uncertain outcome, X is the observed information, U is utility, and E denotes conditional expected value. This rule assumes that utilities, feasible actions and relevant probabilities are sufficiently specified.

2. Machine-Learning Foundations

Machine learning learns a function from examples rather than relying entirely on hand-written rules. A dataset is commonly represented by a feature matrix X and target vector y. A model with parameters θ produces predictions ŷ = f_θ(X). Training selects parameters that minimise a loss, often with regularisation:

L measures prediction error, Ω(θ) penalises model complexity, λ ≥ 0 controls the penalty, and n is the number of training cases. The expression applies when a differentiable or optimisable objective has been defined; different algorithms use different optimisation procedures.

Learning paradigms

Supervised learning uses labelled outcomes. Regression predicts numeric values, such as energy demand. Classification predicts categories or class probabilities, such as legitimate versus fraudulent transactions. Common algorithms include linear and logistic regression, decision trees, random forests, gradient-boosted trees, support-vector machines and neural networks.

Unsupervised learning works without a target label. Clustering groups similar records, dimensionality reduction represents data in fewer dimensions, and anomaly detection identifies unusual observations. Results require interpretation because mathematical similarity is not automatically business significance.

Semi-supervised and self-supervised learning use large quantities of unlabelled data with limited labels or construct learning tasks from the data itself. These approaches are important where labelling is costly, including language, images and sensor data.

Generalisation and overfitting

A model should perform well on unseen data drawn from the intended deployment population. Overfitting occurs when a model memorises noise or accidental characteristics of training data. Underfitting occurs when its structure is too simple to capture useful patterns. Model complexity, regularisation, early stopping, feature selection and sufficient data can improve generalisation.

A proper split separates training, validation and test data. The training set fits parameters, the validation set supports model or hyperparameter selection, and the test set provides a final unbiased estimate. Time-dependent data should generally be split chronologically rather than randomly. For grouped data, records from the same person, household or device should not appear across both training and test sets when that would reveal identity-specific information.

3. Scalable Data and Feature Engineering

Feature engineering transforms raw data into variables that a model can use. It may include cleaning, joining, aggregation, encoding, scaling, text representation, temporal windows and domain-specific transformations.

A scalable workflow separates:

1. Ingestion: collect batch or streaming data with schema, provenance and quality metadata.
2. Preparation: standardise types, remove duplicates, handle missing values and validate constraints.
3. Feature construction: create variables using reproducible transformations.
4. Feature storage: publish versioned features for training and serving.
5. Training and evaluation: fit models using leakage-safe datasets.
6. Deployment and monitoring: serve predictions and observe performance, data quality and fairness.

Leakage prevention

Data leakage occurs when training uses information unavailable at prediction time. For example, using a customer’s cancellation date to predict whether that customer will cancel produces an apparently accurate but operationally useless model. Every feature must have an “as-of” time. Aggregates must use only events occurring before that time, and preprocessing parameters such as means or category dictionaries must be fitted on training data only.

Distributed processing

Large datasets are commonly partitioned across nodes. Partitioning enables parallel computation, but joins can cause data movement, commonly called a shuffle. Efficient systems minimise unnecessary columns, filter early, use suitable partition keys, cache reused data carefully, and select join strategies appropriate to data size and skew. A very frequent key can create a “hot” partition and limit parallelism.

Batch processing is appropriate for periodic reports and retraining. Stream processing is appropriate when decisions must use events close to real time. Streaming systems must address event time versus processing time, late arrivals, duplicate events, out-of-order data and exactly-once or effectively-once business effects. A technically successful stream that produces duplicate payments is not operationally correct.

Feature examples

For a retail customer, useful features might include:

• recency: time since the latest purchase;
• frequency: number of purchases in a defined window;
• monetary value: total or average spend;
• trend: change in purchase activity over time;
• text or image representations generated by an appropriate model.

Categorical variables may be one-hot encoded, frequency encoded or represented through learned embeddings. High-cardinality identifiers require caution: an account ID may cause memorisation rather than generalisable learning. Missingness itself may be informative, but missing values should not be filled without considering why they are absent.

4. Model Evaluation

Evaluation must reflect both statistical performance and the consequences of errors.

For binary classification, the confusion matrix contains true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Important measures include:

Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives detected. These formulas require non-zero denominators. Accuracy can be misleading when classes are highly imbalanced. In medical screening, missing a true case may be more costly than investigating a false alarm, so recall may be prioritised. The F1 score is the harmonic mean of precision and recall when both are important:

For regression, mean absolute error and root mean squared error are common:

Both have the same units as the target. RMSE penalises large errors more strongly than MAE. A model should also be evaluated for calibration, meaning whether predicted probabilities correspond to observed frequencies. Ranking metrics such as ROC-AUC or precision-recall AUC are useful for threshold-independent comparisons, but they do not replace selecting an operational threshold.

Cross-validation estimates performance by repeatedly training on subsets and validating on held-out folds. It must be adapted for time series, groups and repeated entities. Statistical performance should be accompanied by confidence intervals or uncertainty estimates where decisions are consequential.

5. Explainability, Fairness and Responsible AI

Explainability has several meanings. Interpretability concerns whether the model is understandable in itself, as with a small decision tree. Post-hoc explanation describes a complex model after training, using tools such as feature attribution, partial-dependence analysis or local examples. Explanations are approximations and may be unstable or misleading if the underlying data are biased or correlated.

A useful governance record includes the intended purpose, population, features, training data, evaluation design, limitations, model version, decision threshold, human oversight and incident process. Explanations should be appropriate to the audience: engineers need diagnostics, managers need risk and performance summaries, and affected individuals may need actionable reasons for a decision.

Fairness is not a single universal metric. Depending on the application, teams may examine selection rates, false-positive and false-negative rates, calibration, or error differences across protected or relevant groups. Some fairness criteria cannot generally all be satisfied simultaneously when groups have different base rates. Therefore, fairness is a sociotechnical decision involving law, policy, domain expertise and affected communities, not merely a choice of metric.

Bias can enter through historical labels, sampling, measurement, missingness, proxy variables, deployment practices and feedback loops. Removing a protected attribute does not guarantee fairness because correlated variables may act as proxies. Mitigations include better sampling, label review, reweighting, threshold analysis, human review, impact assessment and post-deployment monitoring.

Privacy and security require data minimisation, access controls, encryption, retention limits, pseudonymisation where appropriate and protection against model-inversion or membership-inference risks. Compliance obligations vary by jurisdiction and sector; technical teams must work with institutional policy and applicable data-protection, consumer-protection and anti-discrimination requirements.

6. Generative AI in Big Data Workflows

Generative AI models produce text, code, images, audio or structured outputs from learned statistical representations. Large language models generate sequences by estimating likely next tokens conditioned on context. Their fluent output is not proof of truth, reasoning validity or source authority.

Useful applications include summarising internal documents, generating draft queries, assisting code development, extracting structured fields and supporting analysts with natural-language interfaces. High-risk uses include unsupervised decisions about employment, credit, health, education or public benefits; production code or queries copied without testing; and sending confidential data to an unauthorised service.

A responsible workflow uses approved data, access controls, retrieval from authoritative sources, source citations where feasible, output validation, prompt and model versioning, human review and logging. Retrieval-augmented generation can provide relevant organisational documents to a model, but it does not eliminate incorrect retrieval, prompt injection, privacy leakage or fabricated answers. Structured output schemas and automated tests improve reliability but do not guarantee correctness.

Evaluation should include factuality, relevance, completeness, toxicity, privacy leakage, robustness to adversarial prompts, cost, latency and unequal performance across languages or groups. Generative AI should be treated as a probabilistic component within a governed system, not as an autonomous authority.

7. Deployment, Monitoring and Model Operations

A model is a component of a production service. Deployment choices include batch scoring, online APIs, edge inference and embedded analytics. The choice depends on latency, cost, availability, privacy and update requirements.

Monitoring should cover:

• data quality: missingness, range violations, schema changes and duplicates;
• data drift: changes in feature distributions;
• concept drift: changes in the relationship between inputs and outcomes;
• prediction drift: changes in score or class distributions;
• performance: accuracy, error, calibration and business outcomes;
• fairness: group-specific performance and impact;
• operations: latency, throughput, failures and resource use.

A simple distribution comparison may use the population stability index:

Here, Aⱼ and Eⱼ are the actual and expected proportions in bin j, and both must be positive. PSI is a monitoring heuristic, not proof that a model has become invalid. Drift alerts require investigation, business context and, where labels are available, direct performance evaluation.

Reproducibility requires versioned data definitions, code, features, dependencies, models and configuration. A rollback plan, champion-challenger testing, approval gates and incident response reduce operational risk. Retraining should be triggered by evidence and documented criteria rather than by an arbitrary schedule alone.

Applied Case: Illustrative Online Retail Churn System

Background. An online retailer wants to reduce customer churn. It has transaction records, customer-service contacts, product browsing events and subscription status.

Problem. The organisation needs a weekly list of customers likely to cancel within the next 30 days, together with actions that are commercially and ethically defensible.

Analysis. Analysts first define churn and the prediction time. They create weekly snapshots, ensuring that features contain only information available at each snapshot. A chronological split is used because the future should not influence the past. Churn is relatively rare, so accuracy alone is rejected as the principal metric.

Approach. Features include recent purchase count, days since last purchase, service-contact frequency, delivery delays and changes in browsing activity. A regularised logistic regression provides a transparent baseline; a gradient-boosted tree is then evaluated for possible improvement. The model is assessed using precision-recall measures, calibration, expected campaign cost and subgroup performance. A probability threshold is selected using the cost of contacting a customer, the expected value of retention and the harm of unwanted communication.

The deployment pipeline scores customers weekly, excludes those who have opted out, records the model version and sends a limited list to trained retention staff. Staff receive reason codes such as “increased delivery delays” rather than unsupported claims about personal characteristics.

Outcome. Before deployment, the team discovers that a post-cancellation service code was included in an early feature table. Removing it substantially reduces offline performance but produces a valid estimate. Monitoring is configured for data quality, score distributions, campaign outcomes and subgroup error rates. The model is reviewed after policy changes and seasonal shifts.

Learning points. The principal achievement is not selecting the most complex algorithm. It is defining the target, preventing leakage, aligning evaluation with business costs, respecting consent, providing useful explanations and monitoring the complete decision system.

Challenges and Limitations

Big data does not automatically produce better evidence. More records may contain duplicated, biased or systematically missing observations. Correlation-based models can fail when policies, markets or user behaviour change. Distributed computation introduces reproducibility and data-governance challenges. Labels may encode past institutional decisions rather than objective truth. Model explanations may create false confidence, while human review can itself be inconsistent or discriminatory.

Organisations should avoid unnecessary data collection, unsupported causal claims, metric shopping and deployment without ownership. They should document uncertainty, test alternative specifications, involve domain experts and affected stakeholders, and provide mechanisms for appeal or correction when automated decisions affect people.

Best-Practice Checklist

1. Define the decision, outcome, prediction time and affected population.
2. Establish data ownership, lawful use, quality checks and provenance.
3. Build leakage-safe, reproducible feature pipelines.
4. Use splits and metrics appropriate to time, groups, imbalance and cost.
5. Compare against simple baselines and assess uncertainty.
6. Test performance, calibration, fairness, robustness and explainability.
7. Deploy with access controls, human responsibility, logging and rollback.
8. Monitor data, model, operational and social outcomes.
9. Reassess after population, policy, data or system changes.
10. Treat generative AI output as reviewable assistance, not verified fact.

Chapter Summary

Big data analytics progresses from description to diagnosis, prediction and prescription. Machine learning estimates patterns from features and outcomes, but its value depends on valid problem definition, scalable data preparation, leakage prevention and appropriate evaluation. Supervised, unsupervised and generative methods serve different purposes and require different evidence. Classification and regression metrics must reflect imbalance, costs and units, while calibration and uncertainty support responsible decisions. Explainability, fairness, privacy, security and governance are essential parts of an analytical system rather than optional additions. Finally, deployment requires monitoring for data drift, concept drift, performance degradation and unequal impact. Generative AI can assist analytics workflows, but it must be controlled through approved data, validation, human oversight and explicit accountability.

Key Terms

Analytics: Systematic examination of data to produce insight or support decisions.

Descriptive analytics: Analysis of what has happened.

Diagnostic analytics: Analysis of why an outcome occurred.

Predictive analytics: Estimation of future or unknown outcomes.

Prescriptive analytics: Recommendation of actions under objectives and constraints.

Feature: An input variable supplied to a model.

Target: The outcome a supervised model learns to predict.

Data leakage: Use of information unavailable at the intended prediction time.

Overfitting: Learning noise or peculiarities that do not generalise.

Regularisation: A method that penalises excessive model complexity.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are detected.

Calibration: Agreement between predicted probabilities and observed frequencies.

Concept drift: Change in the relationship between inputs and outcomes.

Data drift: Change in the distribution of input data.

Explainability: The extent to which model behaviour or outputs can be understood.

Fairness: The principled assessment and mitigation of unequal model impacts.

Generative AI: AI that produces new content such as text, code or images.

Model governance: Policies and controls for responsible model development and use.

Feature store: A managed repository of reusable, versioned features.

Review Questions

Short Answer Questions

1. Distinguish descriptive, diagnostic, predictive and prescriptive analytics.
2. What is data leakage, and why does it invalidate evaluation?
3. Define precision and recall.
4. Explain the difference between data drift and concept drift.
5. Why is accuracy potentially misleading for an imbalanced classification problem?

Long Answer Questions

1. Explain how a machine-learning workflow can be designed for a distributed big-data environment, from ingestion through monitoring.
2. Compare regression, classification, clustering and generative AI, including suitable applications and limitations.
3. Discuss how feature engineering, temporal splitting and leakage prevention affect model validity.
4. Explain why fairness cannot be reduced to removing a protected attribute or selecting one metric.
5. Discuss a responsible governance framework for deploying a generative-AI assistant in an organisation.

Analytical / Application Questions

1. A fraud model identifies 90 fraudulent transactions and misses 10. It also incorrectly flags 30 legitimate transactions. Calculate recall and precision, and explain which additional business information is needed before selecting a threshold.
2. Design a churn-prediction experiment for a subscription service. Specify the target, prediction time, feature windows, data split, evaluation metrics and monitoring plan.
3. A deployed model’s score distribution changes sharply after a pricing-policy change. Analyse possible causes and recommend an evidence-based response rather than automatically retraining the model.

Multiple Choice Questions with Answers

Question 1

Which type of analytics recommends an action by combining predictions, objectives and constraints?

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: D. Prescriptive analytics

Question 2

Which practice most directly prevents a future cancellation code from contaminating a churn model’s training features?

A. Increasing the number of trees

B. Applying an as-of prediction-time rule

C. Using a larger test set

D. Removing all categorical variables

Correct Answer: B. Applying an as-of prediction-time rule

Question 3

A classifier has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 4

Why can accuracy be unsuitable as the only metric for rare-event fraud detection?

A. It cannot be calculated for classification

B. It may appear high even when most fraud cases are missed

C. It always penalises false positives more than false negatives

D. It measures only model latency

Correct Answer: B. It may appear high even when most fraud cases are missed

Question 5

Which statement best describes concept drift?

A. A change in the relationship between inputs and the target

B. A change in the file format only

C. A reduction in hardware capacity

D. A method for encoding categories

Correct Answer: A. A change in the relationship between inputs and the target

Question 6

What is the primary purpose of regularisation?

A. To increase data duplication

B. To penalise excessive model complexity

C. To guarantee causal conclusions

D. To remove the need for validation data

Correct Answer: B. To penalise excessive model complexity

Question 7

Which evaluation design is generally most appropriate for predicting future demand from historical time-series data?

A. Randomly mixing all observations before splitting

B. Using only the training set

C. Chronological training and validation splits

D. Deleting observations from high-demand periods

Correct Answer: C. Chronological training and validation splits

Question 8

Which statement about generative-AI output is most accurate?

A. Fluent output is automatically factual

B. Human review and validation remain necessary for consequential uses

C. Retrieval eliminates all hallucinations

D. Confidential data can always be entered safely

Correct Answer: B. Human review and validation remain necessary for consequential uses

Question 9

A model has precision 0.8 and recall 0.5. What is its F1 score, approximately?

A. 0.31

B. 0.50

C. 0.62

D. 0.80

Correct Answer: C. 0.62

Question 10

Which activity is part of responsible model operations?

A. Disabling logs to reduce storage

B. Monitoring data quality, performance and subgroup impacts

C. Retraining after every single prediction

D. Selecting the model only by training accuracy

Correct Answer: B. Monitoring data quality, performance and subgroup impacts