9. Anomaly Detection, Imbalanced Data and Rare Events

Introduction and Context

Many data-mining problems are not concerned primarily with predicting the most common outcome. They are concerned with identifying the unusual event that may have the greatest operational or financial consequence: a fraudulent transaction, a failing machine, a cyberattack, a defective product, or a patient at elevated clinical risk. Such events are often rare, poorly represented in historical data, and difficult to distinguish from legitimate variation.

This chapter examines two closely related challenges. Anomaly detection seeks observations, sequences, or patterns that differ substantially from expected behaviour. Imbalanced-data modelling addresses situations in which one class is much less frequent than another. Rare-event analytics combines both concerns, but they are not identical. A minority class may be rare yet internally consistent and well defined, whereas an anomaly may not belong to a previously labelled class at all.

The consequences of error are also asymmetric. In fraud detection, allowing a fraudulent payment may be more costly than reviewing a legitimate one. In predictive maintenance, a missed failure may cause production loss, while an unnecessary inspection consumes limited engineering capacity. Consequently, model development must connect statistical performance with costs, operational capacity, human judgement, and governance.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish global, contextual, and collective anomalies.

2. Explain the relationship between anomaly detection, novelty detection, and imbalanced classification.

3. Apply statistical, distance-based, density-based, isolation-based, and reconstruction-based methods.

4. Select suitable resampling, class-weighting, thresholding, and ensemble strategies.

5. Evaluate rare-event models using appropriate metrics and validation designs.

6. Design an end-to-end solution for fraud, cybersecurity, quality, risk, or predictive-maintenance applications.

7. Identify leakage, drift, bias, privacy, and governance risks in rare-event systems.

The Nature of Anomalies and Rare Events

An anomaly is an observation or pattern that is inconsistent with a reference model of normal behaviour. The reference model may be learned from data, specified by experts, or constructed from physical and business rules. An observation is not inherently anomalous; its status depends on the population, time period, features, and purpose of analysis.

A global anomaly is unusual relative to the entire dataset. For example, a transaction of an exceptionally large amount may be globally anomalous. A contextual anomaly is unusual only under particular conditions. A temperature of 30°C may be normal in summer but anomalous for a refrigerated warehouse. Contextual detection therefore requires contextual variables such as time, location, season, operating mode, or customer segment. A collective anomaly consists of a group, sequence, or spatial arrangement that is abnormal even though individual observations may appear normal. A sequence of small login attempts from different locations can constitute a collective cybersecurity anomaly.

A useful distinction is between point anomalies, conditional anomalies, and group anomalies. These terms overlap with global, contextual, and collective anomaly categories but emphasize the unit being assessed. The unit may be a row, an event conditional on context, or a window of events.

In supervised learning, an imbalanced classification problem has unequal class frequencies. Let the positive class represent fraud or failure. If only 0.2 per cent of cases are positive, a classifier that predicts every case as negative achieves 99.8 per cent accuracy while being operationally useless. The relevant question is not whether the model predicts the majority class frequently, but whether it identifies enough valuable positives at an acceptable cost and workload.

Important Terminology

The contamination rate is the assumed or estimated proportion of anomalies in a dataset. A novelty is an observation that differs from a model trained primarily on normal data. Outlier detection generally identifies unusual observations within a dataset, while novelty detection evaluates new observations against a learned normality model. Concept drift occurs when the relationship between inputs and outcomes changes over time. Data drift refers to a change in the distribution of input variables, even if the predictive relationship remains stable.

A decision threshold converts a score or estimated probability into an action. A lower threshold usually increases recall and decreases precision. Precision is the proportion of flagged cases that are truly positive; recall, or sensitivity, is the proportion of actual positives detected. Specificity is the proportion of negatives correctly left unflagged. False-positive rate is the proportion of negatives incorrectly flagged.

A Framework for Detecting Unusual Behaviour

A reliable anomaly solution begins by defining normality and the unit of detection. A useful workflow contains the following stages:

1. Define the event, population, time horizon, and consequences of errors.

2. Establish whether reliable labels exist and how they were generated.

3. Construct features that describe behaviour, context, history, and relationships.

4. Select a detection or classification method.

5. Calibrate scores or probabilities and choose an operational threshold.

6. Evaluate using temporally and operationally realistic validation.

7. Deploy with monitoring, investigation procedures, feedback, and periodic review.

The first stage is often the most difficult. “Unusual” may mean statistically rare, inconsistent with a customer’s prior behaviour, unsafe according to an engineering limit, or suspicious according to a combination of weak signals. A sound project records this definition explicitly rather than allowing it to be hidden inside an algorithm.

Statistical and Distance-Based Methods

For approximately well-behaved numerical data, standardised distance from the centre can provide a simple baseline. A z-score is

`z_i = (x_i - mean(x)) / standard deviation(x)`

and unusually large absolute values may be flagged. This approach is limited when distributions are skewed, variables are correlated, or the data are high dimensional. Robust alternatives use the median and median absolute deviation. The Mahalanobis distance accounts for covariance among variables, but covariance estimates become unstable in high-dimensional settings and can themselves be distorted by anomalies.

Distance-based methods compare an observation with its neighbours. A point whose nearest-neighbour distance is large may be unusual. These methods are intuitive but sensitive to scaling, irrelevant variables, sparse data, and the choice of neighbourhood size. Feature engineering and dimensionality reduction may therefore be as important as the detection algorithm.

Density-Based Detection

Density methods assess whether an observation lies in a region with unusually low local density. Local Outlier Factor (LOF) compares the local density around an observation with the densities around its neighbours. An observation in a sparse region within an otherwise variable dataset can receive a high outlier score. Density-based methods are useful when normal observations form clusters of different densities, although their results depend strongly on neighbourhood parameters.

Clustering can also support detection. Small, isolated clusters may indicate unusual behaviour, but a small cluster is not automatically anomalous. It may represent a legitimate customer segment or a new operating mode. Domain review is required before translating cluster size into a risk decision.

Isolation-Based Methods

Isolation Forest identifies observations that can be separated from others using relatively few random partitions. Anomalies are expected to be easier to isolate because they occur in sparse regions. It handles nonlinear relationships and large datasets effectively, although its score is not automatically a probability of fraud or failure. The contamination setting, feature representation, and sampling strategy influence results.

One-Class and Boundary Methods

One-Class Support Vector Machines learn a boundary around normal observations. New points outside the boundary receive an anomalous classification. Such methods can be effective with carefully selected features but require attention to kernel and regularisation parameters. They may be computationally demanding and sensitive to scaling.

Reconstruction and Representation Models

Autoencoders learn to reconstruct input data. If trained mainly on normal cases, a high reconstruction error may indicate an anomaly. The method is attractive for complex, nonlinear data such as sensor signals, images, or network traffic. However, a sufficiently flexible autoencoder may reconstruct anomalies well, reducing the usefulness of reconstruction error. Neural models also require substantial data, careful regularisation, and interpretable monitoring.

For sequential data, recurrent networks, temporal convolutional models, transformers, or simpler rolling-window features can model expected sequences. A sequence should be assessed in its operational context; an unusual event may be normal during maintenance but suspicious during production.

Supervised, Semi-Supervised and Hybrid Approaches

When trustworthy labels exist, supervised classifiers such as logistic regression, gradient boosting, random forests, or neural networks can estimate the probability of a rare event. Supervised learning usually outperforms unsupervised detection when labels are representative, but labels may be delayed, incomplete, biased toward investigated cases, or contaminated by prior detection rules.

A hybrid design combines a supervised risk model with anomaly scores, rules, graph features, or expert constraints. For example, a fraud system may combine a gradient-boosting probability with a customer-deviation score and a rule identifying impossible travel. Hybrid systems are often more operationally useful because they combine predictive evidence with explanations and controls.

Imbalanced Classification: Beyond Accuracy

Class imbalance affects model fitting, probability interpretation, threshold selection, and evaluation. Standard algorithms may minimise a loss dominated by the majority class. A model can therefore appear accurate while failing to learn the minority class.

The confusion matrix records true positives, false positives, true negatives, and false negatives. Key measures include:

• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)
• Specificity = TN / (TN + FP)
• F1 score = 2(precision × recall) / (precision + recall)

The F1 score is useful when precision and recall are both important, but it assumes that they deserve equal emphasis. In a medical screening or safety application, recall may receive greater weight. In an investigation team with limited capacity, precision may be more important.

The precision–recall curve is generally more informative than the ROC curve when the positive class is very rare. The area under the precision–recall curve reflects performance across thresholds, although it should be interpreted relative to the base rate. A model that ranks positives near the top may be valuable even if its global threshold-free metric is modest.

Expected cost makes the decision problem explicit:

`Expected cost = C_FN × FN + C_FP × FP + C_review × reviewed cases`

where costs may include financial loss, investigation time, customer inconvenience, safety consequences, or regulatory exposure. Thresholds should be selected using this cost structure and operational capacity, not simply by choosing 0.5 or maximising F1.

Resampling Strategies

Random undersampling removes majority examples, reducing computational imbalance but potentially discarding useful variation. Random oversampling duplicates minority cases, which can cause overfitting. SMOTE creates synthetic minority observations by interpolating between minority neighbours. Variants such as Borderline-SMOTE focus on difficult cases, while ADASYN generates more synthetic examples in locally difficult regions.

Synthetic methods require caution. They can create implausible records, blur class boundaries, amplify noise, and produce leakage if applied before train–test splitting. Resampling must occur inside each training fold, never across the entire dataset. For temporal data, synthetic generation must also respect time and causal availability.

Cost-Sensitive Learning and Ensembles

Class weights assign greater penalty to errors on the minority class. This approach preserves the data and is often preferable to aggressive resampling. The weight ratio should reflect both class frequency and business cost; it should not be selected mechanically.

Balanced random forests, easy-ensemble methods, and boosting algorithms can improve minority detection by repeatedly exposing models to informative minority and majority subsets. Nevertheless, the final system still requires probability calibration, threshold selection, and monitoring. A high score is not automatically a reliable probability.

Feature Engineering for Rare Events

Rare-event performance depends heavily on behavioural features. Raw values often provide less information than deviations, rates, and relationships. Useful features include:

• deviation from a customer’s historical mean or usual location;
• transaction velocity over several time windows;
• time since the previous event;
• ratio of current load to rated machine capacity;
• count of failed logins by account, device, or network;
• change-point and rolling volatility measures;
• peer-group comparisons;
• graph connectivity, shared devices, or repeated counterparties;
• missingness patterns and data-quality indicators.

Features must be available at prediction time. Using a later investigation outcome, a post-failure sensor reading, or a manually assigned fraud label creates leakage. Aggregations should be computed using historical data only. For temporal applications, a time-based split is usually more credible than a random split.

Thresholding and Operational Decision Design

A model produces a score; an organisation must decide what to do with that score. A three-zone design is often effective: low-risk cases are automatically approved, high-risk cases are blocked or escalated, and an intermediate zone is sent for review. Thresholds may vary by customer segment, transaction value, risk appetite, or available investigation capacity, but segment-specific thresholds must be governed carefully to avoid unfair treatment.

Threshold selection should consider prevalence, costs, review capacity, service-level agreements, and the consequences of delayed decisions. It should be validated on a holdout period and periodically recalibrated. When prevalence changes, precision can change substantially even if the model’s ranking ability remains stable.

Applications Across Industries

In financial services, anomaly and rare-event models identify card fraud, account takeover, money laundering indicators, and credit default risk. Transaction history, device information, merchant relationships, and velocity features are commonly combined. Human review remains important because false positives can inconvenience legitimate customers.

In cybersecurity, models analyse authentication events, network flows, endpoint activity, and process behaviour. Collective anomalies are especially important: a sequence of individually ordinary actions may indicate lateral movement or data exfiltration.

In manufacturing, sensor data support predictive maintenance and quality control. The challenge is that failures are infrequent, operating modes change, and labels may arrive only after a breakdown. Contextual models should distinguish normal variation across products, shifts, and machine states.

In healthcare and public risk, rare-event models may identify deterioration, adverse events, or high-risk cases. Here, calibration, explainability, clinical oversight, and unequal error impacts are critical. Automated outputs should support professional judgement rather than silently replace it.

Illustrative Case Study: Predictive Maintenance in a Packaging Plant

Background

A packaging plant operates 80 industrial motors. Failure events are uncommon, but an unplanned stoppage can interrupt production and require urgent replacement. The plant has vibration, temperature, current, load, and maintenance-log data collected at five-minute intervals.

Situation and Problem

Only 18 confirmed failures occurred over two years. A standard classifier predicted “no failure” for nearly every observation and achieved high accuracy. Engineers also reported that temperature increases were normal during certain high-load production cycles, making a global threshold unreliable.

Analysis

The data-science team first defined the prediction target as failure within the next 48 hours. They created time-based training, validation, and test periods. Features included rolling means and standard deviations, temperature relative to machine-specific baselines, vibration change over one hour, load-adjusted temperature, time since maintenance, and the number of recent warning signals. Sensor readings collected after a failure began were excluded.

The team compared logistic regression with class weights, a gradient-boosting model, an isolation forest trained on healthy operating periods, and a rule-based engineering baseline. Evaluation used recall, precision, the precision–recall curve, expected maintenance cost, and the number of alerts per week. Random oversampling was tested only within the training folds.

Approach and Solution

The selected design combined a supervised failure-risk model with an anomaly score. A low-risk threshold produced no action, an intermediate threshold generated a maintenance work order for review, and a high-risk threshold triggered immediate inspection. Thresholds were chosen according to the cost of a missed failure, inspection labour, and available weekly capacity.

Engineers reviewed the most influential features and examined sensor plots for high-scoring cases. A monitoring dashboard tracked alert volume, precision by machine, sensor missingness, score distribution, and failures missed. The system was retrained only after investigating whether changes represented genuine drift, sensor problems, or new operating modes.

Outcome

The approach improved detection of early deterioration without treating every high-temperature reading as abnormal. Its principal benefit was not a single metric but a controlled decision process: alerts were prioritised, explanations were available, and maintenance staff could provide feedback. The organisation retained a manual override for safety-critical decisions.

Learning Points

The case demonstrates that rare-event analytics requires contextual features, temporal validation, cost-sensitive thresholds, and cooperation between data scientists and domain experts. It also shows why anomaly scores should support—not automatically replace—engineering judgement.

Challenges, Risks and Common Mistakes

A common mistake is to use accuracy as the primary metric. Another is to apply SMOTE before splitting data, causing information from the test set to influence training. Randomly splitting highly related transactions, users, machines, or time windows can also produce overly optimistic results.

Labels may reflect investigation effort rather than true incidence. If only suspicious customers are examined, uninvestigated fraud remains hidden. Delayed labels create censoring: recent cases may appear negative simply because their outcomes are not yet known. Feedback loops arise when a model determines which cases receive investigation and those investigations become the future training data.

Unsupervised methods can flag legitimate but unfamiliar behaviour. High-dimensional distance measures may become meaningless, and anomaly scores from different models are not directly comparable. Models may also fail under concept drift, adversarial adaptation, sensor failure, or changes in business policy.

Rare-event systems create operational risk when alert volumes exceed human capacity. Excessive false positives produce alert fatigue, while opaque decisions may be difficult to challenge. Model explanations should distinguish correlation from causal evidence and should not encourage investigators to treat a score as proof.

Ethical, Governance and Regulatory Considerations

Data collection and use must respect privacy, purpose limitation, retention rules, and applicable data-protection requirements. Sensitive attributes and proxy variables require particular scrutiny. A fraud or risk model may disproportionately flag a demographic or geographic group because historical data contain structural bias. Fairness assessment should examine error rates, calibration, investigation burden, and outcomes across relevant groups, while recognising that fairness criteria can conflict.

Governance should document data sources, label definitions, feature availability, model version, threshold rationale, performance by segment, and escalation procedures. High-impact decisions require human oversight, auditability, appeal mechanisms, and access controls. Security controls must protect both training data and deployed models, since adversaries may manipulate inputs or probe decision boundaries.

Current Developments and Best Practices

Modern rare-event systems increasingly use graph analytics, self-supervised representation learning, online learning, and multimodal data. Graph methods are valuable where relationships—accounts, devices, merchants, suppliers, or hosts—carry more information than individual records. Streaming architectures support near-real-time scoring, but they also intensify the need for drift detection and resilient feature computation.

Generative AI can assist investigation by summarising evidence or proposing hypotheses, but generated explanations must be grounded in verifiable records. It should not invent evidence or make unreviewed high-impact decisions.

A robust professional practice includes the following:

• define the positive event and error costs before modelling;
• establish a simple, transparent baseline;
• use temporal, group-aware, and leakage-resistant validation;
• engineer contextual and behavioural features;
• perform resampling only within training data;
• compare precision, recall, PR curves, calibration, cost, and workload;
• select thresholds with operational stakeholders;
• provide reason codes and investigation guidance;
• monitor drift, missingness, alert volume, subgroup performance, and delayed outcomes;
• maintain feedback, retraining, documentation, and incident-response procedures.

Chapter Summary

Anomaly detection identifies observations, contexts, or groups that depart from expected behaviour. Global anomalies are unusual overall, contextual anomalies are unusual under particular conditions, and collective anomalies emerge from patterns across multiple events. Detection may use statistical, distance, density, isolation, boundary, reconstruction, supervised, or hybrid methods.

Imbalanced classification requires more than accuracy. Precision, recall, specificity, precision–recall analysis, calibration, expected cost, and operational capacity should guide evaluation and threshold selection. Resampling, class weighting, cost-sensitive learning, and ensembles are useful, but they must be applied without leakage. Behavioural and contextual feature engineering is often decisive.

Rare-event solutions are socio-technical systems. Their success depends on realistic validation, human investigation, monitoring, governance, privacy protection, fairness analysis, and adaptation to drift. In practical settings, the best model is not necessarily the most complex model; it is the one that produces reliable, interpretable, and actionable decisions under real constraints.

Key Terms

Anomaly: An observation or pattern inconsistent with a reference model of expected behaviour.

Global anomaly: An observation unusual relative to the entire population or dataset.

Contextual anomaly: An observation unusual only under a particular context, such as time, location, or operating mode.

Collective anomaly: A group or sequence whose combined pattern is abnormal.

Point anomaly: A single observation treated as the unit of anomaly analysis.

Contamination rate: The assumed or estimated proportion of anomalies in a dataset.

Novelty detection: Detection of new observations that differ from a model of normal data.

Outlier detection: Identification of observations that are unusual within an analysed dataset.

Class imbalance: A substantial difference in the frequency of classes in a classification problem.

Precision: The proportion of flagged positive cases that are truly positive.

Recall: The proportion of actual positive cases that are detected.

Specificity: The proportion of actual negative cases correctly classified as negative.

Decision threshold: A score or probability boundary used to determine an action or class.

SMOTE: A resampling method that creates synthetic minority observations by interpolation.

Cost-sensitive learning: Training that assigns different penalties to different types of error.

Concept drift: A change over time in the relationship between predictors and the target.

Data leakage: Use of information during training that would not be available at prediction time.

Calibration: Agreement between predicted probabilities and observed event frequencies.

Alert fatigue: Reduced attention or responsiveness caused by excessive or low-value alerts.

Expected cost: The estimated aggregate consequence of false negatives, false positives, and related actions.

Review Questions

Short Answer Questions

1. What distinguishes a contextual anomaly from a global anomaly?

2. Why can accuracy be misleading in an imbalanced classification problem?

3. Define precision and recall.

4. What is the purpose of a decision threshold?

5. Why must SMOTE be applied only within training data?

Long Answer Questions

1. Compare statistical, density-based, isolation-based, and reconstruction-based anomaly detection methods.

2. Explain how class weighting, undersampling, oversampling, and SMOTE address class imbalance, including their risks.

3. Discuss how temporal validation and feature availability protect a rare-event project from leakage.

4. Explain how expected cost and operational capacity should influence threshold selection.

5. Describe the governance and ethical requirements for a high-impact fraud or risk model.

Analytical / Application Questions

1. A cybersecurity team observes that individual login events appear normal, but a sequence of logins across several countries is unusual. Identify the anomaly type and propose suitable features and evaluation methods.

2. A fraud model has 95 per cent recall but only 3 per cent precision, while investigators can review only 500 cases daily. Analyse the problem and recommend thresholding and modelling changes.

3. Design an end-to-end predictive-maintenance solution for a machine population with few confirmed failures, specifying labels, features, validation, metrics, human actions, and monitoring.

22. Multiple Choice Questions with Answers

Question 1

A transaction is normal for a customer during business hours but highly unusual at 3 a.m. Which type of anomaly is this?

A. Global anomaly

B. Contextual anomaly

C. Collective anomaly

D. Sampling anomaly

Correct Answer: B. Contextual anomaly

Question 2

Why is accuracy usually unsuitable as the main metric when fraud represents 0.2 per cent of transactions?

A. Accuracy cannot be computed from a confusion matrix

B. Accuracy ignores true negatives

C. A majority-class model can obtain high accuracy while detecting almost no fraud

D. Accuracy always increases when recall decreases

Correct Answer: C. A majority-class model can obtain high accuracy while detecting almost no fraud

Question 3

Which metric answers the question: “Of the cases flagged by the model, what proportion are truly positive?”

A. Recall

B. Specificity

C. Precision

D. False-positive rate

Correct Answer: C. Precision

Question 4

Which practice prevents synthetic minority observations from contaminating the test set?

A. Apply SMOTE before the train–test split

B. Apply SMOTE separately within each training fold

C. Apply SMOTE only to the final predictions

D. Apply SMOTE to the majority class

Correct Answer: B. Apply SMOTE separately within each training fold

Question 5

A machine’s high temperature is normal at high load but abnormal at low load. Which feature would best capture this distinction?

A. A raw global temperature threshold

B. A random row identifier

C. Load-adjusted temperature

D. The machine’s record number

Correct Answer: C. Load-adjusted temperature

Question 6

Which method identifies observations that can be separated using relatively few random partitions?

A. Isolation Forest

B. Logistic regression

C. Random oversampling

D. Mahalanobis standardisation

Correct Answer: A. Isolation Forest

Question 7

When the positive class is extremely rare, which evaluation tool is generally more informative for assessing ranking across thresholds?

A. Precision–recall curve

B. Training accuracy only

C. A histogram of the majority class

D. Mean squared error alone

Correct Answer: A. Precision–recall curve

Question 8

An organisation can investigate only 1,000 alerts per day. What should primarily guide the final alert threshold?

A. A default probability threshold of 0.5

B. The model’s training accuracy

C. Investigation capacity, error costs, and expected value

D. The number of input features

Correct Answer: C. Investigation capacity, error costs, and expected value

Question 9

Which situation is an example of collective anomaly detection?

A. One unusually large payment

B. One temperature reading above a fixed limit

C. A sequence of individually ordinary actions indicating possible lateral movement

D. A missing value in a customer record

Correct Answer: C. A sequence of individually ordinary actions indicating possible lateral movement

Question 10

Why can random splitting produce overly optimistic performance for predictive maintenance data?

A. It removes all minority observations

B. Related observations from the same machine or future period may appear in both training and test sets

C. It prevents the use of precision

D. It guarantees concept drift

Correct Answer: B. Related observations from the same machine or future period may appear in both training and test sets