Mining Complex and Dynamic Data

Introduction

Traditional data mining assumes that data is tabular, relatively stable, and available for batch processing. Modern information systems violate all three assumptions. Documents, Web pages, social networks, maps, sensor streams, transaction sequences, and event logs contain structure that is not represented adequately by rows and columns. Furthermore, many applications generate data continuously while the underlying process changes over time.

Mining complex and dynamic data therefore requires two complementary capabilities: structural analysis, which exploits relationships among objects, and temporal adaptation, which detects and responds to change. A fraud-detection system may need to combine transaction sequences, customer attributes, device graphs, and a continuously changing stream. A search engine must interpret text, hyperlinks, user behaviour, and changing terminology. A smart-city platform must analyze spatial observations and time-dependent sensor readings.

This chapter examines major forms of complex data, their representations, suitable mining methods, and the challenges created by scale, velocity, heterogeneity, privacy, and concept drift.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish text, Web, graph, spatial, temporal, sequence, and streaming data.
2. Select representations appropriate to each data type.
3. Explain topic modelling, link analysis, spatial analysis, time-series methods, sequence mining, and stream mining.
4. Interpret important measures and assumptions used by these methods.
5. Design a pipeline for mining evolving data.
6. Analyze concept drift, delayed labels, imbalance, and ethical risks.
7. Evaluate complex-data systems using technically and operationally meaningful criteria.

1. What Makes Data Complex and Dynamic?

Data is complex when its meaning depends on multiple attributes, relationships, structures, locations, or modalities. It is dynamic when observations arrive over time or when the data-generating process changes.

Important dimensions include:

• High dimensionality: documents may contain thousands of terms and graphs may contain many possible links.
• Heterogeneity: numerical, categorical, textual, spatial, and multimedia data may coexist.
• Relational structure: the relationship between objects can be more informative than the objects themselves.
• Temporal dependence: current events may depend on previous events.
• Continuous arrival: data may be processed before it can be stored in full.
• Nonstationarity: the distribution of inputs or the relationship between inputs and outcomes changes.

A useful abstraction is a sequence of observations (x_t, y_t), where x_t is the input at time t and y_t is an outcome or label when available. In a stationary setting, the joint distribution remains approximately constant. In a dynamic setting, it may become P_t(X,Y), explicitly indicating dependence on time.

2. Text Mining

2.1 Representation

Text is unstructured at the surface but has linguistic structure. A basic document-term matrix represents document d_i by the weights of terms w_j. The common TF–IDF representation is:

Here, TF(t,d) is the frequency of term t in document d, DF(t) is the number of documents containing the term, and N is the total number of documents. The logarithm reduces the influence of terms occurring very frequently across a document collection. The formula assumes that term occurrence is a useful approximation to document meaning; it does not understand syntax, negation, irony, or context by itself.

Preprocessing may include tokenization, normalization, stop-word handling, stemming or lemmatization, and detection of named entities. These operations are not universally beneficial. Removing a word such as “not” can reverse sentiment, and aggressive stemming can merge terms that should remain distinct.

For two document vectors a and b, cosine similarity is:

It measures the angle between nonzero vectors and is independent of document length under normalization. It is appropriate for comparing vector representations but does not establish semantic equivalence.

Modern systems often use dense embeddings produced by language models. Embeddings can capture contextual similarity more effectively than TF–IDF, but they may encode social bias, be difficult to interpret, and change when the model or prompt changes.

2.2 Topic and Classification Methods

Topic modelling identifies recurring latent themes. In latent Dirichlet allocation (LDA), a document is treated as a mixture of topics, and each topic is a probability distribution over words. If θ_d denotes the topic mixture for document d and φ_k denotes the word distribution for topic k, then a generated word is associated with a topic sampled from θ_d and a word sampled from φ_k. LDA is a generative probabilistic model, not a guarantee that discovered topics correspond to human categories.

Topic quality should be examined using coherence, stability across runs, held-out likelihood where appropriate, and human interpretation. A topic with high statistical likelihood may still be meaningless to domain experts.

Text classification assigns documents to predefined classes such as spam, complaint, or legal category. Models include multinomial naïve Bayes, linear classifiers, tree-based methods, and neural encoders. Evaluation must account for class imbalance. Accuracy can be misleading when one class dominates; precision, recall, F1-score, and cost-sensitive measures are often more informative.

3. Web Mining

Web mining applies mining methods to Web content, structure, and usage.

• Content mining analyzes text, images, and metadata.
• Structure mining analyzes hyperlinks and Web-page relationships.
• Usage mining analyzes clickstreams, sessions, queries, and navigation paths.

The Web can be represented as a directed graph in which pages are vertices and hyperlinks are directed edges. Link analysis exploits the idea that a page is important not merely because it has many incoming links, but because important pages may link to it.

The PageRank-style recurrence is:

Here, r_i is the score of page i, N is the number of pages, B_i is the set of pages linking to i, L_j is the number of outgoing links from page j, and d is a damping factor between 0 and 1. The first term represents random teleportation. The formula assumes a relatively stable link graph and can be manipulated through link farms; therefore, ranking is not equivalent to truth, quality, or safety.

Web usage mining commonly constructs sessions from timestamped requests. Sessionization is uncertain when users share devices, use multiple devices, or omit identifying information. Recommendations should therefore be evaluated not only by click-through rate but also by relevance, diversity, long-term satisfaction, and potential exposure to harmful content.

4. Graph and Network Mining

A graph is represented as G=(V,E), where V is a set of vertices and E is a set of edges. Edges may be directed or undirected, weighted or unweighted, and static or time-stamped.

Important graph-mining tasks include:

• Centrality: identifying influential or structurally important vertices.
• Community detection: finding densely connected groups.
• Link prediction: estimating whether an unobserved relationship will appear.
• Graph classification: assigning a label to an entire graph or a vertex.
• Anomaly detection: locating unusual subgraphs or interactions.

Degree centrality counts incident edges. In a directed graph, in-degree and out-degree must be distinguished. Betweenness centrality measures how often a vertex lies on shortest paths between other vertices; it can identify brokers but is computationally expensive on large graphs. Eigenvector-based centrality gives greater weight to connections with important vertices.

A simple link-prediction score is common-neighbour similarity:

where Γ(u) is the neighbourhood of vertex u. This score is useful for some social networks but ignores edge direction, time, attributes, and the possibility that high-degree vertices naturally share many neighbours.

Graph neural networks aggregate information from neighbouring vertices. A generic layer can be written as:

Here, h_v^(l) is the representation of vertex v at layer l, N(v) is its neighbourhood, AGG is an aggregation function, W_l is a learned parameter matrix, and σ is an activation function. The method assumes that neighbouring vertices provide useful information. It can perform poorly when graphs are heterophilous, rapidly changing, very large, or affected by missing and adversarial links.

5. Spatial Data Mining

Spatial data includes geometry, location, distance, direction, topology, and spatial context. Examples include points representing incidents, polygons representing land parcels, and raster cells representing temperature.

Spatial autocorrelation means that nearby observations may be more similar than distant observations. A common global measure is Moran’s I:

Here, n is the number of observations, x_i is the value at location i, x̄ is the mean, w_ij is a spatial-weight value, and W=∑_i∑_j w_ij. The weights must be defined in advance, for example by adjacency or distance. Interpretation depends on the null model and spatial scale; a positive value generally indicates clustering of similar values, while a negative value suggests spatial dispersion.

Spatial mining tasks include hotspot detection, spatial clustering, co-location discovery, interpolation, and spatial regression. A critical issue is the modifiable areal unit problem: results can change when boundaries or aggregation levels change. Distance-based analysis must also respect the coordinate system; Euclidean distance in degrees of latitude and longitude is not generally an accurate physical distance over large areas.

6. Temporal and Sequence Mining

Temporal data records values indexed by time, whereas sequence data records ordered events. A time series may be regular, such as hourly temperature, or irregular, such as medical visits. A sequence may be ⟨A,B,C⟩, where order matters even if the intervals are not equal.

A decomposition of a series is often expressed as:

for an additive model, where T_t is trend, S_t is seasonal variation, and R_t is the remainder. This model assumes that components combine additively. A multiplicative model, y_t=T_t×S_t×R_t, is more appropriate when seasonal variation grows with the level of the series.

Autoregressive models use previous observations. An AR(p) model is:

where c is a constant, φ_i are coefficients, ε_t is an error term, and p is the lag order. Standard assumptions may include stable relationships, adequately modelled residuals, and suitable treatment of trend and seasonality. Forecast accuracy should be assessed using time-ordered validation rather than random shuffling, which can leak future information into training.

Sequence mining discovers frequent ordered patterns. A sequence ⟨A,B⟩ occurs in a longer sequence if A appears before B, not necessarily immediately. Support is commonly defined as:

The definition must specify whether repeated occurrences in one sequence count once or multiple times. Sequential-pattern mining can support recommendations and process analysis, but frequent patterns are not necessarily causal or useful.

7. Data Streams and Online Mining

A data stream is an unbounded, potentially high-rate sequence of observations. Stream systems cannot usually store every item or repeatedly scan the entire history. They use windows, summaries, sampling, sketches, and incremental models.

A tumbling window divides data into non-overlapping intervals. A sliding window contains the most recent observations and moves continuously. A landmark window retains data from a fixed start point. Exponential fading gives greater importance to recent observations. For a statistic updated incrementally, a simple exponentially weighted mean is:

where 0<α≤1 controls responsiveness, x_t is the newest observation, and m_t is the updated estimate. Larger α adapts faster but is more sensitive to noise.

A stream-mining architecture normally includes ingestion, event-time handling, stateful processing, feature computation, model scoring, monitoring, and action. Event time is the time at which an event occurred; processing time is when the system handled it. Out-of-order events require watermarks, allowed lateness, or correction mechanisms.

Approximate structures such as Bloom filters, count–min sketches, and reservoir sampling reduce memory usage. Their errors must be measured and communicated. For example, a Bloom filter can produce false positives but, under its standard design, not false negatives for inserted items. Approximation is acceptable only when its error is compatible with the decision risk.

8. Concept Drift and Evolving Patterns

Concept drift occurs when the relationship between observations and targets changes. Let P_t(X,Y) denote the joint distribution at time t. Drift occurs when this distribution changes. A change in P(X) is called virtual drift when the input distribution changes but the predictive relationship may remain stable. A change in P(Y|X) is a real concept drift because the meaning of predictors changes.

Drift may be:

• Abrupt: a sudden change, such as a new fraud strategy.
• Gradual: old and new concepts coexist during transition.
• Incremental: parameters change continuously.
• Recurring: a previous concept returns, such as seasonal demand.

Detection methods compare distributions, error rates, or summary statistics across windows. Detection alone is not adaptation. A reliable system must decide whether to retrain, reweight recent data, maintain multiple models, or request human review.

Performance monitoring should distinguish data drift, concept drift, label delay, and system failure. A falling accuracy may result from delayed ground truth rather than model deterioration. Monitoring should include calibration, subgroup performance, latency, missingness, feature distributions, and action outcomes.

9. Integrated Mining Architecture

A practical pipeline for complex and dynamic data has the following stages:

1. Data acquisition: collect events with timestamps, provenance, and consent status.
2. Representation: convert text, graphs, coordinates, sequences, and signals into suitable structures.
3. Cleaning and alignment: resolve duplicates, missing values, inconsistent time zones, and entity identity.
4. Feature and embedding construction: create interpretable measures or learned representations.
5. Mining and prediction: apply models appropriate to the task and data-generating assumptions.
6. Evaluation: use temporal, spatial, graph-aware, or session-aware validation.
7. Deployment: score data under latency, availability, and cost constraints.
8. Monitoring and adaptation: detect drift, data-quality failures, bias, and degradation.

The central design principle is that validation must reproduce deployment. Randomly splitting observations can place future information in the training set, split linked entities across training and testing, or distribute nearly identical events across both sets.

10. Illustrative Case: Urban Transit Disruption Detection

Background

Consider a fictional metropolitan transit authority that operates buses and trains. It receives vehicle GPS events, station entries, service alerts, passenger messages, weather observations, and a network map.

Problem

The authority wants to detect disruptions quickly, distinguish routine delay from abnormal events, and recommend alternative routes. The data is heterogeneous, streaming, spatial, temporal, and relational.

Analysis

GPS events form time series attached to spatial road segments. Stations and routes form a graph. Passenger messages provide text evidence, but messages may contain slang, duplication, and unverified claims. Weather and planned maintenance produce contextual variables. Labels such as “confirmed disruption” may arrive 20 minutes after the event, creating delayed supervision.

Approach and Solution

A streaming pipeline maintains a sliding window of vehicle delays. It computes robust route-level summaries and compares current values with historical conditions for the same time and day. A graph representation models stations as vertices and route segments as edges. Text messages are embedded and classified into categories such as delay, safety issue, or irrelevant comment. Spatial clustering identifies concentrations of abnormal vehicle positions.

The system combines these signals using a calibrated classifier. It uses event time rather than arrival time, permits late events, and records model version and evidence for every alert. A human operations team confirms high-impact alerts. Performance is evaluated with detection delay, false-alert rate, recall for severe disruptions, and subgroup analysis across neighbourhoods.

Outcome

In an illustrative deployment, the main benefit is not merely higher predictive accuracy. Operators receive earlier, geographically localized alerts with explanations such as “six vehicles on adjacent segments exceed the historical 95th percentile and passenger reports mention stalled service.” The system must still handle GPS outages, coordinated false reports, changes in travel patterns, and seasonal schedules.

Learning Points

1. No single representation captures all relevant evidence.
2. Event-time processing is essential when messages arrive late.
3. Historical baselines must be updated as routes and behaviour change.
4. Human confirmation is valuable for high-consequence decisions.
5. Evaluation must measure operational utility, not only classification scores.

11. Challenges, Ethics, and Governance

Complex-data mining creates technical and social risks.

Privacy: location traces, search histories, messages, and graphs can identify individuals even after direct identifiers are removed. Data minimization, access control, retention limits, aggregation, and privacy-preserving methods are necessary.

Consent and secondary use: data collected for transport operation or communication may not automatically be appropriate for profiling or commercial inference. Purpose limitation and transparent notices are important.

Bias and representation: text and graph data reflect unequal participation. A model trained on online reports may underrepresent people with limited connectivity. Spatial models may reproduce historical over-policing or unequal service provision.

Security: streaming systems are vulnerable to poisoning, fake events, link manipulation, and denial of service. Provenance, authentication, anomaly detection, and incident response should be designed from the beginning.

Explainability and accountability: embeddings and graph neural networks may be difficult to interpret. Explanations should identify influential evidence without implying causation. Decisions with legal, financial, employment, or safety consequences require documented governance and appeal mechanisms.

Reproducibility: dynamic systems need versioned data definitions, code, features, models, thresholds, and monitoring policies. A model cannot be meaningfully audited if its training window and drift responses are unknown.

12. Contemporary Developments and Best Practices

Large language models have expanded text mining through retrieval, summarization, extraction, and semantic search. However, generated output can be factually incorrect, reveal sensitive information, or obscure the source of evidence. Retrieval systems should preserve citations, confidence indicators, and human review for consequential uses.

Multimodal systems combine text, images, graphs, and sensor data. Their main challenge is not only representation learning but also alignment: timestamps, entities, locations, and uncertainty must correspond across modalities. Digital twins and Internet-of-Things platforms increasingly require continuous graph and stream mining.

Best practices include:

• define the decision and cost of errors before selecting a model;
• preserve timestamps, provenance, and data-quality indicators;
• use domain-aware representations rather than indiscriminate preprocessing;
• validate with time, geography, entity, or session splits as appropriate;
• report uncertainty and subgroup performance;
• monitor both inputs and outcomes after deployment;
• plan retraining and rollback before production;
• use approximate algorithms only with quantified error bounds;
• protect privacy throughout collection, storage, modelling, and release;
• retain human oversight where errors can cause significant harm.

Chapter Summary

Complex and dynamic data differs from conventional tabular data because meaning is carried by language, relationships, location, order, and continuous change. Text mining uses representations such as TF–IDF and embeddings, with topic modelling and classification supporting discovery and prediction. Web mining combines content, structure, and usage analysis; link analysis treats pages as graph vertices. Graph mining studies centrality, communities, links, anomalies, and learned neighbourhood representations. Spatial mining accounts for distance, adjacency, and spatial autocorrelation. Temporal and sequence mining model trends, seasonality, dependence, and ordered events. Stream mining uses windows, incremental updates, summaries, and approximate structures to meet memory and latency constraints.

Concept drift requires monitoring changes in input distributions and predictive relationships. Successful systems align representations, validation, deployment, governance, and adaptation. Technical accuracy alone is insufficient: privacy, fairness, security, explainability, provenance, and operational utility must also be evaluated.

Key Terms

Complex data: Data whose meaning depends on multiple structures, relationships, modalities, or contexts.

Dynamic data: Data that arrives over time or whose generating process changes.

TF–IDF: A term-weighting method combining within-document frequency with inverse collection frequency.

Cosine similarity: The normalized dot product used to compare nonzero vectors.

Topic modelling: Discovery of latent themes represented by distributions over words.

Web mining: Mining Web content, hyperlinks, and usage behaviour.

PageRank: A link-analysis method that distributes importance through directed links.

Graph: A structure consisting of vertices and edges.

Centrality: A family of measures describing structural importance in a graph.

Community detection: Identification of groups with relatively dense internal connections.

Spatial autocorrelation: Dependence among observations based on spatial proximity.

Time series: Ordered observations indexed by time.

Sequence mining: Discovery of frequent ordered events or item patterns.

Sliding window: A moving subset containing recent observations.

Data stream: An unbounded or continuously arriving sequence of data.

Concept drift: Change in the distribution or predictive relationship underlying data.

Event time: The time at which an event actually occurred.

Processing time: The time at which a system processes an event.

Data provenance: Recorded information about data origin, transformations, and use.

Embedding: A learned vector representation of an object or structure.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading in imbalanced text classification?
2. Distinguish event time from processing time.
3. What is the purpose of a damping factor in PageRank?
4. Define concept drift and give one example.
5. Why must spatial distance be interpreted with reference to a coordinate system?

Long Answer Questions

1. Explain the representations and mining tasks used in text mining, including the limitations of TF–IDF and embeddings.
2. Compare Web mining and graph mining, showing how hyperlinks support link analysis.
3. Explain additive time-series decomposition and autoregressive modelling, including their assumptions.
4. Describe a complete architecture for mining a high-velocity data stream.
5. Discuss privacy, fairness, security, and accountability issues in complex-data mining.

Analytical / Application Questions

1. Design a validation strategy for a fraud model trained on transaction sequences and a customer graph. Explain why a random train–test split may be inappropriate.
2. A stream model’s accuracy falls after a new payment authentication rule is introduced. Propose a diagnostic process to distinguish data drift, concept drift, delayed labels, and implementation failure.
3. Design an urban-disaster analytics system combining social-media text, road graphs, GPS data, and weather streams. Specify representations, models, evaluation measures, and governance safeguards.

Multiple Choice Questions with Answers

Question 1

Which representation assigns greater weight to a term that is frequent in one document but rare across the collection?

A. One-hot encoding only

B. TF–IDF

C. PageRank

D. Moran’s I

Correct Answer: B. TF–IDF

Question 2

In the PageRank recurrence, what does the damping factor primarily model?

A. The number of words in a page

B. Random teleportation or continuation probability

C. The geographic distance between pages

D. The number of documents in a corpus

Correct Answer: B. Random teleportation or continuation probability

Question 3

Which graph task estimates whether an unobserved relationship will appear?

A. Link prediction

B. Topic segmentation

C. Seasonal decomposition

D. Spatial interpolation

Correct Answer: A. Link prediction

Question 4

What does positive spatial autocorrelation generally indicate?

A. Nearby values tend to be dissimilar

B. Nearby values tend to be similar

C. All observations are independent

D. The coordinate system is invalid

Correct Answer: B. Nearby values tend to be similar

Question 5

In an additive time-series decomposition, which expression is correct?

A. y_t=T_t+S_t+R_t

B. y_t=T_t/S_t/R_t

C. y_t=T_t×S_t×R_t only

D. y_t=T_t−S_t−R_t

Correct Answer: A. y_t=T_t+S_t+R_t

Question 6

What is the main advantage of a sliding window in stream mining?

A. It gives equal weight to all historical observations forever

B. It focuses computation on a recent moving subset

C. It eliminates all out-of-order events

D. It guarantees zero classification error

Correct Answer: B. It focuses computation on a recent moving subset

Question 7

A change in P(X) while P(Y|X) remains stable is commonly called:

A. Virtual drift

B. Label encoding

C. Graph centrality

D. Seasonal decomposition

Correct Answer: A. Virtual drift

Question 8

Why is random shuffling often unsuitable for evaluating time-series forecasts?

A. It always reduces the dataset size

B. It can allow future information to influence training

C. It prevents calculation of precision

D. It removes all seasonal patterns

Correct Answer: B. It can allow future information to influence training

Question 9

Which statement about a standard Bloom filter is correct?

A. It can produce false positives but is designed to avoid false negatives for inserted items

B. It always produces exact counts

C. It stores every original record

D. It detects concept drift automatically

Correct Answer: A. It can produce false positives but is designed to avoid false negatives for inserted items

Question 10

Which practice best supports trustworthy deployment of a dynamic mining system?

A. Disable monitoring after deployment

B. Use only accuracy regardless of class balance

C. Version data, features, models, and drift responses

D. Remove timestamps to simplify processing

Correct Answer: C. Version data, features, models, and drift responses