Temporal, Textual and Other Complex Data

Introduction

Data preparation is most difficult when observations are not independent rows of a rectangular table. Time-series observations are ordered and often dependent; text is symbolic, sparse and context-sensitive; geospatial data are governed by coordinate systems and spatial relationships; images are structured arrays with very high dimensionality; and event data record irregular occurrences whose meaning depends on sequence and context. These modalities are therefore not merely alternative file formats. They require different representations, quality controls, analytical assumptions and validation strategies.

A central principle of this chapter is that representation determines what can be learned. Converting a timestamp into an inappropriate integer, removing punctuation from a legal document, mixing coordinate reference systems, or flattening an image without preserving spatial structure can destroy information before modelling begins. Preparation must consequently be treated as an analytical design activity rather than routine cleaning.

Learning Objectives

By the end of this chapter, a learner should be able to:

• distinguish the structural properties of temporal, textual, geospatial, image and event data;
• align, resample and aggregate time-series data while avoiding temporal leakage;
• diagnose trend, seasonality, autocorrelation and missingness in temporal data;
• select appropriate tokenization, normalization and vectorization strategies for text;
• manage coordinates, projections, spatial joins and location effects;
• prepare images using modality-appropriate normalization and augmentation;
• represent event streams using sequences, windows, intervals and aggregates;
• address high dimensionality, sparsity, leakage, bias, privacy and reproducibility;
• design quality controls and validation schemes appropriate to each modality.

1. Complex Data as Structured Observation

A conventional tabular record often represents a case by a vector of attributes. Complex data contain additional structure:

Complex-data preparation has four recurring questions:

1. What is the observational unit? A minute, document, location, pixel, transaction or session?
2. What is the ordering or neighbourhood relation? Time, sentence position, physical distance or event sequence?
3. What information is invariant and what information is meaningful? For example, lowercasing may be harmless in some sentiment tasks but harmful when case identifies an acronym.
4. What information was available at the prediction time? This question is essential for preventing leakage.

A useful data contract specifies the unit, timestamp semantics, coordinate reference system, encoding, missing-value rules, permissible transformations and provenance for every derived feature.

2. Temporal Data

2.1 Time, timestamps and intervals

A timestamp identifies an instant, whereas an interval has a start and end. Analysts must distinguish event time from processing time. Event time is when an observation occurred; processing time is when it entered a system. Streaming systems may receive late or out-of-order records, so processing order cannot automatically substitute for event-time order.

Time zones and daylight-saving transitions create additional complications. Store instants in a standard representation such as UTC, while retaining the original time zone when local calendar interpretation matters. A date such as “2025-10-26 01:30” may occur twice in a daylight-saving transition in some jurisdictions. A naive timestamp without offset information is therefore potentially ambiguous.

The sampling interval should also be documented. A value labelled daily may represent an instantaneous measurement, an end-of-day value, a daily average or a daily total. These meanings affect aggregation. Additive quantities such as sales can usually be summed across periods; rates and temperatures generally require an appropriate mean or weighted mean.

2.2 Alignment and resampling

Temporal alignment places observations from different sources on a common time basis. Suppose demand is recorded hourly and weather every 15 minutes. One valid design may aggregate weather to hourly means before joining it to demand. Another may retain 15-minute weather and associate each demand observation with a window summary. The choice must reflect the causal and operational process.

For an interval [t₀, t₁] with observations xᵢ and weights wᵢ, a weighted mean is:

Here xᵢ is a measurement and wᵢ may represent duration or reliability. The formula assumes weights are non-negative and that the denominator is non-zero. It is appropriate for quantities whose averaging is meaningful, not for additive counts that should be summed.

A temporal join should specify tolerance. For example, a sensor reading may be matched to the most recent calibration record only if the calibration is no more than 30 days old. Unbounded nearest-neighbour joins can silently attach future information to past observations.

2.3 Missingness, irregularity and quality

Missing temporal values have several causes: sensor failure, scheduled downtime, transmission loss, censoring or a value that was never defined. A missing value is not automatically zero. Record missingness indicators and investigate whether missingness depends on time, operating conditions or the target.

Interpolation can be useful for short gaps in smooth physical measurements. Linear interpolation between (t₁, x₁) and (t₂, x₂) estimates a value at t as:

This assumes a locally linear trajectory and should not be used blindly across long gaps, abrupt events or categorical states. Forward filling is reasonable for slowly changing configuration states in some systems, but can create false persistence in measurements.

Quality controls should test monotonicity of timestamps, duplicate keys, impossible intervals, clock drift, gaps, extreme values and consistency between cumulative and incremental measures. A cumulative meter that decreases may indicate a reset rather than a negative consumption event.

2.4 Trend, seasonality and dependence

A time series may contain level, trend, seasonal variation, cycles, shocks and noise. Seasonality is a repeating pattern linked to a known calendar or period, such as weekday effects or annual demand. A cycle may recur but lack a fixed period. These distinctions matter for feature construction and forecasting.

Autocorrelation measures association between observations separated by a lag. For a weakly stationary series with mean μ and variance σ², the lag-k autocorrelation is:

where Cov is covariance and Var is variance. Stationarity here means that relevant distributional properties do not change over time. Strong autocorrelation violates the independence assumption of many ordinary statistical procedures and affects validation design.

Differencing can remove a changing level:

Seasonal differencing at period s is ΔₛXₜ = Xₜ − Xₜ₋ₛ. Differencing may improve stationarity but can also amplify noise and obscure interpretation. It should be justified diagnostically rather than applied automatically.

For a seasonal decomposition, an additive conceptual model is:

where Tₜ is trend, Sₜ is seasonal component and Rₜ is remainder. The additive form assumes seasonal amplitude is approximately constant. A multiplicative model, Xₜ = Tₜ × Sₜ × Rₜ, is more suitable when seasonal variation grows with the level and all components are positive.

2.5 Temporal validation and leakage

Randomly splitting rows is usually invalid for forecasting because future observations may enter the training set. Preferred strategies include chronological holdout, rolling-origin evaluation and expanding-window validation. Every transformation that learns from data—scaling, imputation, feature selection and vocabulary construction—must be fitted using training periods only.

A feature such as “mean sales during the next seven days” is leakage when predicting today’s sales. Even a seemingly harmless rolling mean leaks if its window includes the current target or future records. A past-looking window should be explicitly defined, for example using observations strictly before the prediction timestamp.

3. Textual Data

3.1 Text as data

Text contains multiple levels of structure: characters, tokens, sentences, documents, speakers, discourse sections and collections. Meaning depends on syntax, negation, domain terminology, pragmatics and context. A preprocessing operation that improves one task can damage another. Removing stopwords may reduce dimensionality for document retrieval, but it can harm sentiment analysis when negation words are removed.

Important metadata include language, author or source, document date, version, access permissions and whether the text is generated by a human or a system. Preserve the raw text separately from derived representations so that decisions can be audited.

3.2 Encoding, normalization and tokenization

Verify character encoding before processing. Unicode normalization can reconcile visually equivalent representations, but aggressive normalization may remove meaningful distinctions. Lowercasing, punctuation removal, spelling correction and accent stripping should be task-specific.

Tokenization divides text into units. Word tokenization is interpretable for many applications; subword tokenization handles misspellings, rare words and morphologically rich languages; character tokenization is robust to unknown words but produces longer sequences. Sentence segmentation is itself non-trivial for abbreviations, quotations and multilingual documents.

Stemming truncates words using heuristic rules, whereas lemmatization attempts to map inflected forms to dictionary lemmas using linguistic information. Neither is universally beneficial. In modern language-model pipelines, task-specific subword tokenization often replaces extensive stemming, but domain-specific terms may still require custom handling.

3.3 Vector representations

A bag-of-words representation records token occurrence but ignores order. Term frequency–inverse document frequency (TF–IDF) assigns greater weight to terms frequent in a document but less common across the corpus. One common form is:

where tf(t,d) is the term frequency of term t in document d, df(t) is the number of documents containing t, and N is the number of documents. The added one is a smoothing choice, not a universal requirement. Vocabulary construction and any document-frequency calculation must be performed within the training data to prevent leakage.

Dense embeddings represent words, sentences or documents in a continuous vector space. They can encode semantic similarity but may inherit biases from their training corpora. Contextual embeddings represent a token differently according to surrounding words, helping distinguish meanings such as “bank” as a financial institution or river edge. However, high-dimensional representations increase computational cost and do not guarantee causal or factual understanding.

3.4 Text quality and evaluation

Text pipelines should test empty documents, duplicated documents, encoding errors, boilerplate, markup, language mismatch, personally identifiable information and label contamination. Near-duplicate documents are particularly dangerous when copies appear in both training and test sets.

Evaluation must reflect the intended task. Accuracy can be misleading for imbalanced classification; precision, recall and F1 may be more informative. For information retrieval, ranking metrics such as mean average precision or normalized discounted cumulative gain may be appropriate. For generated text, automatic scores are imperfect proxies for factuality, usefulness, toxicity and stylistic quality; human review and targeted error analysis remain necessary.

4. Geospatial Data

4.1 Location and coordinate reference systems

Geospatial data may describe points, lines, polygons, rasters or networks. A coordinate reference system (CRS) defines how locations relate to the Earth. Geographic coordinates such as latitude and longitude are angular, usually measured in degrees; projected coordinates are planar and commonly measured in metres.

Never calculate ordinary Euclidean distance on longitude and latitude over large areas without considering the Earth’s curvature. A haversine distance between two points with latitudes φ₁, φ₂ and longitudes λ₁, λ₂ in radians is:

Here R is the chosen Earth radius, approximately 6,371 km for a spherical approximation, and d has the same distance unit as R. This approximation is suitable for many moderate-distance applications but not for high-precision surveying.

For buffering, area or distance operations, a locally appropriate projected CRS is often preferable. Reprojection must be documented because it can alter geometry slightly and unsuitable projections can distort area, distance or direction.

4.2 Spatial joins and spatial effects

A spatial join associates records using containment, intersection, nearest distance or another geometric relation. Boundary cases require explicit rules: a point on a polygon boundary may be assigned to one region, both, or neither depending on the operation.

Nearby observations often resemble one another, producing spatial autocorrelation. If ignored, standard errors may be too small and random train-test splits may overstate generalization. Spatial blocking—holding out geographic regions—provides a stronger test of transfer to new locations.

The modifiable areal unit problem (MAUP) occurs when results change with the boundaries or scale of spatial aggregation. The ecological fallacy occurs when relationships observed for areas are incorrectly attributed to individuals. Location can also act as a proxy for protected or sensitive characteristics, so spatial features require fairness and privacy review.

4.3 Geocoding and spatial quality

Geocoding converts addresses into coordinates and can introduce positional uncertainty, duplicate matches and systematic undercoverage. Retain match confidence and the original address. Do not silently replace an uncertain location with a precise-looking coordinate. Sensitive coordinates, such as health or domestic-violence locations, may require aggregation, masking, access control or differential disclosure policies.

5. Image Data

An image is commonly represented as a tensor with height, width and channels. Pixel values may be integers such as 0–255 or normalized floating-point values. Metadata should include sensor, exposure, spatial resolution, colour space, orientation and acquisition conditions.

Preparation commonly includes resizing, cropping, channel conversion, normalization and quality screening. Resizing can alter small objects; cropping can remove the target; compression can introduce artefacts. Normalization must match the model’s training convention. For a pixel value x, min–max scaling to [0,1] is:

where x_min and x_max are defined by the intended scale, such as 0 and 255 for 8-bit channels. If standardization is used, training-set statistics must be applied to validation and test images without refitting.

Augmentation can improve robustness through transformations that preserve the label, such as modest crops or horizontal flips when orientation is irrelevant. It is inappropriate to flip medical images when laterality matters or to alter colours when colour is diagnostic. Augmented versions of one original image must remain in the same data partition; otherwise, near-duplicates create leakage.

Images are high-dimensional, but dimensionality reduction may discard task-relevant detail. Principal component analysis (PCA) finds directions of maximum variance, not necessarily directions that best predict the target. A common computational practice is to centre the feature matrix using training means before fitting PCA, then apply the learned transformation to later data.

6. Event and Sequence Data

Event data record occurrences such as payments, web clicks, machine alarms or patient transitions. A useful event schema includes event identifier, entity identifier, event type, event time, ingestion time, source, attributes and version. Event identity supports deduplication; idempotent processing ensures that replaying a message does not create a second business event.

Events may be represented as ordered sequences, intervals, counts within windows, transition matrices or process graphs. Window features must be anchored to a reference time. For a customer-risk prediction at time t, counts should use events in [t − 30 days, t), excluding events at or after t unless their availability is guaranteed.

Out-of-order and late-arriving events require watermarks, correction procedures or recomputation. A system should distinguish event correction from a new event. Sessionization, which groups events into sessions, requires an explicit inactivity threshold and careful treatment of midnight crossings, device changes and concurrent sessions.

Event data are often subject to censoring: the absence of a recorded event may mean that it did not occur, was not observed or was deleted. Survival and process analyses must therefore distinguish observation windows from true non-occurrence.

7. High Dimensionality and Multimodal Integration

Text vocabularies, image pixels, sensor channels and event histories can produce far more features than observations. High dimensionality increases storage and computation, creates sparse matrices and raises overfitting risk. Useful responses include domain-informed feature selection, regularization, dimensionality reduction, hashing, embedding, aggregation and hierarchical modelling.

Multimodal integration can occur at three levels:

• Early fusion: concatenate representations before modelling. This is simple but requires compatible scales and synchronized observations.
• Intermediate fusion: learn modality-specific representations and combine them in a later layer or model.
• Late fusion: make separate predictions and combine them. This is robust when modalities are missing or arrive at different times.

Alignment is the principal difficulty. A document may describe an event occurring over a period; an image may be captured near but not exactly at a sensor measurement; a location may change while a customer record remains constant. The integration design should preserve provenance and record uncertainty rather than forcing false precision.

8. An Integrated Quality-Control Framework

A professional pipeline should include the following controls:

Data profiling should be stratified by source, time, geography, language and demographic group. Global averages can hide local failures. Quality thresholds should distinguish hard failures, warnings and accepted exceptions, with reasons recorded.

9. Applied Case: Predictive Maintenance for a Distributed Equipment Fleet

This is an illustrative case.

Background

A manufacturer operates pumps at hundreds of geographically distributed sites. Each pump produces vibration and temperature readings, maintenance events and technician notes. The organization wants to predict failures seven days ahead while minimizing unnecessary inspections.

Problem

The data arrive at different frequencies: vibration every minute, temperature every five minutes, maintenance events irregularly and notes after technician visits. Sites use different time zones, some sensors have clock drift, and several pumps have changed sensors during the study period.

Analysis

The team first defines the observational unit as a pump-day and the prediction time as 00:00 local site time. All instants are stored in UTC with the site offset retained. Sensor data are checked for duplicated timestamps, impossible temperatures, flatlined channels and gaps. Maintenance records are assigned to pumps using stable asset identifiers rather than sensor identifiers.

Features are computed only from data before the prediction time. Vibration is summarized by robust statistics and the proportion of threshold exceedances; temperature is summarized using duration-weighted averages because intervals are irregular. A failure label is created when a confirmed failure occurs in the following seven days, with censoring rules for pumps leaving service.

Technician notes are tokenized using a domain vocabulary. Boilerplate signatures are removed, but negations and component names are retained. The text representation is fitted only on the training period. Site coordinates are used to create spatially blocked evaluation groups, not as an unrestricted proxy for operating conditions.

Approach and solution

The validation design uses rolling temporal splits and holds out entire sites in a secondary test. Sensor, text and event representations are first evaluated separately, then combined using late fusion so that missing notes do not eliminate a prediction. Thresholds are selected using training data and reviewed against inspection capacity and unequal false-negative costs.

Outcome

The resulting evaluation estimates performance on later periods and unfamiliar sites more realistically than a random row split. Error analysis shows that failures after sensor replacement are underdetected, leading to a new sensor-lineage feature and a monitoring rule for post-replacement uncertainty. The organization adopts a human-in-the-loop workflow: predictions prioritize inspections, but technicians make the final intervention decision.

Learning points

The case demonstrates that temporal alignment, event identity, text handling, spatial validation and governance are interdependent. The most sophisticated model cannot repair an incorrect prediction time, duplicated maintenance event or leakage from post-failure notes.

10. Ethics, Governance and Contemporary Practice

Complex data often contain personal, sensitive or inferential information. Text may reveal health, political or religious attributes; location traces can expose routines; images may identify people; event streams can support detailed behavioural profiling. Apply purpose limitation, data minimization, access control, retention limits, encryption and documented lawful basis. De-identification is not automatically irreversible, especially when locations and timestamps can be linked with external data.

Fairness assessment should examine performance and missingness across relevant groups and locations. Removing a protected attribute does not remove proxy information. Human review is necessary when automated outputs affect employment, credit, healthcare, policing or access to services.

Contemporary systems increasingly use foundation models, vector databases, multimodal encoders and streaming feature stores. These technologies improve representation and retrieval but create new risks: opaque training data, prompt or retrieval leakage, copyright concerns, model inversion, hallucination and distribution shift. Store model and embedding versions, document retrieval sources, monitor drift and evaluate against adversarial and out-of-distribution cases.

Best practice is not to seek one universal preprocessing recipe. Instead, construct a documented, testable and reversible pipeline in which every transformation has a reason, a fitted-data boundary, a validation test and an owner.

Chapter Summary

Temporal, textual, geospatial, image and event data contain structure that ordinary tabular methods may destroy. Temporal preparation requires correct timestamp semantics, alignment, resampling, missingness treatment, seasonality analysis and time-safe validation. Text preparation requires careful encoding, tokenization, normalization and representation choices, with attention to context, vocabulary leakage and bias. Geospatial analysis depends on coordinate reference systems, appropriate distance calculations, spatial joins and geographic validation. Image preparation must preserve spatial and semantic information while controlling scale, normalization, augmentation and leakage. Event data require durable identity, ordering, windows, late-event handling and censoring awareness.

Across modalities, quality control involves validity, completeness, provenance, alignment, leakage prevention, representational adequacy and governance. High-dimensional and multimodal data are manageable when representations are selected according to the analytical task and when deployment conditions guide validation. Ethical practice requires privacy protection, fairness analysis, transparency and human oversight.

Key Terms

Temporal alignment: Matching observations from different sources to a common time basis.

Event time: The time at which an event actually occurred.

Processing time: The time at which a system received or processed an event.

Resampling: Converting observations to a different temporal frequency.

Seasonality: A repeating pattern associated with a known period.

Autocorrelation: Association between observations separated by a time lag.

Temporal leakage: Use of future or unavailable information in a prediction.

Tokenization: Dividing text into computational units.

TF–IDF: A term-weighting method combining within-document frequency and corpus rarity.

Embedding: A numerical vector representation intended to encode useful relationships.

Coordinate reference system: A specification for representing geographic locations.

Spatial join: Associating records according to a geometric relationship.

Spatial autocorrelation: Dependence among observations based on geographic proximity.

Modifiable areal unit problem: Sensitivity of results to spatial aggregation boundaries or scale.

Raster: A spatial grid of cells or pixels.

Augmentation: Label-preserving transformation used to expand or diversify training data.

Event deduplication: Identification and removal or reconciliation of repeated event records.

Sessionization: Grouping related events into sessions using defined rules.

Late-arriving event: An event received after the expected processing order or time.

Multimodal fusion: Combining representations or predictions from different data modalities.

Review Questions

Short Answer Questions

1. Why should event time and processing time be stored separately?
2. Distinguish interpolation from forward filling in a time-series pipeline.
3. What information is lost by a bag-of-words representation?
4. Why is a projected CRS often preferred for local distance calculations?
5. Give two reasons why image augmentation can be harmful.

Long Answer Questions

1. Explain how trend, seasonality, autocorrelation and differencing relate to time-series preparation. Include assumptions and risks.
2. Compare word, subword and character tokenization, and discuss when each may be appropriate.
3. Describe the consequences of using an unsuitable coordinate reference system in geospatial analysis.
4. Explain how spatial and temporal validation differ from random row splitting and why they are important.
5. Design a quality-control framework for an event-stream pipeline, including identity, ordering, missingness and governance.

Analytical / Application Questions

1. A forecasting model uses a seven-day rolling average that includes the current day’s target. Identify the leakage and redesign the feature.
2. A health study joins patient addresses to neighbourhood outcomes and reports strong associations. Explain at least two possible spatial or ethical problems and propose controls.
3. Design a multimodal predictive-maintenance pipeline using sensor, text, location and event data. Specify the observational unit, prediction time, representations, validation strategy and principal risks.

Multiple Choice Questions with Answers

Question 1

Which validation design is generally most appropriate for forecasting future observations?

A. Randomly splitting individual rows

B. Chronological or rolling-origin validation

C. Splitting tokens at random

D. Duplicating rare future observations in training

Correct Answer: B. Chronological or rolling-origin validation

Question 2

In a temporal join, why is an explicit matching tolerance important?

A. It guarantees that all values are non-missing

B. It prevents every numerical variable from being standardized

C. It prevents implausibly distant or future records from being matched

D. It converts event time into processing time

Correct Answer: C. It prevents implausibly distant or future records from being matched

Question 3

What does the inverse-document-frequency component of TF–IDF primarily do?

A. Increases the weight of terms common in every document

B. Reduces the weight of terms appearing in many documents

C. Preserves the exact word order of a document

D. Converts text directly into an image

Correct Answer: B. Reduces the weight of terms appearing in many documents

Question 4

Why should longitude and latitude not generally be treated as ordinary planar coordinates for large-distance calculations?

A. They are usually measured in angular degrees and reflect a curved Earth

B. They cannot be stored digitally

C. They contain no information about location

D. They are always categorical variables

Correct Answer: A. They are usually measured in angular degrees and reflect a curved Earth

Question 5

Which issue is an example of the modifiable areal unit problem?

A. A pixel value being stored as an integer

B. A result changing when geographic boundaries or aggregation scale change

C. A document containing two languages

D. An event arriving after its processing deadline

Correct Answer: B. A result changing when geographic boundaries or aggregation scale change

Question 6

A training image and a lightly augmented copy are placed in different data partitions. What is the principal risk?

A. Underflow in pixel normalization

B. Spatial autocorrelation in a map

C. Data leakage through near-duplicate images

D. Loss of all image metadata

Correct Answer: C. Data leakage through near-duplicate images

Question 7

For a prediction made at time t, which event window is normally safest when future information is unavailable?

A. [t, t + 30 days]

B. [t − 30 days, t)

C. [t − 30 days, t + 7 days]

D. All events in the complete dataset

Correct Answer: B. [t − 30 days, t)

Question 8

What is the main purpose of retaining stable event identifiers?

A. To make all event timestamps identical

B. To support deduplication and idempotent processing

C. To remove the need for access controls

D. To convert events into geographic polygons

Correct Answer: B. To support deduplication and idempotent processing

Question 9

Which statement about PCA is correct?

A. It always finds the features most predictive of the target

B. It finds directions of maximum variance in the input data

C. It eliminates the need for validation

D. It is applicable only to text documents

Correct Answer: B. It finds directions of maximum variance in the input data

Question 10

Which practice best reduces leakage when fitting a text vectorizer?

A. Build its vocabulary using the complete dataset before splitting

B. Fit it on training data and apply it unchanged to held-out data

C. Fit a separate vocabulary using the future test labels

D. Remove all rare words after evaluating the test set

Correct Answer: B. Fit it on training data and apply it unchanged to held-out data