Time Series Data Management and Quality

Introduction

Forecasting quality is constrained by data quality. A sophisticated forecasting model cannot compensate for timestamps that are misinterpreted, observations aggregated at the wrong frequency, values revised after the forecast date, or future information accidentally included in training data. In time series work, data preparation is therefore not a preliminary clerical task; it is part of statistical modelling.

A time series is an ordered collection of observations indexed by time. Let y_t denote the value observed at time t. The index may represent an instant, such as 2025-03-01 14:35:00 UTC, or an interval, such as sales during March 2025. This distinction is fundamental. A point-in-time measurement and an interval total do not have the same aggregation rules, missing-value meaning, or forecasting interpretation.

This chapter develops a framework for creating analysis-ready temporal data. It addresses time representation, frequency, aggregation, alignment, calendars, time zones, missing observations, outliers, revisions, censoring, structural breaks, leakage, and the difference between real-time and revised datasets.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish event time, observation time, publication time, and processing time.
2. Construct a reliable time index with appropriate frequency, calendar, and time-zone rules.
3. Aggregate and align temporal data without changing the meaning of the variables.
4. Diagnose and treat missing values, outliers, censoring, revisions, and structural breaks.
5. Detect temporal leakage and design realistic training and validation datasets.
6. Explain the difference between revised historical data and the information available in real time.
7. Establish governance controls for reproducible, auditable forecasting data.

1. The Temporal Data Model

1.1 Time is more than a column

A timestamp is not merely a label. It determines the ordering of observations, the period to which an observation belongs, and the information available when a forecast is made. A useful temporal record often contains at least four times:

For forecasting, publication time is often as important as event time. A revised inflation value may describe January, but if the revision was published in March, it must not be used to train a model intended to reproduce a January forecast made in February.

1.2 Instants, intervals, and durations

An instant identifies a point on a time scale. An interval has a beginning and an end. A duration is the elapsed time between two instants. These concepts should not be conflated. “Revenue at 31 March” may mean a balance measured at an instant, whereas “revenue for March” is an interval total.

A time interval is commonly represented as half-open:

Here, a is the inclusive start and b is the exclusive end. Half-open intervals prevent double counting when adjacent periods are joined: [a,b) and [b,c) share no instant and cover [a,c) together.

1.3 Granularity and frequency

Granularity is the precision at which data are recorded; frequency is the regular spacing of observations. A dataset can have minute-level granularity but irregular frequency if some minutes are absent. Conversely, daily data may be regularly spaced in calendar time while representing only business days.

Common frequencies include tick, second, minute, hour, day, week, month, quarter, and year. The correct frequency depends on the decision. Hourly data may be appropriate for staffing, but excessive for annual capital planning. Higher frequency is not automatically better: it can increase noise, storage cost, missingness, and computational burden.

2. Timestamps, Calendars, and Time Zones

2.1 Canonical timestamp representation

A robust pipeline should retain the original timestamp and create a canonical representation. A recommended design is:

• preserve the raw source value;
• record the source time zone and parsing rule;
• convert an operational timestamp to a canonical standard, commonly UTC;
• retain a local-time attribute when local business rules matter;
• record data-quality flags for ambiguous or invalid timestamps.

A timestamp without a time zone is not necessarily wrong, but its interpretation must be documented. “2025-10-26 01:30” occurs twice in some regions when daylight saving time ends. A system that stores only the local clock time cannot determine which occurrence was intended.

2.2 Calendar systems

Calendar choice affects period boundaries, seasonality, and aggregation. Relevant calendars include:

• Gregorian calendar;
• fiscal calendars, such as a year ending in June;
• ISO weeks, in which weeks begin on Monday and week 1 is defined by ISO rules;
• retail calendars, including 4–4–5 structures;
• trading calendars with exchange holidays and early closures;
• school, energy, or operational calendars.

A “week” should therefore be specified as a calendar object, not assumed to mean seven observations. A monthly average based on daily values may differ depending on whether non-trading days are excluded, treated as zero, or carried forward.

2.3 Daylight saving time

Daylight saving changes create 23-hour or 25-hour local days. Aggregation in local time can therefore produce unequal-duration periods. For physical rates, an hourly total should account for elapsed duration. If r_i is a rate and ∆t_i is the duration represented by observation i, the interval quantity is:

The formula applies when r_i is measured per unit time and the rate is appropriately representative over ∆t_i. Units are preserved: quantity per hour multiplied by hours gives quantity. For counts already representing completed events, duration weighting is generally inappropriate.

3. Aggregation and Alignment

3.1 Aggregation must follow measurement meaning

Different variable types require different aggregation operators.

If daily sales are s_d, monthly sales are usually:

where m is the set of valid days belonging to month m. This assumes the daily observations are interval flows and that all relevant days are represented. Missing days must be investigated before summing.

For a weighted average rate:

where w_i is an exposure or volume weight. The formula is applicable when the weighted average has a meaningful denominator and the weights are measured consistently.

3.2 Temporal alignment

Alignment places observations from multiple sources on a common time axis. Suppose demand is recorded hourly, temperature every 15 minutes, and promotions by campaign interval. Alignment requires explicit decisions:

1. What period does each value describe?
2. Is the value known at the forecast origin?
3. Should the variable be summed, averaged, carried forward, or joined by interval overlap?
4. What happens when periods only partially overlap?

A careless database join can create duplicated rows and inflate totals. Joining a daily table to a transaction table on date, for example, repeats the daily value for every transaction. The solution may be to aggregate transactions first, or to use a carefully defined many-to-one relationship.

3.3 Resampling direction

Downsampling converts high-frequency data to lower frequency and requires aggregation. Upsampling creates a finer grid and requires a rule for assigning values. Forward filling a monthly policy rate into daily records may be defensible if the rate remains effective until changed. Forward filling daily demand is generally not defensible because it invents demand on days without observations.

Interpolation is appropriate only when the variable is plausibly continuous and the missing interval is short relative to its dynamics. Linear interpolation between values y_a and y_b at time t is:

This assumes a meaningful numerical scale, a sufficiently smooth path, and no unobserved event causing a discontinuity. It should not be used automatically for sales, defaults, or categorical states.

4. Missing Values and Data Availability

4.1 Missingness is a business signal

A missing observation can mean sensor failure, a closed store, a non-trading day, a value not yet published, a value below a reporting threshold, or a genuine zero. These meanings are operationally different.

A useful classification is:

• structural missingness: the observation is not defined, such as a weekend market closure;
• technical missingness: the value should exist but was not captured;
• reporting delay: the value exists but is not yet available;
• censored or suppressed value: the value is known only to lie within a range;
• unknown missingness: the cause is not established.

A zero should be recorded as zero only when the process confirms that no activity occurred. Replacing every missing sales value with zero can create artificial collapses in demand.

4.2 Treatment strategies

Possible treatments include deletion, constant replacement, forward or backward filling, interpolation, model-based imputation, and explicit missingness indicators. No strategy is universally correct.

For forecasting, imputation must respect information timing. A missing value at time t may be imputed using observations after t for retrospective analysis, but that would be invalid when simulating what was known at time t. A practical dataset may therefore maintain two versions: a revised analytical series and a real-time historical-vintage series.

4.3 Diagnosing missingness

Missingness should be summarized by time, entity, source, and cause. Useful diagnostics include missingness rates, consecutive-gap length, weekday patterns, and association with outages or operational events. A sudden increase in missing values may itself be a leading indicator of system degradation and should not be silently repaired.

5. Outliers, Censoring, and Anomalous Observations

5.1 Outliers are not automatically errors

An outlier may be a data-entry error, a genuine demand surge, a promotion, a holiday, a supply disruption, or a change in measurement. Removing it without investigation can eliminate the very event the forecast must learn to anticipate.

A robust screening statistic is the median absolute deviation (MAD):

A robust standardized score can be defined as:

The constant 0.6745 makes the score approximately comparable with a standard deviation under a normal distribution. This method assumes MAD is nonzero and is useful for screening, not for proving that an observation is erroneous. Thresholds must be selected in context.

Other approaches include interquartile ranges, control charts, seasonal decomposition, peer comparison, and domain rules. A robust workflow labels observations as confirmed error, unusual but valid, or unresolved, rather than deleting them by default.

5.2 Censoring and truncation

Censoring occurs when the exact value is unavailable but a bound is known. For example, a survey may report income as “more than 100,000,” or a monitoring device may record “at least 500.” Truncation occurs when units outside a range are not observed at all.

Replacing a censored value with the censoring threshold generally biases summaries downward or upward depending on the situation. Censored observations may require specialized statistical models or interval-valued representations. At minimum, the censoring rule should be retained as metadata.

6. Revisions, Vintage Data, and Real-Time Forecasting

6.1 First release is not final truth

Economic, financial, and operational data are often revised. A historical observation can therefore have multiple vintages. Let y_τ^(v) denote the value referring to event time τ as published in vintage v. The event date and the publication vintage are distinct dimensions.

A revised dataset is useful for estimating relationships using the best current information. It is inappropriate for evaluating a historical forecast if the revised values were unavailable at the forecast origin. Such evaluation creates a form of look-ahead bias.

6.2 Two valid research questions

A forecasting study should state whether it asks:

• reconstruction: How well would a model forecast using data as currently revised?
• real-time replication: How well would the model have performed using information actually available then?

Real-time evaluation requires data vintages, release calendars, revision histories, and a precise forecast-origin timestamp. When vintages are unavailable, the limitation should be disclosed rather than hidden.

7. Structural Breaks and Regime Changes

A structural break is a change in the data-generating process, such as a pricing policy, merger, pandemic, regulatory reform, sensor replacement, or market redesign. A historical relationship may not remain valid after the break.

Breaks can affect level, trend, variance, seasonality, or relationships among variables. Diagnostics include plots, rolling statistics, intervention records, forecast-error monitoring, and formal tests such as change-point methods. Statistical tests depend on assumptions and should support, not replace, domain investigation.

Possible responses include intervention variables, segmented models, re-estimation using post-break data, regime-switching models, down-weighting obsolete history, or separate models for distinct regimes. A break should not be treated as an outlier when it represents a lasting change.

8. Leakage and Temporal Validation

8.1 Definition of temporal leakage

Temporal leakage occurs when information unavailable at the forecast origin influences a feature, target, imputation, selection decision, or model estimate. Examples include:

• using a revised economic series in a historical backtest;
• calculating a moving average with future observations;
• normalizing the entire dataset before splitting it;
• using a transaction later cancelled to predict the original decision;
• imputing an early missing value with future data in a real-time simulation;
• joining a feature by event date when it was published later.

Leakage can produce excellent validation results and poor operational performance.

8.2 Forecast-origin logic

For a forecast made at origin T, every predictor must satisfy an availability condition:

The condition applies even if the predictor describes an earlier event. A feature about last month’s sales is unavailable if the reporting system publishes it only after the forecast is made.

Time-series validation should use chronological splits, rolling-origin evaluation, or expanding windows. Random shuffling usually breaks temporal dependence and can place future information in the training set. Feature transformations, scaling, feature selection, and imputation should be fitted separately within each training window.

9. A Data-Quality Framework for Forecasting

An analysis-ready temporal table should be accompanied by a data contract. The contract specifies:

• entity key and grain, such as one store-product-hour;
• event-time definition and time zone;
• interval boundary convention;
• expected frequency and calendar;
• units and aggregation rules;
• allowed ranges and data types;
• publication and revision behavior;
• missingness codes and imputation policy;
• outlier and censoring flags;
• lineage, source system, extraction time, and version.

A practical quality-control sequence is:

1. Schema checks: verify columns, types, keys, and uniqueness.
2. Temporal checks: identify invalid timestamps, duplicates, gaps, overlaps, and out-of-order records.
3. Semantic checks: verify units, sign conventions, interval meaning, and aggregation logic.
4. Availability checks: compare publication time with forecast origin.
5. Distribution checks: monitor levels, variance, seasonality, and unusual changes.
6. Reconciliation checks: compare totals with authoritative financial or operational systems.
7. Provenance checks: record source, transformation, code version, and data vintage.

Data quality is multidimensional. Accuracy, completeness, timeliness, consistency, validity, uniqueness, and lineage should be assessed separately. A complete dataset can still be temporally invalid; a timely dataset can still contain incorrect units.

10. Illustrative Case: Retail Demand Forecasting Across Time Zones

Background

A multinational retailer forecasts hourly demand for stores in London, New York, and Singapore. Point-of-sale transactions are stored in local time, inventory snapshots are recorded in UTC, and promotions are supplied as campaign intervals. The original forecasting pipeline reports unexpectedly high demand during daylight-saving transitions and systematically poor forecasts after promotion launches.

Problem

An investigation finds three issues. First, local timestamps are converted to UTC without recording whether an ambiguous time occurred during the autumn clock change. Second, daily demand is aggregated by UTC date, causing some local business days to be split across two UTC dates. Third, promotion data are joined using campaign start date, even though the campaign file is published several hours after the start.

Analysis and solution

The team retains raw timestamps, assigns the source time zone, resolves ambiguous local times using the point-of-sale event sequence, and stores UTC as the canonical instant. Local business date is retained as a separate attribute. Demand is aggregated within each store’s local calendar day, while inventory snapshots are aligned by the latest snapshot available before the forecast origin.

Promotion features are generated using publication time rather than only campaign event time. The backtest is rebuilt with historical data vintages. Missing transaction intervals are classified as store closure, system outage, or unresolved. Genuine promotional spikes are retained and represented with promotion indicators rather than deleted as outliers.

Outcome

Forecast accuracy improves, but the more important result is validity: the new evaluation estimates performance under the information conditions that would have existed operationally. The retailer also detects that one country’s inventory system publishes revisions overnight, so same-day forecasts must use the previous available vintage.

Learning points

The case illustrates that time-zone conversion, calendar definition, aggregation, publication delay, and anomaly treatment are interdependent. Data preparation decisions alter the target itself. Forecasting teams should therefore review temporal data contracts with domain owners, not leave them solely to software developers.

11. Governance, Ethics, and Contemporary Practice

Temporal data may contain personal, location, health, or financial information. Governance should address lawful collection, purpose limitation, access control, retention, auditability, and secure handling. Time stamps can reveal behavioral patterns even when names are removed. Aggregation may reduce risk, but re-identification remains possible when records are detailed and linked with external data.

Automated anomaly detection and AI-assisted data cleaning can accelerate quality control, but they introduce risks. A model may incorrectly label a rare but important event as an error, encode historical reporting bias, or silently alter values. Automated corrections should be reversible, explainable, versioned, and subject to human review for high-impact applications.

Modern data platforms increasingly support data versioning, event sourcing, immutable raw layers, feature stores, and observability metrics. These technologies are valuable only when their temporal semantics are explicit. A feature store that records a value but not its publication time cannot guarantee leakage-free historical retrieval.

Best-Practice Checklist

Before modelling, confirm that:

• the row grain and target meaning are documented;
• event, observation, and publication times are distinguished;
• time zones and daylight-saving rules are explicit;
• interval boundaries are consistent and non-overlapping;
• aggregation matches variable type and units;
• calendar and holiday definitions reflect the business process;
• missingness has been classified rather than blindly filled;
• outliers have been investigated and labelled;
• censoring and truncation are retained as metadata;
• revisions and vintages are available or limitations disclosed;
• every feature was available at the forecast origin;
• transformations are fitted within each training period;
• data lineage and quality checks are reproducible.

Chapter Summary

Time series data management is a modelling discipline. Reliable forecasting requires an explicit distinction between instants, intervals, durations, event time, and publication time. Frequency and calendar choices determine the meaning of observations, while time zones and daylight-saving transitions can create duplicated, missing, or unequal-duration periods.

Aggregation must follow measurement semantics: flows are generally summed, stocks are usually observed at a point or period end, and rates often require exposure-weighted averages. Missing values must be interpreted in context, because a gap may represent a technical failure, closure, delayed publication, censoring, or a genuine zero. Outliers may be errors or valuable evidence of structural events. Revisions require vintage-aware evaluation, and structural breaks require investigation of changing regimes.

The central operational principle is temporal availability. A feature is valid for a forecast only if it was available by the forecast origin. Chronological validation, leakage controls, data contracts, lineage, and reproducible quality checks turn raw records into defensible forecasting data.

Key Terms

Temporal data: Data whose observations are indexed by time.

Event time: Time at which the underlying event occurred.

Observation time: Time at which a value was measured or recorded.

Publication time: Time at which information became available to users.

Processing time: Time at which a system ingested or processed a record.

Frequency: Regular spacing between observations.

Granularity: Precision or detail at which data are recorded.

Calendar: Rule system defining periods, weeks, holidays, or fiscal boundaries.

Time zone: Regional offset and rules governing local clock time.

Resampling: Converting data from one temporal frequency to another.

Aggregation: Combining observations into a larger temporal unit.

Missingness: Absence of an expected observation or value.

Outlier: An observation unusually distant from a relevant reference pattern.

Censoring: Partial observation in which a value is known only within a bound or category.

Structural break: A change in the underlying data-generating process.

Data vintage: A version of historical data available at a particular publication date.

Temporal leakage: Use of information unavailable at the forecast origin.

Forecast origin: Time at which a forecast is generated.

Data lineage: Record of a dataset’s sources, transformations, versions, and ownership.

Data contract: Formal specification of a dataset’s structure, semantics, timing, and quality rules.

Review Questions

Short Answer Questions

1. Distinguish event time from publication time.
2. Why are half-open intervals useful in temporal data systems?
3. Give two examples of variables that should not be aggregated by summation.
4. Why can replacing missing sales observations with zero be dangerous?
5. What is temporal leakage?

Long Answer Questions

1. Explain how time zones, daylight saving time, and business calendars affect hourly and daily aggregation.
2. Compare revised historical data with real-time vintage data and explain when each is appropriate.
3. Discuss a complete framework for diagnosing and treating missing values in forecasting data.
4. Explain why outliers should be classified rather than automatically removed.
5. Describe the components of a data contract for an analysis-ready time series.

Analytical / Application Questions

1. A company joins hourly demand to a daily promotion table using local date, but the promotion file is published after the forecast is issued. Identify the risks and design a leakage-free feature-generation procedure.
2. A sensor records a rate every 15 minutes, but one daylight-saving transition day contains 23 local hours. Explain how the daily total should be calculated and what metadata should be retained.
3. An economic indicator is revised twice after its initial release. Design a backtesting strategy that separately measures performance using current revised history and information available in real time.

Multiple Choice Questions with Answers

Question 1

Which time concept is most directly relevant to determining whether a feature was available when a forecast was issued?

A. Event time

B. Publication time

C. Fiscal year

D. Processing duration

Correct Answer: B. Publication time

Question 2

Which aggregation is generally appropriate for monthly total sales when daily sales are interval flows?

A. The sum of valid daily sales

B. The last daily sale

C. The maximum daily sale

D. The simple average of daily sales

Correct Answer: A. The sum of valid daily sales

Question 3

What is the main purpose of using half-open intervals such as [a, b)?

A. To estimate missing values

B. To remove seasonality

C. To avoid double counting at adjacent boundaries

D. To convert local time to UTC

Correct Answer: C. To avoid double counting at adjacent boundaries

Question 4

Which is an example of structural missingness?

A. A sensor fails during operating hours

B. A market is closed on a scheduled holiday

C. A database load is interrupted

D. A report is delayed by an overnight process

Correct Answer: B. A market is closed on a scheduled holiday

Question 5

Why might a genuine promotional demand spike be retained rather than deleted?

A. It is always normally distributed

B. It may represent a valid event the forecast should learn

C. Outliers cannot affect forecasts

D. Deleting observations is prohibited

Correct Answer: B. It may represent a valid event the forecast should learn

Question 6

A feature is calculated using observations after the forecast origin. What problem does this create?

A. Censoring

B. Temporal leakage

C. Calendar alignment

D. Downsampling

Correct Answer: B. Temporal leakage

Question 7

Which variable is most commonly aggregated using an end-of-period value rather than a sum?

A. Daily units sold

B. Hourly transaction count

C. Bank account balance

D. Monthly shipment volume

Correct Answer: C. Bank account balance

Question 8

What does a data vintage represent?

A. The physical age of a database server

B. A version of historical data available at a particular time

C. The frequency of a time series

D. A method for detecting outliers

Correct Answer: B. A version of historical data available at a particular time

Question 9

When is linear interpolation most defensible?

A. For a categorical store status

B. For a short gap in a plausibly smooth continuous measurement

C. For censored income values

D. For all missing demand observations

Correct Answer: B. For a short gap in a plausibly smooth continuous measurement

Question 10

Which validation design is generally most appropriate for forecasting?

A. Randomly shuffled cross-validation without time restrictions

B. A chronological rolling-origin evaluation

C. Training only on the final observation

D. Replacing all historical values with revised values

Correct Answer: B. A chronological rolling-origin evaluation