Data Preparation and Time-Series Features

Introduction

Forecasting quality depends not only on the forecasting model but also on the quality and timing of the data supplied to it. A sophisticated algorithm cannot correct a calendar misalignment, duplicated observation, incorrect unit conversion, or feature constructed with information that was unavailable at the forecast origin. Data preparation for time series therefore differs from ordinary data cleaning: the order and availability of observations are part of the information structure.

This chapter develops a disciplined workflow for converting operational records into analysis-ready time-indexed data. It covers aggregation, time alignment, missing values, transformations, deflation, calendar adjustment, lag and rolling features, and prevention of temporal leakage. The emphasis is on business applications such as sales, demand, prices, website traffic, employment, and financial indicators.

Learning objectives

By the end of this chapter, you should be able to:

1. Construct a valid time index and select an appropriate observation frequency.
2. Aggregate and align data from sources with different frequencies and time conventions.
3. Diagnose and treat missing, duplicated, irregular, and anomalous observations.
4. Apply transformations, deflation, and calendar adjustments appropriately.
5. Construct lagged, differenced, rolling, and expanding-window features.
6. Prevent look-ahead bias and temporal leakage during preparation and evaluation.
7. Document a reproducible preparation pipeline and assess its business implications.

1. The time-series data structure

A time series is a sequence of observations indexed by time. Let y_t denote the measured variable at time t. The index may represent a date, timestamp, week, month, quarter, or year. A useful dataset contains at least:

• a time index;
• one or more measured variables;
• a clear unit of measurement;
• a frequency or rule governing observation intervals; and
• metadata describing collection time, revisions, and definitions.

The distinction between event time and processing time is important. Event time is when an economic or operational event occurred; processing time is when it was recorded or became available. Forecasting must use the information available at the forecast origin, not necessarily the value eventually published after revisions.

A regular time series has observations at consistent intervals, such as every day or every month. An irregular series has variable intervals, as with individual transactions. Irregular records may be aggregated into a regular series, but aggregation requires a defensible rule.

Measurement frequency and business frequency

The chosen frequency should reflect both the decision and the data-generating process. Daily observations may be appropriate for staffing, while monthly data may be more suitable for budgeting. Higher frequency is not automatically better: it can increase noise, create many missing values, and produce unstable seasonal patterns.

A frequency conversion must preserve the meaning of the variable. A flow, such as revenue, is generally summed over a period. A stock, such as inventory at the end of a month, is generally represented by the final or average observation. A rate, such as an interest rate or conversion rate, may require a weighted average.

For a weighted rate, if subgroup rates are r_i and their relevant weights are w_i, the aggregate rate is:

Here, the weights must represent the appropriate denominator, such as exposures, visitors, or units. An unweighted average can be misleading when subgroup sizes differ.

2. Building a reliable time index

Time-index construction begins with explicit decisions about time zone, timestamp precision, calendar definition, and boundary conventions. A timestamp such as 2025-03-01 00:30 may belong to a different business day in another time zone. Daylight-saving changes can produce a repeated hour or a missing hour. These issues matter for hourly energy, web, transportation, and call-centre data.

A preparation checklist includes:

1. Parse dates using an unambiguous convention.
2. Convert timestamps to a documented time zone.
3. Sort observations chronologically.
4. Check for duplicate keys.
5. Identify gaps against the expected calendar.
6. Record whether periods are closed, open, fiscal, or calendar-based.
7. Preserve the original timestamp and the derived period key.

A duplicate timestamp is not always an error. Multiple transactions may legitimately occur at the same time. The correct key may be a combination of timestamp, product, location, and transaction identifier. Duplicates become problematic when the same business event is loaded twice.

Alignment across sources

Joining time series requires alignment on both time and entity. A daily sales table and a monthly economic indicator cannot be joined meaningfully without a declared mapping, such as assigning the monthly value to every day in that month. However, that mapping must respect availability. If the monthly indicator was released on the fifteenth day of the following month, it must not be assigned retrospectively to earlier forecast dates unless the forecasting exercise assumes later access.

Common alignment operations are:

• as-of join: match each observation to the most recent value known at that time;
• forward fill: carry a known value forward when it remains valid until an update;
• resampling: convert observations to another frequency using a defined aggregation rule; and
• calendar mapping: assign dates to fiscal periods, weeks, holidays, or promotion windows.

Forward filling a price quotation may be reasonable between trading days, but forward filling daily demand would create artificial demand and should generally be avoided.

3. Missing values and irregular observations

Missingness should be diagnosed before it is treated. A missing value may mean that no event occurred, that the system failed, that the market was closed, that the value was suppressed, or that the observation had not yet been published. These meanings imply different actions.

Three useful conceptual categories are:

• Missing completely at random: missingness is unrelated to observed or unobserved values.
• Missing conditionally on observed information: missingness depends on known factors, such as a store being closed.
• Informative missingness: missingness is related to the unobserved value, such as outages occurring during periods of unusually high demand.

These categories are analytical assumptions rather than labels that can always be proven from the dataset.

Treatment strategies

Deletion is acceptable only when missing periods are rare, non-informative, and the model can handle irregular spacing. Deleting dates from a daily demand series may incorrectly imply that time did not pass.

Structural zeros should be used only when zero is substantively correct. No recorded sale could mean zero sales, missing reporting, or a closed store. A calendar of store closures is needed to distinguish these cases.

Interpolation estimates values between known observations. Linear interpolation between y_a and y_b at time t is:

The method assumes a reasonably smooth path and should not be used automatically for demand spikes or variables whose values are only meaningful at observation times. Interpolation using a future value is acceptable for retrospective descriptive analysis but can cause leakage in real-time forecasting if the future value would not yet be known.

Last observation carried forward is appropriate for slowly changing attributes or valid-until-replaced reference data. It is not generally appropriate for rapidly changing demand.

Model-based imputation may use regression, state-space methods, or neighbouring series. It can be useful but must be fitted within the training period and accompanied by an indicator showing that the value was imputed. Imputation uncertainty should be considered in high-stakes decisions.

A missingness indicator can be defined as:

The indicator can preserve information about reporting or operational conditions, but it should not be treated as proof that missingness has a causal meaning.

4. Outliers, anomalies, and data errors

An outlier is an unusually large or small observation; an error is an incorrect observation. They are not equivalent. A major promotion may produce a legitimate sales peak, while a duplicated transaction may create an artificial peak. Removing all extreme values can destroy the very events a forecasting system must learn.

Anomaly investigation should combine statistical evidence with business context. Useful checks include:

• impossible values, such as negative quantities where prohibited;
• abrupt level shifts after a system migration;
• repeated records;
• unit changes;
• unusual values concentrated in one location or product; and
• disagreement with related measures.

Possible treatments include correction from the source system, retaining a validated intervention, winsorization, robust transformations, or adding event indicators. Every alteration should be logged with the original value, revised value, reason, and author or system responsible.

5. Transformations and scale management

Transformations can stabilize variance, reduce skewness, and express changes in a more useful form. They should be chosen for interpretation and model assumptions, not applied automatically.

The first difference is:

It measures an absolute change and has the same units as y_t. The growth rate is:

This requires y_t-1 ≠ 0 and is unstable when the denominator is near zero. The log difference is:

It requires positive values and approximates the proportional growth rate when changes are small. For example, if sales increase from 100 to 110, the absolute change is 10 units, the growth rate is 0.10 or 10%, and the log difference is approximately 0.0953.

The Box–Cox transformation for positive y is:

Here λ is a selected parameter. The transformation is defined for positive observations and should be interpreted in relation to the model and back-transformation procedure. For count data with zeros, alternatives such as log(1+y_t) may be used, but the resulting interpretation differs from a standard log transformation.

Scaling, such as standardization, may help algorithms that use distances or regularization:

Here μ_x and σ_x are the mean and standard deviation estimated from the training data only. Re-estimating them using the test period leaks future distributional information.

6. Nominal values, real values, and deflation

Financial quantities measured in money may reflect both changes in physical activity and changes in prices. Deflation converts nominal values into values expressed at a common price level. If N_t is a nominal amount and P_t is a price index with a common base, the real amount is:

The price index and nominal amount must refer to compatible categories, geography, and periods. If the index equals 125 with base 100, nominal revenue of 250,000 corresponds to real revenue of 250000/(125/100) = 200000 in base-period currency units. Deflation is not appropriate merely because a variable is monetary; the analytical question must concern real purchasing power, volume, or activity.

For forecasting, the release date and revision history of the deflator matter. A final revised price index should not be used in a historical real-time evaluation if it was unavailable at the forecast date.

7. Calendar and seasonal adjustment

Business data are affected by day-of-week composition, holidays, trading days, month length, school terms, promotions, and moving events such as Easter. A month with five Saturdays may have higher retail demand even when underlying daily demand is unchanged.

Calendar adjustment attempts to separate systematic calendar effects from the underlying signal. A simple regression representation is:

One day category is omitted as the reference category to avoid perfect multicollinearity. The coefficients describe average calendar differences conditional on the model and sample. For count or nonnegative data, a transformed or generalized model may be more suitable.

Adjustment can be performed explicitly with calendar features or by using a statistical seasonal-adjustment procedure. The choice depends on purpose. For operational forecasting, calendar variables are often retained because future holidays are known. For economic interpretation, adjusted and unadjusted series should be clearly distinguished. Never label a series “seasonally adjusted” unless the adjustment method and revision policy are documented.

8. Time-series features

Features summarize information available up to a forecast origin. For a target y_t, a lag of order k is:

A one-period lag uses the previous observation. Lags can capture persistence, delayed responses, payment cycles, and recurring operational patterns. For monthly data, lag 12 often represents the same month in the previous year; for daily data, lag 7 may represent the same weekday, subject to calendar changes.

A difference can be written using lag notation:

Rolling statistics summarize a trailing window. A trailing mean of width w is:

This definition deliberately excludes y_t when the feature is used to predict y_t. A trailing sample variance is:

It requires at least two valid prior observations and assumes the window is being used as a descriptive estimate, not as a claim that observations are independent.

Other useful features include trailing minimum and maximum, exponentially weighted averages, cumulative totals, rolling counts of events, price changes, and ratios. A trailing exponentially weighted mean can be defined recursively as:

The feature uses prior information only when initialized and updated without y_t. The parameter α controls responsiveness: larger values give greater weight to the most recent known observation.

External predictors require the same temporal discipline. Weather forecasts, interest rates, advertising plans, and inventory data should be joined according to their publication or availability time. A future realized temperature is not a valid predictor for a real-time demand forecast unless the forecasting design explicitly assumes perfect future weather knowledge.

9. Leakage prevention and evaluation design

Temporal leakage occurs when preparation uses information that would not have been available at the time a forecast was made. Common examples include:

• calculating a global mean using the entire dataset;
• standardizing before splitting into training and test periods;
• interpolating a missing training value using a later test value;
• using revised economic data rather than the original release;
• constructing a centered moving average; and
• randomly splitting observations from the same time series.

A safe design separates the data chronologically. Suppose the training period ends at T. Parameters such as scaling means, imputation models, transformation choices, and feature-selection rules are estimated using observations at or before T. They are then applied to later observations without refitting on the test set.

Rolling-origin evaluation is often more realistic than a single split. A model is trained through one date, forecasts the next period or horizon, then the origin advances. For a forecast horizon h, the forecast error may be represented as:

Here ŷ_t+h|t is the forecast made at time t for time t+h. Preparation must reproduce what would have happened at each origin. This includes data availability, revisions, feature windows, and imputation.

A practical pipeline is:

1. Ingest raw data and preserve immutable copies.
2. Validate keys, units, timestamps, and source metadata.
3. Establish the calendar and target frequency.
4. Aggregate each variable according to its measurement type.
5. Align sources using event and availability time.
6. Diagnose missingness, duplicates, and anomalies.
7. Split chronologically into development and evaluation periods.
8. Fit transformations and imputation rules on the development data.
9. Construct lagged and rolling features using past information only.
10. Validate feature availability and document every rule.

Figure: A leakage-safe preparation workflow separates fitting from later application and constructs features from information available at the forecast origin.

10. Illustrative case: forecasting weekly grocery demand

Background

A grocery retailer wants to forecast weekly demand for a perishable product at each store. The raw data contain transaction records, product prices, promotions, store closures, holidays, and weather observations.

Problem

The transaction system records local timestamps, while the promotional database uses Monday-to-Sunday weeks. Some stores report zero transactions during closures; others fail to upload data. Weather observations are available daily, but the forecast is produced every Thursday for the following week.

Analysis and approach

The analyst first converts timestamps to each store’s local time zone and defines the business week explicitly. Transactions are aggregated by store and product using sums because sales are flows. Prices are represented by a sales-weighted average, while inventory is represented by the final available stock observation.

Store-closure calendars distinguish structural zeros from missing uploads. Missing weather observations are filled only when the source provider’s values are valid for the interval; otherwise, a missingness flag is retained. Promotion indicators are joined using the promotion publication schedule. The model uses lagged sales at one and four weeks, a trailing four-week mean calculated from prior weeks, price change, promotion status, holiday indicators, and store-closure information.

The retailer creates a chronological development period and evaluates performance using rolling-origin forecasts. Scaling parameters and any imputation models are fitted separately at each origin. The team rejects a preliminary feature based on next week’s realized weather because it would not be available when the forecast is issued.

Outcome

The resulting dataset contains one row per store-product-week, documented units, explicit missingness codes, reproducible features, and a forecast-time availability field. Forecast accuracy improves not because the most complex algorithm was selected, but because the target and predictors were aligned with the operational decision process.

Learning points

1. Aggregation rules depend on measurement type.
2. A zero is meaningful only when the business context supports it.
3. Availability time is as important as event time.
4. Rolling features must exclude the target period when predicting that period.
5. Evaluation must imitate the way forecasts will actually be produced.

11. Governance, ethics, and implementation practice

Data preparation can materially affect employees, customers, suppliers, and financial decisions. Analysts should document definitions, preserve provenance, control access to sensitive records, and avoid silently changing observations. Personally identifiable information should be minimized or removed when it is not required. When forecasts influence staffing, credit, pricing, or service access, the organization should test whether data gaps or measurement practices systematically disadvantage particular groups or locations.

A production pipeline should be version-controlled, automatically tested, monitored for schema changes, and auditable. Tests should check chronological ordering, duplicate keys, expected frequency, allowable ranges, feature availability, and the absence of future timestamps in training features. Data drift monitoring is also important: a valid preparation rule may become inappropriate after a product redesign, pricing change, acquisition, or system migration.

Artificial intelligence can assist anomaly detection, entity matching, and feature generation, but generated transformations require human validation. An AI system that imputes values or selects features may inadvertently use future information or encode undocumented assumptions. Reproducibility and explainability remain necessary, especially in regulated or high-impact settings.

Challenges and limitations

No preparation method is universally correct. Interpolation can smooth away genuine shocks; deflation depends on an appropriate price index; calendar adjustment may remove variation that managers actually need to forecast; and rolling features can be unstable with short histories. Structural breaks may make historical relationships unreliable. Revisions and delayed releases complicate retrospective analysis. Finally, clean-looking data can still represent biased measurement processes. Preparation should therefore be treated as an analytical decision subject to validation, not as a purely mechanical prelude to modelling.

Chapter Summary

Analysis-ready time-series data require a valid time index, consistent frequency, appropriate aggregation, and careful alignment of event time with availability time. Flows, stocks, rates, prices, and counts require different aggregation rules. Missing values must be interpreted before being imputed, and anomalies must be investigated rather than automatically removed. Differences, growth rates, logarithms, scaling, and Box–Cox transformations alter both statistical behaviour and interpretation. Monetary series may require deflation when the analytical objective concerns real values. Calendar and seasonal effects can be represented through known features or formal adjustment procedures.

Lagged, differenced, rolling, and exponentially weighted features are powerful summaries of history, but they must be constructed from information available at the forecast origin. Chronological splits, rolling-origin evaluation, source vintage control, and training-only estimation of preprocessing parameters are central protections against leakage. A professional workflow also requires provenance, documentation, testing, privacy safeguards, and monitoring for structural change.

Key Terms

Time index: Ordered variable identifying when each observation occurred.

Event time: Time at which the measured event happened.

Availability time: Time at which information became available for use.

Regular time series: Series observed at consistent intervals.

Resampling: Converting data from one frequency to another.

Flow variable: Quantity accumulated over a period.

Stock variable: Quantity measured at a point in time.

As-of join: Join that matches an observation to the latest value known at that time.

Structural zero: A zero that is substantively valid because no activity could occur.

Interpolation: Estimation of values between observed points.

Missingness indicator: Variable recording whether an original value was missing.

Outlier: Observation unusually distant from typical values.

Deflation: Conversion of nominal monetary values into common price-level units.

Calendar adjustment: Adjustment or modelling of systematic calendar effects.

Lag: Earlier value of a variable used as a time-indexed feature.

Rolling statistic: Statistic calculated over a moving historical window.

Expanding window: Calculation using all available prior observations up to a date.

Temporal leakage: Use of information unavailable at the forecast origin.

Forecast origin: Time at which a forecast is generated.

Rolling-origin evaluation: Repeated chronological training and forecasting procedure.

Review Questions

Short Answer Questions

1. Why must aggregation rules differ for flows, stocks, and rates?
2. Distinguish event time from availability time.
3. When is a structural zero different from a missing value?
4. State one condition required for using a logarithmic transformation.
5. Why should a trailing moving average exclude the current target observation?

Long Answer Questions

1. Explain a complete workflow for converting transaction-level records into a regular weekly time series.
2. Compare deletion, interpolation, forward filling, and model-based imputation for time-series missing values.
3. Explain how deflation changes the interpretation of a nominal financial series.
4. Discuss calendar adjustment and the role of holidays, weekdays, and moving events in business forecasting.
5. Explain temporal leakage and describe at least four ways it can occur during preprocessing.

Analytical / Application Questions

1. A retailer has daily sales, monthly inflation data released on the fifteenth of the following month, and weekly promotion plans. Design a leakage-safe feature-construction process for a forecast made every Monday.
2. A monthly series has values 100, 110, and 99. Calculate the first difference from month two to month three, the growth rate, and the log difference. Interpret why the measures differ.
3. A data scientist standardizes the entire dataset before a chronological train-test split and uses a centred moving average as a predictor. Identify the leakage mechanisms and propose corrected procedures.

Multiple Choice Questions with Answers

Question 1

Which aggregation is generally most appropriate for daily revenue when constructing monthly revenue?

A. Sum

B. Last observation only

C. Unweighted average of daily revenue rates

D. Minimum value

Correct Answer: A. Sum

Question 2

What is the main distinction between event time and availability time?

A. Event time is always earlier than calendar time

B. Availability time identifies when information could be used

C. Event time applies only to financial data

D. Availability time is the date of data deletion

Correct Answer: B. Availability time identifies when information could be used

Question 3

Which condition is required for the standard log transformation log(y_t)?

A. All values must be negative

B. Values must be positive

C. Observations must be equally spaced

D. The series must have no seasonality

Correct Answer: B. Values must be positive

Question 4

For a forecast of y_t, which feature is leakage-safe when calculated correctly?

A. A centred average containing y_{t+1}

B. The realized value of next month’s weather

C. The one-period lag y_{t-1}

D. The full-sample mean including the test period

Correct Answer: C. The one-period lag y_{t-1}

Question 5

A price index equals 125 with base 100. What is the real value of a nominal amount of 250,000 in base-period units?

A. 156,250

B. 200,000

C. 250,000

D. 312,500

Correct Answer: B. 200,000

Question 6

Which treatment is most appropriate for a slowly changing reference value that remains valid until replaced?

A. Forward filling

B. Random deletion

C. Replacing all values with zero

D. Using the future test value

Correct Answer: A. Forward filling

Question 7

Why is an unweighted average of subgroup conversion rates potentially misleading?

A. Rates cannot be averaged under any circumstances

B. Subgroups may have different denominators or sizes

C. Conversion rates always contain negative values

D. Averages remove all seasonality

Correct Answer: B. Subgroups may have different denominators or sizes

Question 8

What does rolling-origin evaluation primarily ensure?

A. Randomized observations in every split

B. Forecast testing that respects chronological production conditions

C. Removal of all outliers

D. Perfect forecasts at every horizon

Correct Answer: B. Forecast testing that respects chronological production conditions

Question 9

Which statement about standardization in a forecasting pipeline is correct?

A. The mean and standard deviation should be estimated using future test data

B. They should be estimated from the relevant training data and then applied forward

C. Standardization always requires logarithmic data

D. Standardization eliminates temporal dependence

Correct Answer: B. They should be estimated from the relevant training data and then applied forward

Question 10

A store has no transactions on a date. What should be checked before recording sales as zero?

A. Whether the store was closed or the reporting system failed

B. Whether the value is larger than the annual maximum

C. Whether the date is in the future only

D. Whether all other stores also had zero sales

Correct Answer: A. Whether the store was closed or the reporting system failed