Stationarity, Autocorrelation, and ARIMA Models

Introduction

Time-series data record observations in chronological order: daily sales, monthly unemployment, quarterly revenue, hourly electricity demand, or annual inflation. Unlike ordinary cross-sectional observations, successive time-series observations may be statistically dependent. This dependence is useful: it allows analysts to extract persistence, cycles, and seasonal structure for forecasting. It also creates risks. Regression and forecasting methods can produce misleading results when a series is non-stationary, when residuals remain autocorrelated, or when seasonal patterns are ignored.

This chapter develops the central Box–Jenkins framework for time-series forecasting. It explains stationarity, white noise, autocorrelation and partial autocorrelation, differencing and unit roots, autoregressive and moving-average models, ARIMA and seasonal ARIMA models, parameter estimation, diagnostic checking, and forecasting. The emphasis is both mathematical and practical: a good model must be statistically defensible, interpretable, and useful for business decisions.

Learning objectives

By the end of this chapter, you should be able to:

1. Explain weak stationarity and why it matters for forecasting.
2. Interpret the ACF and PACF and use them to identify candidate models.
3. Distinguish white noise, random walks, AR, MA, ARMA, ARIMA, and SARIMA processes.
4. Apply differencing while avoiding overdifferencing.
5. Explain unit roots and the logic of stationarity tests.
6. Estimate model parameters and evaluate residual diagnostics.
7. Produce and interpret forecasts and prediction intervals.
8. Assess the practical, ethical, and governance implications of automated forecasting.

1. Time-series notation and dependence

Let y_t denote the observed value at time t, where t = 1, 2, ..., T. The series may be represented as a combination of systematic and random components. A useful general representation is

where f(t) represents predictable structure and ε_t is an unpredictable disturbance. In ARIMA modelling, systematic structure is represented through lagged observations and lagged disturbances rather than necessarily through a separate deterministic trend.

The lag operator B is defined by B y_t = y_t-1. Consequently, B^k y_t = y_t-k. The first-difference operator is ∆ = 1 - B, so

The lag notation makes ARIMA models compact and is especially useful for seasonal differencing.

2. Stationarity

A process is weakly, or covariance-, stationary if its mean, variance, and covariance structure do not change over time. Formally, for all relevant times,

where μ and σ² are constant, and γ_k depends only on the lag k, not on the calendar time t. The autocorrelation at lag k is

where γ₀ = Var(y_t). Stationarity does not mean that the observed series is constant. A stationary series can fluctuate substantially; it means that its probabilistic behavior is stable over time.

A deterministic trend, changing variance, structural break, or evolving seasonal pattern can violate stationarity. A non-stationary process may still be forecastable, but standard ACF interpretation and many statistical tests become unreliable unless the relevant non-stationarity is modelled.

Why stationarity matters

Stationarity supports:

• stable estimates of dependence;
• meaningful historical relationships between observations;
• reliable use of ACF and PACF patterns;
• conventional parameter inference and residual diagnostics;
• forecasts whose uncertainty can be quantified from a stable process.

A series may be transformed using logarithms, seasonal adjustment, detrending, or differencing. A logarithmic transformation is useful when variability increases with the level. For positive data, z_t = ln(y_t) often converts multiplicative effects into additive effects. It does not, however, guarantee stationarity.

3. White noise and benchmark processes

A white-noise process is a sequence of uncorrelated disturbances with constant mean and variance. A common specification is

meaning that the disturbances have mean zero, variance σ², and zero autocovariance at every nonzero lag. White noise need not be normally distributed for the process to be uncorrelated, although normality is often assumed for likelihood-based inference and prediction intervals.

A white-noise model is a crucial benchmark. If forecast errors resemble white noise, the model has extracted the systematic temporal information available under its assumptions. If residuals remain autocorrelated, the model is incomplete.

A random walk is defined by

or equivalently ∆y_t = ε_t. If the initial value is fixed, its variance increases with time because shocks accumulate. Thus the level series is non-stationary, although its first differences are white noise. A random walk with drift is

where c is the average change per period.

4. ACF and PACF

The sample autocorrelation function (ACF) measures linear association between observations separated by k periods:

where ȳ is the sample mean and T is the number of observations. The ACF includes both direct and indirect associations. For example, a relationship at lag 2 may arise because both observations are related to the intervening observation.

The partial autocorrelation at lag k measures the association between y_t and y_t-k after controlling for lags 1 through k-1. It can be obtained from the final coefficient in an autoregression containing the first k lags.

For many approximate large-sample applications, ACF spikes outside ±1.96/√T suggest statistical significance under a white-noise null hypothesis. This is only an approximation: the bounds can be affected by parameter estimation, non-normality, multiple comparisons, and non-stationarity.

Identification patterns

“Cuts off” means that correlations are approximately zero after the stated lag. These patterns are guides, not rules. Sampling variation, mixed dynamics, seasonality, outliers, and structural breaks can obscure them.

5. Autoregressive and moving-average models

Autoregressive models

An AR(p) model expresses the current value as a linear function of its previous values plus a white-noise innovation:

Here c is an intercept, φ_i are autoregressive coefficients, and ε_t is white noise. For an AR(1),

Stationarity requires |φ| < 1 for the AR(1). Its long-run mean is μ = c/(1-φ). If φ is positive and close to 1, shocks have highly persistent effects. Negative values can produce alternating movements.

For higher-order AR models, stationarity requires that the roots of the characteristic polynomial lie outside the unit circle. With the AR polynomial

the roots of φ(B)=0 must have modulus greater than one.

Moving-average models

An MA(q) model expresses the current value as a function of the current and past innovations:

The innovations are unobserved shocks. An MA(1) has

An MA(q) is stationary for finite coefficients, but invertibility is usually imposed so that the model has a unique representation in terms of current and past observations. For an MA(1), invertibility requires |θ| < 1.

ARMA models

An ARMA(p,q) model combines the two structures:

where φ(B) is the AR polynomial and θ(B) is the MA polynomial. ARMA models apply to stationary series. If the original series is non-stationary, differencing leads to ARIMA models.

6. Differencing, unit roots, and ARIMA

The ARIMA(p,d,q) model applies an ARMA(p,q) model to the series after differencing d times:

The symbols p, d, and q denote autoregressive order, differencing order, and moving-average order. Depending on software convention, the constant may be represented as a mean, drift, or intercept in the differenced equation; these interpretations must be checked carefully.

A unit root occurs when an AR characteristic root lies on the unit circle. In the AR(1) model, φ=1 gives the random walk. Unit-root processes are non-stationary and often require differencing. The augmented Dickey–Fuller (ADF) test examines a regression such as

The null hypothesis is typically γ=0, indicating a unit root; the alternative is γ<0, indicating stationarity around the included deterministic terms. Standard t-distribution critical values do not apply directly, and the test has low power in short samples. The KPSS test reverses the usual logic by taking stationarity as the null. Tests should be combined with plots, domain knowledge, and model diagnostics rather than treated as automatic decision rules.

Choosing the differencing order

Use the smallest d that produces a plausibly stationary series. Evidence includes a stable time plot, an ACF that decays relatively quickly, and tests used with appropriate caution. Overdifferencing can create unnecessary moving-average behavior, inflate forecast uncertainty, and reduce interpretability. A practical workflow is to examine levels, first differences, and, if justified, second differences. Second differencing should be uncommon and substantively justified.

7. Seasonal ARIMA models

Seasonality repeats at a known period s, such as s=12 for monthly data with annual seasonality or s=4 for quarterly data. Seasonal differencing is

A seasonal ARIMA model is written SARIMA(p,d,q)(P,D,Q)_s:

Here P, D, and Q are seasonal AR order, seasonal differencing order, and seasonal MA order. The seasonal polynomials are Φ(B^s) and Θ(B^s). Seasonal differencing should be used only when seasonal persistence is non-stationary; ordinary seasonal fluctuations can often be handled by seasonal AR or MA terms, deterministic seasonal variables, or other methods.

For example, SARIMA(0,1,1)(0,1,1)_12 models monthly data after both ordinary and annual seasonal differencing, with nonseasonal and seasonal MA terms. Such a model is widely useful for business series, but its adequacy must be verified rather than assumed.

8. Estimation and model selection

Parameters may be estimated by conditional least squares, maximum likelihood, or exact likelihood methods. Maximum likelihood chooses parameter values that make the observed data most plausible under the specified distribution, commonly Gaussian innovations. Gaussian assumptions are not required for the conditional mean forecast to be useful, but they affect likelihood-based standard errors and prediction intervals.

A practical identification process is:

1. Plot the series and inspect changes in level, variance, outliers, breaks, and seasonality.
2. Transform the series if variance grows with the level.
3. Apply justified ordinary or seasonal differencing.
4. Inspect ACF and PACF of the transformed series.
5. Fit several parsimonious candidate models.
6. Compare AIC or BIC, while remembering that information criteria do not replace diagnostics. AIC is generally AIC = -2ℓ + 2k, and BIC is BIC = -2ℓ + k ln(T), where ℓ is maximized log-likelihood, k is the number of estimated parameters, and T is sample size.
7. Check residuals and evaluate genuine out-of-sample forecasts.

Models should be identifiable and preferably satisfy stationarity and invertibility conditions. Very high orders often indicate that trend, seasonality, breaks, or outliers have not been handled properly.

9. Residual diagnostics

Let e_t = y_t - ŷ_t|t-1 denote a one-step residual or innovation estimate. A satisfactory model should leave residuals that are approximately:

• zero mean;
• uncorrelated;
• stable in variance;
• free from systematic seasonal structure.

The residual ACF should show no meaningful pattern. The Ljung–Box statistic tests whether a group of autocorrelations is jointly zero:

where h is the maximum diagnostic lag. Under suitable conditions, the statistic is compared with a chi-square distribution using degrees of freedom adjusted for estimated model parameters. A small p-value indicates remaining autocorrelation, not that the model is necessarily useless; a large p-value does not prove that all assumptions hold.

Residual histograms and Q–Q plots help assess distributional shape. ARCH-type tests or residual plots can reveal changing variance. Outliers should be investigated: an unusual observation may represent data error, a genuine event, a level shift, or an intervention. Automatically deleting it can damage forecasts and obscure important business information.

10. Forecasting with ARIMA models

A point forecast is the conditional expectation given information available at time T:

where h is the forecast horizon and I_T is all information available through time T. Forecast errors are

For a stationary AR(1), y_t = c + φy_t-1 + ε_t, the h-step forecast is

where μ=c/(1-φ). If |φ|<1, the forecast converges to the long-run mean as h increases. The forecast-error variance, assuming independent innovations with variance σ², is

For a 95% approximate prediction interval under normal innovations,

For non-normal or strongly asymmetric data, bootstrap or simulation-based intervals may be more appropriate. When forecasts are made on a logarithmic scale, back-transforming the point forecast can introduce bias; a simple exponentiation is generally the median under lognormal assumptions, not necessarily the mean on the original scale.

Worked example: identifying a monthly demand model

Suppose a retailer has 60 months of demand observations. The plot shows increasing variability with demand level and a repeating annual pattern. After taking logarithms, the analyst computes a seasonal difference at lag 12. The transformed series has a stable mean, but its ACF remains high at lag 1 and declines gradually. The PACF has a prominent lag-1 spike and then mostly small values.

The evidence suggests a nonseasonal AR(1) component after seasonal differencing, although an MA alternative should also be fitted. The analyst estimates SARIMA(1,0,0)(0,1,0)_12 and compares it with SARIMA(0,0,1)(0,1,0)_12 and a model including both terms. The selected model is not chosen solely because its AIC is smallest. Its residual ACF must show no systematic spikes, the Ljung–Box test should not indicate substantial remaining correlation, and rolling-origin forecast accuracy should be competitive.

If the final model forecasts log demand, the forecasts are transformed back to demand units for inventory decisions. Prediction intervals, not only point forecasts, are supplied because safety stock depends on uncertainty. Managers should also review whether a promotion, stockout, competitor entry, or supply disruption explains unusual observations. Such events may require intervention variables or scenario analysis rather than a purely automatic ARIMA response.

11. Applied case: forecasting hospital emergency demand

The following is an illustrative case.

Background and problem

A regional hospital must forecast daily emergency-department arrivals for staffing and bed planning. The data contain four years of observations, with weekday effects, winter increases, occasional epidemics, and several closure-related anomalies. Underforecasting creates unsafe workload pressure; overforecasting increases staffing cost.

Analysis

The forecasting team first plots arrivals and checks data quality. It identifies missing days, records the service disruptions, and distinguishes true low-demand days from incomplete reporting. A variance-stabilizing transformation is considered, but the count nature of the data is retained in the operational interpretation. ACF and PACF plots show strong lag-7 dependence, so weekly seasonality is explicitly modelled. Differencing is considered only after checking whether the apparent trend is persistent or caused by a temporary epidemic.

Candidate seasonal ARIMA models are estimated using a rolling-origin evaluation. Each training window ends before the validation period, preventing future information from entering the model. Residual diagnostics are examined separately for ordinary and seasonal lags. Intervention indicators are added for known closure days, and the team compares forecast accuracy using MAE and a cost-weighted measure that penalizes underprediction more heavily.

Solution and outcome

The selected model combines weekly seasonal dynamics with intervention variables. Forecasts are provided as point estimates and prediction intervals. Staffing uses the upper interval for high-risk days, while routine scheduling uses the central forecast. During a new epidemic, the model is not allowed to operate without review: recent data are monitored, forecasts are compared with epidemiological intelligence, and parameters may be re-estimated.

Learning points

1. Data quality and institutional knowledge are part of model quality.
2. Seasonal dependence must be distinguished from long-term non-stationarity.
3. Forecast evaluation must replicate the way forecasts will be used.
4. A cost-sensitive decision may require more than minimizing average error.
5. Human oversight is essential when structural conditions change.

12. Limitations, governance, and contemporary practice

ARIMA models are effective for many univariate series, but they have limitations. They assume that historical dependence is informative about the future. Structural breaks, policy changes, technological disruption, extreme events, and changing customer behavior can invalidate this assumption. ARIMA models also focus primarily on temporal dependence; causal drivers such as prices, advertising, weather, and macroeconomic variables require dynamic regression or related models.

Modern forecasting systems often compare ARIMA with exponential smoothing, state-space models, gradient-boosting methods, and neural networks. Machine learning may improve accuracy when many predictors and nonlinearities are available, but it does not eliminate the need for stationarity awareness, temporal validation, leakage prevention, uncertainty quantification, and interpretability. Hybrid systems may use ARIMA for residual temporal structure after modelling explanatory variables.

Forecast governance requires versioned data, documented transformations, reproducible code, monitoring of forecast errors, and explicit approval rules for overrides. In high-impact settings, analysts should communicate uncertainty and avoid presenting a model forecast as a guaranteed outcome. Forecasts can influence staffing, credit, prices, and public services; biased historical data may therefore reproduce inequities. Ethical practice includes auditing performance across relevant groups, protecting confidential data, documenting assumptions, and ensuring that decision-makers understand model limitations.

Best-practice checklist

• Plot the raw series before modelling.
• Check missing values, duplicate timestamps, outliers, and structural breaks.
• Transform variance only when justified.
• Use the minimum effective differencing order.
• Treat ACF and PACF as identification evidence, not proof.
• Compare parsimonious candidate models.
• Inspect residuals at both ordinary and seasonal lags.
• Use rolling or expanding-window validation.
• Report prediction intervals and the forecast horizon.
• Monitor performance after deployment and investigate model failure.

Chapter Summary

Stationarity means that the mean, variance, and covariance structure of a time series are stable over time. It is central to meaningful dependence analysis and reliable ARIMA modelling. White noise provides a benchmark for unpredictable residual variation, while the ACF measures total linear dependence at each lag and the PACF measures dependence after controlling for shorter lags.

AR models use lagged observations, MA models use current and past innovations, and ARMA models combine both for stationary data. Differencing converts many non-stationary series into stationary ones; ARIMA models apply ARMA structure after ordinary differencing, while SARIMA models add seasonal AR, differencing, and MA components. Unit-root tests, plots, ACF/PACF patterns, domain knowledge, and diagnostics should be used together.

Reliable forecasting requires parameter estimation, model comparison, residual checking, and genuine out-of-sample validation. Ljung–Box tests and residual ACFs help identify remaining dependence, but no test replaces substantive judgment. Prediction intervals communicate uncertainty, and governance procedures are essential when forecasts affect people, resources, or safety.

Key Terms

Stationarity: Stability of a process’s mean, variance, and covariance structure over time.

Weak stationarity: Stationarity defined through constant first and second moments.

White noise: Uncorrelated random disturbances with constant mean and variance.

Random walk: A process whose current level equals its previous level plus a disturbance.

Lag operator: Operator B that shifts a series one period backward.

Differencing: Subtracting a previous observation, or a seasonal observation, to remove persistence.

Autocorrelation function (ACF): Correlation between observations separated by specified lags.

Partial autocorrelation function (PACF): Correlation at a lag after controlling for shorter lags.

AR model: Model using lagged values of the dependent series.

MA model: Model using current and past innovations.

ARMA model: Combination of autoregressive and moving-average terms for stationary data.

ARIMA model: ARMA model applied after ordinary differencing.

SARIMA model: Seasonal ARIMA model with seasonal AR, differencing, and MA terms.

Unit root: A characteristic root on the unit circle, commonly producing non-stationarity.

Invertibility: A condition allowing a moving-average process to have a unique observational representation.

Innovation: A new, unpredictable disturbance entering a time-series model.

Ljung–Box test: Joint test for residual autocorrelation over selected lags.

Prediction interval: Range intended to contain a future observation with stated probability.

Rolling-origin validation: Forecast evaluation using repeatedly updated historical training windows.

Structural break: A change in the data-generating process over time.

Review Questions

Short Answer Questions

1. Define weak stationarity and state its three principal requirements.
2. What is the difference between the ACF and the PACF?
3. Why is white noise an important benchmark for residual diagnostics?
4. Explain the roles of p, d, and q in ARIMA(p,d,q).
5. What does a unit root imply about a time series?

Long Answer Questions

1. Explain how ACF and PACF patterns can guide the selection of AR and MA orders. Discuss why these patterns should not be used mechanically.
2. Compare AR, MA, ARMA, ARIMA, and SARIMA models, including their assumptions and typical applications.
3. Describe a complete Box–Jenkins modelling workflow from data inspection through forecast validation.
4. Explain the purpose and limitations of ADF and KPSS tests in assessing stationarity.
5. Discuss why residual diagnostics and prediction intervals are essential for operational forecasting.

Analytical / Application Questions

1. A monthly sales series has a slowly decaying ACF, a large spike at lag 12, and increasing variance. Propose a defensible sequence of transformations and diagnostics before fitting a SARIMA model.
2. An AR(1) process has c=4 and φ=0.8. Calculate its long-run mean and explain what happens to long-horizon forecasts.
3. A forecasting model has a low AIC but significant residual autocorrelation at lags 1 and 7. Evaluate whether it should be deployed and recommend specific next steps.

Multiple Choice Questions with Answers

Question 1

Which condition is required for weak stationarity?

A. The mean and variance must increase over time

B. The covariance depends only on the lag, not calendar time

C. Every observation must equal the previous observation

D. All observations must be normally distributed

Correct Answer: B. The covariance depends only on the lag, not calendar time

Question 2

For a stationary AR(1) model, which condition is required?

A. φ = 1

B. φ > 1

C. |φ| < 1

D. φ = 0 only

Correct Answer: C. |φ| < 1

Question 3

Which pattern most commonly suggests an MA(q) process?

A. ACF cuts off after lag q and PACF decays

B. PACF cuts off after lag q and ACF is zero

C. Both ACF and PACF cut off immediately

D. Both functions must increase without bound

Correct Answer: A. ACF cuts off after lag q and PACF decays

Question 4

What does first differencing calculate?

A. y_t + y_{t-1}

B. y_t − y_{t-1}

C. y_t ÷ y_{t-1}

D. y_t × y_{t-1}

Correct Answer: B. y_t − y_{t-1}

Question 5

In an ARIMA(p,d,q) model, d denotes the:

A. Seasonal period

B. Number of autoregressive parameters

C. Number of differences applied

D. Number of forecast periods

Correct Answer: C. Number of differences applied

Question 6

For monthly data with annual seasonality, the usual seasonal period is:

A. 4

B. 7

C. 12

D. 52

Correct Answer: C. 12

Question 7

What is the main purpose of the Ljung–Box test?

A. To test whether residuals are jointly uncorrelated over selected lags

B. To estimate the seasonal period automatically

C. To calculate the sample mean

D. To guarantee normality of the original series

Correct Answer: A. To test whether residuals are jointly uncorrelated over selected lags

Question 8

For an AR(1) model with c = 2 and φ = 0.5, the long-run mean is:

A. 1

B. 2

C. 4

D. 5

Correct Answer: C. 4

Question 9

Which practice best prevents look-ahead bias in forecast evaluation?

A. Randomly shuffling all observations

B. Using future observations in model estimation

C. Using rolling-origin or expanding-window validation

D. Evaluating only the in-sample fitted values

Correct Answer: C. Using rolling-origin or expanding-window validation

Question 10

What is a likely consequence of overdifferencing?

A. It always removes all forecast uncertainty

B. It can create unnecessary moving-average behavior

C. It guarantees a better AIC

D. It makes every residual normally distributed

Correct Answer: B. It can create unnecessary moving-average behavior