Stochastic Processes, Martingales and Markov Models

Introduction

Quality, reliability, demand, defect occurrence and process performance evolve over time. A single probability distribution can describe uncertainty at one instant, but it cannot by itself represent dependence between successive observations. Stochastic-process theory supplies the framework for modelling such time-dependent uncertainty.

A stochastic process is a collection of random variables indexed by time, written {X_t:t∈T}. The index set T may be discrete, such as production cycles t=0,1,2,…, or continuous, such as calendar time t≥0. The state X_t may represent a machine condition, a quality rating, inventory level, number of failures or customer status.

Three related ideas are central. A Markov model describes systems whose future depends on the present state rather than the complete past. A martingale represents a fair-game or no-systematic-predictability condition relative to available information. A stationary distribution describes long-run probabilistic behaviour. Absorbing-state analysis identifies conditions from which a system cannot escape, such as permanent failure or successful completion.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Define stochastic processes, filtrations, state spaces and sample paths.
2. Distinguish independent, stationary, Markov and martingale behaviour.
3. Construct and interpret transition matrices for finite-state Markov chains.
4. Calculate multi-step transition probabilities and classify states.
5. Determine stationary distributions and assess convergence conditions.
6. Analyse absorbing chains, expected absorption times and absorption probabilities.
7. Apply these models critically to quality-management and operational decisions.

1. Foundations of stochastic processes

A stochastic process consists of random variables X_t defined on a common probability space. A sample path is one realised trajectory through time. The state space is the set of possible values of X_t. For example, a supplier may be classified as {Excellent, Acceptable, Watch, Disqualified}; this is a finite state space.

The process is discrete-time when observations occur at distinct times. It is continuous-time when changes may occur at any instant. This chapter concentrates mainly on discrete-time Markov chains, while noting continuous-time extensions.

A filtration {F_t} represents information available up to time t, with F_s⊆F_t for s≤t. The filtration is important because statements such as “the process is fair” must be made relative to what is known.

Important distinctions

• Independent observations: the past gives no probabilistic information about the future.
• Stationarity: the probabilistic structure is unchanged by shifts in time. Strict stationarity requires all finite-dimensional distributions to be invariant under time shifts; weak stationarity usually requires constant mean, constant variance and covariance depending only on lag.
• Markov property: conditional on the present, the future is independent of the past.
• Martingale property: conditional expectation of the next value equals the current value, given available information.

These properties are not equivalent. A Markov process need not be stationary; a stationary process need not be Markov; and a martingale need not have independent increments.

2. Conditional expectation and martingales

Let F_t contain the information available at time t. A process {M_t} is a martingale if:

The integrability condition ensures that the conditional expectation exists. The process is a submartingale if the conditional expectation is at least the current value, and a supermartingale if it is at most the current value.

A martingale does not mean that every observation remains constant. It means that, given current information, the expected future value equals the present value. Random fluctuations are still possible.

Example: a fair inspection score

Suppose a score changes by +1 with probability 0.6 and by −1 with probability 0.4. If the current score is M_t, then

The process is therefore a submartingale, not a martingale. A fair process would require equal expected upward and downward changes, or more generally a zero conditional expected increment.

Martingales are useful in sequential quality monitoring, risk analysis and stopping-time arguments. For example, a cumulative process formed from centred forecast errors may be expected to remain stable under an in-control process. Persistent upward movement can indicate a systematic shift.

Stopping times

A random time τ is a stopping time if the event “τ≤t” can be determined using information available by time t. Examples include the first time a control statistic exceeds a limit or the first machine failure. Optional-stopping results can be powerful, but they require conditions such as bounded stopping times, integrability or suitable uniform-integrability assumptions. It is unsafe to assume that every martingale remains unchanged at every arbitrarily selected stopping time.

3. Markov chains

A discrete-time process with state space S is a Markov chain if, for all states and times,

The right side is the one-step transition probability, denoted p_ij. For a finite state space with n states, the transition matrix is

Each entry satisfies 0≤p_ij≤1, and each row sums to one:

The row convention used here treats probability distributions as row vectors. If π₀ is the initial row distribution, then

Worked example: a three-state quality process

Consider states Good (G), Marginal (M) and Failed (F), with transition matrix

The state F is absorbing because its row is (0,0,1). If the process starts in G, the probability of being in F after two periods is the G-to-F two-step probability:

Thus the probability of failure by exactly the second transition, when starting in G and allowing the first state to be G, M or F, is 0.12. Because F is absorbing, this is also the probability of having reached failure by time 2.

Transition diagrams

A transition diagram represents states as nodes and non-zero transition probabilities as directed arrows. Self-loops indicate persistence. The matrix is usually more convenient for calculation, while the diagram is often better for communicating structure.

4. State classification and communicating classes

State i can reach state j if some power P^m has positive entry (P^m)_ij>0. States communicate when each can reach the other. Communication partitions the state space into communicating classes.

A class is closed if no transition leaves it. A state is absorbing if p_ii=1. In a finite chain, a state may be:

• Transient: it may be left and never revisited.
• Recurrent: once entered, it is returned to with probability one.
• Periodic: returns are restricted to multiples of a period greater than one.
• Aperiodic: the period is one.

For finite irreducible chains, all states are recurrent. A finite irreducible chain is ergodic when it is also aperiodic; then long-run probabilities converge to a unique stationary distribution.

5. Stationary distributions and long-run behaviour

A probability row vector π is stationary if

The equations state that the distribution is unchanged after one transition. Stationarity does not necessarily mean that an individual system remains still; individual paths continue to move while the population proportions remain stable.

Worked example: stationary quality proportions

For the two-state matrix

let π=(π_G,π_B). The balance equation for G is

Since π_B=1−π_G,

In the long run, approximately 80% of observations are in G and 20% in B, provided the chain is irreducible and aperiodic. These are probabilistic long-run proportions, not a guarantee for any short sample.

Detailed balance and reversibility

A sufficient condition for stationarity is detailed balance:

A chain satisfying this condition is reversible with respect to π. Detailed balance is stronger than stationarity and is not required for general Markov chains.

6. Absorbing Markov chains

An absorbing chain contains at least one absorbing state, and every transient state can reach an absorbing state. Reorder states so transient states come first and absorbing states last. Its transition matrix has canonical form

where Q contains transient-to-transient probabilities, R transient-to-absorbing probabilities, I is the identity matrix and 0 is a zero matrix.

The fundamental matrix is

when absorption occurs with probability one. The entry N_ij is the expected number of visits to transient state j when starting in transient state i, including the initial visit.

The expected time to absorption is

where 1 is a column vector of ones. Absorption probabilities are

Worked example: rework or scrap

Suppose a unit begins in state A (under inspection) or B (rework). It moves according to

where the third state is Scrap. Here

The matrix I−Q and its inverse are

Therefore, starting in A, the expected number of transitions before absorption is

The probability of eventual scrap from A is

Thus 87.5% of units eventually reach Scrap under this model, and the expected absorption time from A is five transitions. If “accepted” were added as another absorbing state, the same method would provide the probabilities of acceptance and scrap separately.

7. Estimation, validation and model use

Transition probabilities may be estimated from observed transitions. If n_ij is the number of observed transitions from state i to state j, and n_i=∑_j n_ij, the maximum-likelihood estimate is

provided n_i>0. This estimate assumes that transitions are correctly classified and that the transition mechanism is sufficiently stable over the observation period.

Practical validation should examine:

1. State definition: states must be mutually interpretable and operationally measurable.
2. Markov adequacy: test whether older history adds predictive information after conditioning on the current state.
3. Homogeneity: investigate whether transition probabilities vary by shift, product, supplier or operating regime.
4. Sampling uncertainty: report confidence intervals or Bayesian posterior uncertainty rather than treating estimates as exact.
5. Time scale: a monthly chain and a daily chain answer different questions.
6. Absorbing assumptions: permanent failure may be inappropriate if repair or re-entry is possible.

A chi-square goodness-of-fit comparison can assess whether observed transitions are compatible with a proposed matrix, but sparse cells reduce reliability. Bootstrap methods and Bayesian models can be useful when data are limited.

8. Application to total quality management

Markov models support quality decisions in several ways:

• Reliability: operating, degraded and failed states can produce expected failure probabilities.
• Preventive maintenance: compare transition matrices under different maintenance policies.
• Supplier development: model supplier ratings and estimate long-run proportions of acceptable deliveries.
• Customer experience: represent satisfaction states and estimate retention or churn.
• Corrective action: quantify whether a process is likely to return from an out-of-control state.
• Warranty planning: estimate movement from functioning to degraded and failed conditions.

The model should support, not replace, process knowledge. A high predicted failure probability may reflect an inadequate state definition, confounding by product mix or an unmeasured common cause. In quality management, the most valuable result is often not a single forecast but identification of transitions that improvement activity can change.

Illustrative case: semiconductor inspection escalation

Background. A fabrication facility classifies lots as Green, Yellow or Red after each inspection. Red lots require containment and are assumed to leave the normal flow. Historical data suggest

Problem. Managers want to know whether an early Yellow signal is merely temporary or likely to result in containment.

Analysis. Starting from Yellow, the probability of reaching Red on the next inspection is 0.20. The two-step probability is

Because Red is absorbing in this illustrative model, the probability of containment by two inspections is 29.95%. The two-step figure is substantially larger than the one-step figure because a Yellow lot may remain Yellow and then deteriorate, while a transition through Green can also lead to Red.

Approach. The quality team should verify whether Red lots truly leave the process permanently. If reinspection or recovery occurs, Red should not be absorbing. It should also estimate separate matrices by product family and shift, then compare a baseline policy with an intervention policy such as earlier process adjustment.

Outcome. The model identifies Yellow-to-Red and Yellow-to-Yellow transitions as priority targets. It does not establish causation; engineering investigation is required before changing control limits.

Learning points. State definitions determine conclusions, absorption assumptions must be scrutinised, and transition probabilities can translate inspection history into actionable prevention priorities.

9. Limitations, governance and contemporary developments

Limitations

The first-order Markov assumption may be unrealistic when accumulated damage, time since maintenance or batch effects matter. A higher-order chain, semi-Markov model or state augmentation can address this. Discretising a continuous quality measure may discard information and create artificial transitions. Non-stationary processes require time-varying or covariate-dependent transition probabilities.

Rare events create unstable estimates. An apparently small transition probability may be uncertain rather than safely negligible. Forecasts also inherit measurement error: misclassification of a state biases both estimated transitions and long-run conclusions.

Governance and ethics

Quality classifications can affect suppliers, employees and customers. Organisations should document state definitions, data provenance, estimation procedures and model limitations. Automated decisions should be auditable and contestable. If a model assigns a supplier to a high-risk state, responsible governance requires investigation of possible data bias and contextual causes rather than punitive action based solely on a probabilistic label.

Technology and AI

Modern systems can estimate transition models from sensor streams, survival data and digital-twin simulations. Hidden Markov models are useful when the true process state is unobserved but measurements are available. Reinforcement learning extends sequential decision-making by choosing actions to optimise long-run reward; however, its transition model and reward function must be validated. Machine-learning accuracy does not remove the need for calibration, causal reasoning, uncertainty quantification and human oversight.

Best-practice checklist

1. Define states using measurable, decision-relevant criteria.
2. Select a time interval consistent with the physical process.
3. Display the transition matrix and verify every row sums to one.
4. Test stationarity and Markov assumptions rather than asserting them.
5. Quantify estimation uncertainty and sensitivity to rare transitions.
6. Distinguish observed proportions from stationary predictions.
7. Validate absorbing-state assumptions with domain experts.
8. Use intervention analysis before claiming that a policy caused improvement.
9. Monitor model drift as products, equipment and operating conditions change.
10. Preserve auditability and explainability in consequential decisions.

Chapter Summary

Stochastic processes model uncertain systems evolving through time. Markov chains impose the useful conditional-independence assumption that the present state contains the information needed for predicting the next state. Their transition matrices support multi-step forecasts through matrix powers. State classification distinguishes transient, recurrent, periodic, aperiodic and absorbing behaviour. A stationary distribution satisfies πP=π and represents stable long-run proportions under suitable irreducibility and aperiodicity conditions.

Martingales formalise fair conditional evolution relative to an information filtration and are valuable in sequential monitoring and stopping-time analysis. Absorbing chains use the fundamental matrix N=(I−Q)^-1 to calculate expected visits and absorption times, while B=NR gives absorption probabilities. In total quality management, these methods can analyse reliability, supplier status, inspection escalation and maintenance policy. Their usefulness depends on valid states, appropriate time scales, adequate data, tested assumptions and responsible governance.

Key Terms

Stochastic process: A time-indexed collection of random variables.

State space: The set of possible values or conditions of a process.

Sample path: One realised trajectory of a stochastic process.

Filtration: Information available up to each time point.

Conditional expectation: Expected value given specified information.

Martingale: An integrable process whose next conditional expectation equals its current value.

Submartingale: A process with nondecreasing conditional expectation.

Markov property: Future conditional independence from the past given the present.

Markov chain: A discrete-time process satisfying the Markov property.

Transition probability: Probability of moving from one state to another in one step.

Transition matrix: Matrix containing all one-step transition probabilities.

Communicating class: States that mutually reach one another.

Transient state: A state that may be left without certain return.

Recurrent state: A state returned to with probability one.

Aperiodic state: A state whose return times have greatest common divisor one.

Stationary distribution: A probability distribution unchanged by the transition matrix.

Absorbing state: A state that, once entered, cannot be left.

Fundamental matrix: N=(I−Q)^-1 for a transient block of an absorbing chain.

Stopping time: A random time identifiable from information available by that time.

Hidden Markov model: A model in which states are latent and observations are probabilistically generated from them.

Review Questions

5 Short Answer Questions

1. What is the Markov property?
2. Distinguish stationarity from independence.
3. What condition makes a state absorbing?
4. Define a stationary distribution.
5. What does an entry of the fundamental matrix represent?

5 Long Answer Questions

1. Explain martingales, submartingales and supermartingales, including their relevance to sequential quality monitoring.
2. Describe how a transition matrix is constructed, interpreted and validated.
3. Explain transient, recurrent, periodic and aperiodic states and their implications for long-run behaviour.
4. Derive the fundamental-matrix method for expected absorption times.
5. Critically evaluate the use of first-order Markov chains in supplier-quality management.

3 Analytical / Application Questions

1. A two-state process has transition matrix P=[[0.8,0.2],[0.3,0.7]]. Find its stationary distribution and interpret the result.
2. For an absorbing chain with Q=[[0.5,0.2],[0.1,0.6]] and R=[[0.3],[0.3]], calculate N and the expected absorption time from the first transient state.
3. A factory’s model predicts rising failure risk, but the estimate is based on only 12 observed failures and state labels recorded inconsistently across shifts. Identify the statistical, modelling and governance actions required before changing the maintenance policy.

Multiple Choice Questions with Answers

Question 1

Which condition defines a martingale relative to filtration F_t?

A. E(M_t+1|F_t)=M_t

B. E(M_t+1|F_t)>M_t

C. All successive values are equal

D. All increments are independent and positive

Correct Answer: A. E(M_t+1|F_t)=M_t

Question 2

For a transition matrix, what must every row sum to?

A. Zero

B. One

C. The number of states

D. The stationary probability

Correct Answer: B. One

Question 3

If πP=π, then π is a:

A. Transient vector

B. Stopping time

C. Stationary distribution

D. Fundamental matrix

Correct Answer: C. Stationary distribution

Question 4

In an absorbing chain, what does N=(I−Q)^-1 provide?

A. Expected visits to transient states

B. Only one-step transition probabilities

C. The observation-error variance

D. The sample mean

Correct Answer: A. Expected visits to transient states

Question 5

A state with p_ii=1 is:

A. Periodic only

B. Transient

C. Absorbing

D. Independent

Correct Answer: C. Absorbing

Question 6

For P=[[0.9,0.1],[0.4,0.6]], the stationary probability of the first state is:

A. 0.2

B. 0.4

C. 0.6

D. 0.8

Correct Answer: D. 0.8

Question 7

The Markov property states that, given the present:

A. The past and future must be identical

B. The future is conditionally independent of the past

C. The process is stationary

D. All states are absorbing

Correct Answer: B. The future is conditionally independent of the past

Question 8

If a process has expected increment 0.2 conditional on current information, it is generally a:

A. Martingale

B. Submartingale

C. Supermartingale

D. Stationary distribution

Correct Answer: B. Submartingale

Question 9

For an absorbing chain in canonical form, R contains transitions from:

A. Absorbing to transient states

B. Transient to absorbing states

C. Absorbing to absorbing states only

D. A state to itself only

Correct Answer: B. Transient to absorbing states

Question 10

Which issue most directly challenges a first-order Markov model?

A. Rows of the matrix sum to one

B. The present state omits relevant accumulated damage

C. The state space is finite

D. Transition probabilities are nonnegative

Correct Answer: B. The present state omits relevant accumulated damage