Demand Data, Measurement and Market Intelligence

Introduction

Demand forecasting and pricing strategy are only as reliable as the data on which they are based. A sophisticated forecasting model cannot correct a dataset that confuses sales with demand, ignores stockouts, mixes nominal and real prices, or violates customer privacy. For marketing managers, the central task is therefore not merely collecting more data, but constructing a defensible measurement system that represents market behavior, commercial conditions, and managerial decisions.

Demand data arise from transactions, customer relationship management (CRM) systems, loyalty programs, market panels, surveys, websites, mobile applications, advertising platforms, competitors, macroeconomic databases, and operational systems. These sources differ in coverage, frequency, accuracy, cost, privacy risk, and susceptibility to bias. Market intelligence is the disciplined process of integrating these sources to understand what customers purchased, what they wanted but could not purchase, why behavior changed, and how demand may respond to price, promotion, availability, and external conditions.

This chapter develops a framework for building reliable demand datasets. It addresses measurement design, aggregation, seasonality, stockouts, censoring, missing values, outliers, inflation, data quality, privacy, and managerial use.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish observed sales from underlying customer demand.
2. Evaluate transaction, CRM, panel, survey, web, and external data sources.
3. Design a time-, product-, customer-, and location-based demand dataset.
4. Diagnose aggregation, seasonality, stockout, censoring, missing-data, and outlier problems.
5. Convert nominal measures into comparable real measures when appropriate.
6. Apply data-quality and privacy-governance principles.
7. Translate market intelligence into forecasting and pricing decisions.

1. Demand as a Measurement Problem

In its broadest sense, demand is the quantity of a product or service that customers are willing and able to purchase under specified conditions. Observed sales are not necessarily equal to demand. A simple representation is:

where Q^obs_i,t is observed quantity for product or market segment i in period t, Q^dem_i,t is desired demand, and Q^avail_i,t is available supply. The relationship assumes that sales cannot exceed available inventory or capacity and that unfulfilled demand is not automatically observed. It is most useful as a conceptual starting point; actual sales may also be constrained by payment failure, delivery limits, eligibility rules, or distribution coverage.

A sales record commonly includes product, quantity, price, discount, channel, customer, time, location, and transaction status. A demand record should additionally describe availability, exposure, competition, promotion, and relevant customer or market conditions. Without these variables, analysts may interpret a fall in sales as weak demand when it was actually caused by a stockout or reduced website visibility.

Units of analysis

A demand dataset should define its unit of observation explicitly. Common units include:

• transaction line: one product on one order;
• customer-period: one customer’s activity during a defined period;
• product-store-day: one product at one location on one day;
• product-market-week: aggregated demand in a geographic market;
• offer impression: one customer exposure to a price or promotion;
• survey respondent: one observation of stated attitudes or intentions.

The appropriate unit depends on the decision. Inventory replenishment may require product-store-day data, while annual pricing strategy may use product-market-month data. The unit must be stable enough to support comparison and sufficiently detailed to preserve relevant variation.

2. Major Sources of Demand Data

Transaction and point-of-sale data

Transaction data provide revealed purchasing behavior. They are typically time-stamped and may contain quantities, prices, discounts, taxes, returns, payment methods, and product identifiers. Their strengths are high behavioral validity, detailed temporal resolution, and direct connection to revenue. Their limitations include unobserved nonpurchase, incomplete customer identification, recording errors, and the fact that sales reflect supply constraints.

Returns require special treatment. Gross units sold, returned units, and net units should not be silently combined. A useful reconciliation is:

where all quantities must use the same product definition and period. If returns occur weeks after purchase, cohort-based analysis may be preferable to simple period subtraction.

CRM and loyalty data

CRM systems connect transactions with customer attributes, contacts, service interactions, campaigns, preferences, and retention outcomes. They support customer lifetime analysis, segmentation, churn modeling, and personalized pricing or promotion. However, CRM data are often selective: identified customers may differ from anonymous customers, and fields such as occupation or income may be outdated or self-reported. Customer identifiers should be stable, pseudonymized where possible, and governed by explicit access rules.

Panels and scanner data

Consumer panels repeatedly observe a sample of households or individuals. Retail scanner panels may provide category purchases across participating retailers. These sources improve market coverage and enable competitor or category analysis, but they involve sampling error, panel attrition, reporting burden, and weighting assumptions. Analysts should distinguish the sample size from the effective population coverage.

Surveys and stated-preference data

Surveys measure awareness, attitudes, preferences, purchase intentions, price perceptions, and reasons for nonpurchase. They are particularly valuable for new products, unavailable alternatives, and motivations that transactions cannot reveal. Stated intentions frequently exceed realized behavior, so survey findings should be validated against behavioral data where possible. Question wording, order effects, social desirability, nonresponse, and poorly defined price scenarios can create systematic bias.

Web and digital analytics

Web analytics include visits, sessions, search terms, product views, cart additions, checkout starts, conversions, referral sources, device types, and exposure to digital promotions. These data reveal consideration and funnel behavior before purchase. They are not equivalent to demand: automated traffic, multiple devices, cookie loss, ad-blocking, attribution rules, and privacy restrictions affect measurement. A conversion rate is commonly defined as:

where N^conv is the number of defined conversions and N^eligible is the number of eligible sessions, users, or exposures. The denominator must be specified because session-based and user-based conversion rates answer different questions.

External data

External sources include competitor prices, weather, holidays, population, income, interest rates, consumer confidence, search trends, mobility, regulation, and industry capacity. They can explain demand shifts and improve scenario planning. Their principal risks are mismatched geography, frequency, definitions, revision schedules, and access rights. A macroeconomic indicator should not be joined to a firm-level dataset merely because both contain a date; the economic mechanism and geographic scope must also be defensible.

3. Measurement Architecture and Data Integration

A professional demand dataset requires a data dictionary, a common calendar, master data, and a documented integration process. The data dictionary defines each field, unit, permissible values, source, owner, refresh rate, and transformation history. Master data establish consistent product, customer, store, channel, and geography identifiers.

A typical architecture contains:

1. source systems, such as POS, CRM, web analytics, and external providers;
2. an ingestion layer that records raw data without overwriting history;
3. validation and transformation processes;
4. an analytical warehouse or lakehouse;
5. curated demand tables and metadata;
6. dashboards, forecasting models, and decision workflows.

Integration should preserve lineage: an analyst must be able to identify where a number originated, which transformations were applied, and when a correction occurred. Slowly changing product attributes, such as pack size or category membership, require versioned records rather than retroactively rewriting historical meaning.

Aggregation and the ecological problem

Aggregation reduces storage and noise but can conceal important variation. Daily sales aggregated to monthly totals may hide payday effects, promotions, and stockouts. Customer-level data aggregated to market averages may conceal different price responses among segments. Conversely, excessive disaggregation creates sparse, volatile series and raises privacy risk.

An aggregation rule should specify the operation. Quantities and revenue are generally additive across compatible units; prices are not. A revenue-weighted average price for period t is:

where R_t is total revenue and Q_t is total quantity, both for the same product definition and period. A simple arithmetic average of transaction prices may be misleading when transaction quantities differ. This formula assumes quantity is measured in comparable units and that revenue excludes or consistently includes taxes, shipping, and discounts.

Aggregation also creates the ecological fallacy: relationships observed for markets or groups may not hold for individuals. For example, a market with higher average income may show higher premium-product sales, but this does not establish that every high-income customer prefers premium products.

4. Seasonality, Calendar Effects, and Structural Change

Seasonality is a recurring pattern associated with calendar position, weather, institutional schedules, or cultural events. It should be distinguished from trend, which is persistent long-term movement, and from irregular shocks, such as a strike or sudden regulatory change.

A seasonal index can be calculated, under suitable assumptions, as:

where S_m is the index for month m, Ȳ_m is the average demand for that month across comparable years, and Ȳ is the overall monthly average. An index above 1 indicates above-average demand. The method assumes that the seasonal pattern is reasonably stable and that trend, promotions, price changes, and structural breaks have been addressed. More advanced models estimate seasonal effects jointly with trend and explanatory variables.

Calendar effects include the number of trading days, weekends, public holidays, school terms, religious events, and the timing of paydays. Analysts should avoid treating a moving holiday as fixed. Structural change occurs when the data-generating process changes, for example after a platform redesign, competitor entry, pandemic, or change in distribution. Historical observations may then require segmentation, intervention variables, or reduced weighting rather than blind continuation.

5. Stockouts, Censoring, and Unobserved Demand

Stockouts censor demand because the observed quantity is bounded by availability. A zero-sales day may mean zero interest, unavailable inventory, an incorrectly displayed product, or a closed store. Useful fields include opening inventory, receipts, closing inventory, in-stock minutes, lost-sales flags, search-without-purchase events, substitutions, and backorders.

Censoring also occurs when the dataset records only purchases above a threshold, only selected customers, or only transactions completed through a particular channel. Truncation differs from censoring: truncation excludes observations from the dataset entirely, whereas censoring retains partial information about them. The distinction affects statistical treatment.

A practical diagnostic is to compare sales with availability and exposure. If demand rises when the product is visible and in stock but sales fall during low availability, the apparent sales decline should not be interpreted as a price or preference effect. Imputing lost sales requires assumptions. Simple approaches use comparable stores or periods; more advanced approaches estimate demand conditional on price, promotion, and availability. All imputed observations should be flagged and accompanied by uncertainty rather than presented as observed facts.

6. Missing Values and Outliers

Missingness has several forms:

• MCAR: missing completely at random;
• MAR: missingness related to observed variables;
• MNAR: missingness related to the unobserved value itself.

The distinction matters. A randomly lost sensor record is different from customers refusing to report income or a retailer failing to transmit data specifically during high-volume periods. Missingness should first be described, then investigated through source logs, time patterns, field relationships, and business processes.

Common responses include source correction, explicit missing categories for categorical variables, carefully justified imputation, model-based treatment, and sensitivity analysis. Mean imputation often understates variability and can distort relationships. Imputed values should carry an indicator showing that they were imputed.

Outliers may be errors, legitimate exceptional events, or evidence of structural change. A transaction of 10,000 units could be a key-account order, a duplicated record, or a unit-conversion problem. Detection tools include validation rules, box plots, robust statistics, duplicate checks, and comparison with operational context. A z-score is:

where x_i is an observation, x̄ is the sample mean, and s is the sample standard deviation. It is useful under approximately symmetric distributions but is sensitive to extreme values; robust measures such as the median and median absolute deviation may be preferable for skewed commercial data. Outliers should not be deleted solely because they are inconvenient.

7. Inflation, Prices, and Comparable Value

Nominal revenue reflects current monetary units. For comparisons across time, analysts may need real revenue, which removes general price-level changes. If a price index has base-period value I₀ and period value I_t, real revenue in base-period currency is:

The index must use an appropriate basket, geography, base, and frequency. This conversion addresses economy-wide inflation; it does not remove product-specific quality changes, package-size changes, discounts, or mix effects. A price increase may reflect inflation, improved quality, premium mix, or managerial strategy. Analysts should therefore distinguish list price, realized transaction price, unit price, and price per standardized quantity.

8. Data Quality and Validation

Data quality is multidimensional. Important dimensions include accuracy, completeness, consistency, timeliness, validity, uniqueness, and fitness for purpose. A dataset can be complete but inaccurate, or accurate but too late for a pricing decision.

A validation framework should include:

• range checks, such as nonnegative quantity and plausible discount limits;
• referential integrity, such as every product code matching a master record;
• reconciliation, such as sales plus returns agreeing with ledger totals;
• temporal checks, such as no future transaction dates;
• duplicate detection using order, product, timestamp, and customer keys;
• distribution monitoring for sudden shifts in volume or price;
• source-level monitoring of missingness and transmission failures;
• documented exceptions, owners, deadlines, and resolution status.

Quality metrics should be tied to decision risk. A one-hour delay may be irrelevant for annual assortment planning but unacceptable for dynamic airline pricing. Data stewardship should assign accountability to named business and technical owners rather than treating quality as an analyst’s private responsibility.

9. Privacy, Ethics, and Governance

Demand intelligence can expose identity, location, income, health-related interests, browsing behavior, and purchasing vulnerabilities. Responsible practice follows principles such as purpose limitation, data minimization, transparency, accuracy, security, retention control, and accountability. Applicable law varies by jurisdiction; organizations must comply with relevant privacy and consumer-protection requirements rather than assuming that commercial data are unrestricted.

Practical safeguards include pseudonymization, aggregation, role-based access, encryption, retention schedules, consent management where required, vendor due diligence, and suppression of small cells that could enable re-identification. Pseudonymization reduces direct identification risk but does not make data anonymous if linkage remains possible.

Ethical pricing questions arise when data enable differential treatment based on sensitive or proxy characteristics. A model may produce unequal outcomes even without explicitly using protected attributes because browsing history, postcode, device, or purchase patterns can act as proxies. Governance should include impact assessment, explainability appropriate to the decision, human review, audit logs, appeal mechanisms, and monitoring for disparate outcomes.

10. Market Intelligence and AI-Enabled Analysis

Market intelligence is more than a dashboard. It links evidence to a decision: which segment to serve, how much inventory to position, whether a price change is justified, or why conversion has declined. A useful intelligence brief states the question, evidence, uncertainty, recommended action, and monitoring plan.

AI can assist with anomaly detection, natural-language classification of reviews, entity resolution, demand sensing, and automated data-quality alerts. Yet AI outputs inherit source bias and may create false confidence. Models should be evaluated against out-of-sample data, monitored after deployment, and reviewed for leakage, drift, unfairness, and inappropriate use of personal data. Generative systems should not be allowed to invent missing observations or provide unsupported causal explanations.

11. Illustrative Case: FreshBite Grocery

Background

FreshBite, a regional grocery chain, observed declining weekly sales for a premium ready-meal range. Managers proposed discounting the range permanently.

Problem

The initial dashboard used product-week sales and average transaction price but excluded in-stock minutes, shelf position, promotions, competitor prices, and returns. Several stores had introduced a new replenishment system during the same period.

Analysis

The data team reconciled POS records with inventory and planogram files. They found that two high-volume distribution centers had experienced repeated stockouts, while the online catalog displayed the products as available only intermittently. Sales declined primarily in stores with low availability. A customer survey also showed that the product was attractive, but respondents complained about finding it on the shelf. Web searches and product-page views remained stable, while add-to-cart rates fell during stockout periods.

The team separated observed sales from availability-constrained periods, standardized package sizes, corrected duplicated return records, and compared realized prices rather than list prices. Competitor data showed no major price advantage sufficient to explain the decline.

Approach and outcome

FreshBite improved replenishment parameters, created an availability-adjusted performance report, and tested a modest targeted promotion only in stores with reliable inventory. Sales recovered without a permanent price reduction. The company avoided teaching customers to wait for discounts and improved the credibility of its forecasting data.

Learning points

1. Sales decline is not proof of demand decline.
2. Availability must be joined to transaction data.
3. Price analysis requires consistent units and realized prices.
4. Operational changes can create structural breaks.
5. Market intelligence is valuable when it changes a decision, not merely when it produces a report.

Best-Practice Checklist

Before using a demand dataset, ask:

• Is the unit of observation explicit?
• Are sales distinguished from demand and availability?
• Are product, customer, location, and calendar definitions stable?
• Have returns, promotions, stockouts, and exposure been represented?
• Are missing values, outliers, and imputed records documented?
• Are monetary values comparable across time?
• Can every important figure be traced to its source?
• Is the data use lawful, proportionate, secure, and ethically defensible?
• Are uncertainty and limitations communicated to decision-makers?

Chapter Summary

Reliable demand forecasting begins with measurement. Observed sales are constrained by availability and therefore may understate desired demand. Transaction, CRM, panel, survey, web, and external data each provide different forms of evidence and introduce distinctive biases. Effective data architecture uses clear units of analysis, master data, common calendars, aggregation rules, and lineage.

Seasonality must be separated from trend, calendar effects, and structural change. Stockouts and censoring require availability measures and cautious estimation. Missing values and outliers should be investigated in relation to the business process rather than mechanically deleted. Inflation and package-size changes require consistent monetary and physical units. Data-quality programs should monitor accuracy, completeness, consistency, timeliness, validity, and uniqueness. Privacy, fairness, security, and accountability are essential parts of market intelligence, particularly when AI and customer-level data are used. The managerial objective is not maximal data volume, but decision-relevant, traceable, comparable, and ethically governed evidence.

Key Terms

Demand: Quantity customers are willing and able to purchase under specified conditions.

Observed sales: Recorded transactions, which may be limited by availability.

Stockout: A condition in which requested or expected inventory is unavailable.

Censoring: Partial observation of a value because a measurement boundary limits what is recorded.

Truncation: Exclusion of observations from the dataset based on a selection rule.

Aggregation: Combining observations into broader time, product, customer, or geographic units.

Seasonality: Recurrent variation associated with calendar or environmental cycles.

Structural break: A change in the process generating the data.

CRM data: Customer relationship records linked to interactions, attributes, and transactions.

Panel data: Repeated observations from a defined sample of households, individuals, or firms.

Stated preference: Reported intention or evaluation rather than observed behavior.

Conversion rate: Conversions divided by a specified eligible population or exposure base.

Nominal revenue: Revenue measured in current monetary units.

Real revenue: Revenue adjusted to a reference price level.

Missingness: The condition in which a variable is not recorded for an observation.

Outlier: An observation unusually distant from the prevailing pattern.

Data lineage: Documentation of a value’s source and transformations.

Pseudonymization: Replacing direct identifiers while retaining a controlled linkage capability.

Market intelligence: Decision-oriented interpretation of internal and external market evidence.

Review Questions

5 Short Answer Questions

1. Why are observed sales not always equal to demand?
2. Distinguish censoring from truncation.
3. Why can an arithmetic average transaction price be misleading?
4. Name four dimensions of data quality.
5. What is the purpose of data lineage?

5 Long Answer Questions

1. Compare transaction, CRM, panel, survey, web, and external data in terms of strengths, limitations, and appropriate managerial uses.
2. Explain how aggregation can improve analysis while also creating ecological fallacy and masking operational variation.
3. Discuss how stockouts, returns, and promotions should be represented in a demand dataset.
4. Explain the difference between nominal and real revenue and identify the assumptions behind inflation adjustment.
5. Develop a governance framework for using customer-level data in pricing decisions.

3 Analytical / Application Questions

1. A retailer reports falling sales for a product, but product-page views are stable and in-stock minutes have declined sharply. Design an analysis to determine whether demand or availability caused the decline.
2. Construct a product-store-week data specification for forecasting. Identify required fields, validation rules, and at least three potential sources of bias.
3. A firm wants to use browsing behavior to personalize prices. Evaluate the commercial benefits, privacy risks, fairness concerns, and controls that should be established before deployment.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes observed sales from underlying demand?

A. Observed sales always equal customer willingness to buy.

B. Observed sales may be limited by product availability.

C. Observed sales exclude all effects of price.

D. Observed sales measure only stated preferences.

Correct Answer: B. Observed sales may be limited by product availability.

Question 2

A product-store-day record is an example of which unit of analysis?

A. A customer cohort

B. A transaction line

C. A product-location-time observation

D. A survey response

Correct Answer: C. A product-location-time observation

Question 3

Why is a revenue-weighted average price generally preferable to a simple average of transaction prices when quantities differ?

A. It ignores quantities and discounts.

B. It reflects total revenue divided by total quantity.

C. It converts nominal revenue into real revenue.

D. It removes all seasonal effects.

Correct Answer: B. It reflects total revenue divided by total quantity.

Question 4

If a product is out of stock, recorded zero sales most directly represent:

A. Confirmed zero demand

B. A price elasticity estimate

C. A potentially censored demand observation

D. A completed customer survey

Correct Answer: C. A potentially censored demand observation

Question 5

Which situation is most likely an example of a structural break?

A. A predictable increase in holiday demand

B. A weekly weekend pattern

C. A permanent change after a platform redesign

D. Random variation among transactions

Correct Answer: C. A permanent change after a platform redesign

Question 6

What does the formula CR = N^conv ÷ N^eligible measure?

A. Inventory turnover

B. Conversion rate

C. Real revenue

D. Seasonal intensity

Correct Answer: B. Conversion rate

Question 7

Which practice best supports data lineage?

A. Overwriting corrected historical files without records

B. Removing source-system identifiers

C. Documenting sources and transformations applied to a value

D. Combining all fields into one unlabelled table

Correct Answer: C. Documenting sources and transformations applied to a value

Question 8

Why may mean imputation be problematic for missing commercial data?

A. It always increases observed variance.

B. It can understate variability and distort relationships.

C. It guarantees that missingness is random.

D. It identifies whether data are MNAR.

Correct Answer: B. It can understate variability and distort relationships.

Question 9

What is the main purpose of converting nominal revenue into real revenue?

A. To remove all product-mix effects

B. To compare monetary values using a reference price level

C. To estimate customer identity

D. To correct duplicate transaction records

Correct Answer: B. To compare monetary values using a reference price level

Question 10

Which is an appropriate privacy safeguard for customer-level market intelligence?

A. Unlimited retention of identifiable records

B. Public disclosure of small customer cells

C. Role-based access and pseudonymization

D. Using sensitive attributes without governance

Correct Answer: C. Role-based access and pseudonymization