E-Commerce Analytics, Experimentation and Decision-Making

Introduction and context

E-commerce management increasingly depends on the ability to convert digital traces into sound commercial decisions. Every visit, search, product view, add-to-cart event, payment attempt, return and seller interaction can generate evidence. Yet data alone does not create value. Managers must decide which measures matter, distinguish correlation from causation, account for profitability, and communicate uncertainty.

E-commerce analytics is the systematic collection, organisation, analysis and interpretation of data about digital commerce activities. It supports decisions concerning acquisition, merchandising, pricing, user experience, logistics, sellers, customer retention and investment. Marketplace analytics adds a second layer: the platform must evaluate not only customer demand but also seller quality, assortment health, fulfilment reliability, trust and the economics of intermediation.

A useful analytics system links four elements:

1. Measurement: defining events, entities, dimensions and metrics.
2. Explanation: identifying patterns through funnels, cohorts and attribution.
3. Causal learning: testing whether an intervention produced an outcome.
4. Decision-making: allocating resources in light of objectives, constraints and risk.

The central managerial principle is that a metric is valuable only when it supports a decision. A dashboard that displays hundreds of numbers without ownership, thresholds or action rules is a reporting system, not an analytics system.

Learning objectives

By the end of this chapter, students should be able to:

• construct a KPI framework for an e-commerce or marketplace business;
• calculate and interpret traffic, conversion, basket-size, unit-economic and profitability metrics;
• analyse funnels, cohorts and attribution without confusing descriptive and causal evidence;
• design and evaluate a practical A/B test;
• assess seller and marketplace performance;
• recognise data-quality, privacy, bias and governance risks; and
• translate analytical evidence into defensible managerial decisions.

1. From business objectives to KPI architecture

A KPI framework should begin with the business model and strategic objective, not with the data that happens to be available. A retailer may prioritise profitable repeat purchasing; a marketplace may prioritise liquidity, trust and balanced growth between buyers and sellers. The same metric can therefore have different meanings in different contexts.

A practical hierarchy is:

A north-star metric expresses a central customer or marketplace value proposition. It should not be selected merely because it is large or easy to increase. For example, “successful, on-time orders generating positive contribution” is generally more meaningful than gross visits.

A KPI should have a clear definition, owner, frequency, data source, target or benchmark, acceptable range, and documented action. Definitions must specify the numerator, denominator, population, time window and exclusions. “Conversion” might mean sessions resulting in an order, visitors ordering within a month, or checkout attempts resulting in payment. These are different measures.

Core traffic and conversion measures

Let S be the number of qualified sessions during a period, O the number of completed orders, V the number of unique visitors, and U the number of unique purchasers.

This measure assumes that sessions and orders are recorded consistently and that the order is attributed to the session under a stated rule. It is useful for diagnosing the performance of traffic and the purchase journey, but it is not a complete measure of customer value.

Because one customer may have several sessions or orders, purchaser conversion and session conversion should not be substituted for one another.

Other useful measures include traffic by source, product-detail-view rate, add-to-cart rate, checkout-start rate, payment-success rate, bounce or engagement rate, new-customer share and repeat-purchase rate. Traffic should be segmented by device, geography, customer status, category, campaign, seller and fulfilment method where sample sizes and privacy rules permit.

Basket size and sales measures

Let R be merchandise revenue from completed orders, O the order count, Q the number of units sold, and C the number of customers.

AOV can increase because customers buy more units, choose more expensive products, or pay higher prices. Analysts should therefore decompose it rather than assume that a higher AOV is automatically beneficial. Discounts, taxes, shipping charges and cancellations must be treated consistently. A marketplace should also distinguish gross merchandise value (GMV), the value of merchandise transacted, from net revenue, which may consist of commissions, listing fees, advertising income, fulfilment fees and other platform income.

Unit economics and profitability

Revenue growth can destroy value if variable costs grow faster than revenue. Let P represent item revenue, D discounts, T taxes collected on behalf of authorities, F fulfilment and shipping costs paid by the business, K payment costs, R_f refunds and chargebacks, and V_c variable selling or service costs.

A simplified net revenue measure is:

If taxes are not included in recorded revenue, T should not be subtracted again. Accounting policies must therefore be established before comparison.

These are contribution measures, not full accounting profit. Fixed salaries, technology, rent, depreciation and corporate overhead may remain outside them. Their value is operational: they help determine whether an order, customer segment, campaign or seller relationship contributes toward covering fixed costs.

A simplified customer acquisition cost is:

CAC is meaningful only when the attribution rule, acquisition window and definition of “new customer” are specified. Comparing CAC with customer lifetime value can guide investment, but lifetime value is an estimate requiring assumptions about retention, margin, discounting and future behaviour. It should be presented as a range or scenario, not as a precise fact.

For a customer with expected contribution margin m_t in period t, retention probability r_t, and discount rate i, a simplified discounted lifetime value is:

This formulation assumes the periods, margins, retention estimates and discount rate are defined consistently. It excludes or separately models non-recurring costs, returns and servicing costs. In practice, sensitivity analysis is essential because small changes in retention can materially change LTV.

2. Marketplace and seller performance analytics

A marketplace has two or more participant groups whose behaviour is interdependent. More buyers can attract sellers, while better assortment and fulfilment can attract buyers. Consequently, seller KPIs should be interpreted alongside customer and platform outcomes.

Important seller measures include:

• active sellers and active listings;
• listing completeness and content quality;
• in-stock rate and price competitiveness;
• impressions, click-through rate and product-page conversion;
• order acceptance and cancellation rates;
• on-time dispatch and delivery;
• defect, return, refund and complaint rates;
• seller response time;
• contribution after payment, fulfilment, support and incentives; and
• concentration of GMV among the largest sellers.

A seller quality score may combine dimensions, but weighted scores can conceal serious failures. A seller with excellent sales but unsafe products should not receive a favourable overall score. Therefore, some measures should be non-compensatory guardrails: fraud, counterfeit risk, safety violations and severe service failures can trigger suspension regardless of revenue.

Marketplace liquidity describes how effectively supply and demand are matched. Possible indicators include search-to-purchase success, percentage of listings receiving demand, time to first sale, stock availability and cancellation-free fulfilment. There is no universal liquidity formula; the appropriate measure depends on category, geography, delivery promise and customer mission.

3. Dashboards as decision systems

A dashboard should answer three questions: What happened? Why might it have happened? What should someone do next? A recommended dashboard architecture has four views:

1. Executive view: GMV or net sales, contribution profit, orders, conversion, AOV, repeat rate and major risks.
2. Growth view: traffic, channel efficiency, funnel stages, new versus returning customers and campaign performance.
3. Commercial view: category, product, price, promotion, stock, seller and margin performance.
4. Operations and trust view: fulfilment, payment failure, returns, complaints, fraud, service levels and data quality.

Dashboards require technical controls. Event names, time zones, currency conversions, identity stitching, bot filtering, late-arriving events and cancellation treatment must be documented. A sudden 20% increase in conversion may represent genuine improvement, a tracking failure, a changed denominator or duplicated orders.

Effective dashboards use trend lines, comparisons with a relevant baseline, segmentation and annotations for events such as campaign launches or stockouts. Targets should be accompanied by confidence or tolerance ranges where appropriate. Colour coding should signal action, not merely decorate the page.

Figure: Analytics is a feedback loop connecting objectives, measurement, analysis and action.

4. Funnel, cohort and attribution analysis

Funnel analysis

A funnel represents sequential stages such as landing, product view, add to cart, checkout and payment. For stage [[MATH,j]] with entrants [[MATH,n_j]], the stage conversion is:

The overall conversion from the first to final stage is final completions divided by initial entrants. Funnel analysis identifies where users leave, but it does not by itself explain why. A low payment-success rate may reflect fraud controls, technical errors, declined cards or customer confusion. Segmenting by device, country, payment method and browser often reveals the operational cause.

Funnels can also mislead when users do not follow a linear path, when tracking differs by device, or when the population entering each stage changes. Product analytics should therefore combine funnel evidence with qualitative research, session diagnostics, customer service records and controlled experiments.

Cohort analysis

A cohort is a group sharing a starting event or characteristic, such as acquisition month, first purchase category or seller onboarding month. Cohort tables compare retention, repeat purchase, contribution margin or refund behaviour over the same customer age.

For cohort [[MATH,c]], with [[MATH,N_c]] customers and [[MATH,A_(c,t)]] customers active in period [[MATH,t]] after acquisition:

Cohorts prevent a growing customer base from disguising deteriorating retention. Analysts must distinguish calendar time from customer age and avoid comparing immature cohorts with mature cohorts as though they had equal observation periods.

Attribution analysis

Attribution assigns credit for a conversion to marketing touchpoints. Common rules include first-touch, last-touch, linear, position-based and data-driven attribution. Each is a convention rather than a direct observation of causal contribution. Last-touch attribution, for example, may over-credit branded search or retargeting because these channels often occur near purchase.

Attribution is useful for organising activity and generating hypotheses, but budget decisions should increasingly rely on incrementality evidence: holdout tests, geo experiments, platform experiments or carefully designed quasi-experiments. Privacy restrictions, cookie loss, walled gardens and cross-device identity uncertainty make deterministic attribution less complete than it once appeared.

5. Designing and interpreting A/B tests

An A/B test randomly assigns eligible units—users, sessions, stores, sellers or regions—to a control group and a treatment group. Randomisation aims to balance observed and unobserved characteristics, so differences in outcomes can be attributed to the intervention, subject to implementation quality and statistical uncertainty.

A sound experiment specifies:

• a decision and primary hypothesis;
• the unit of randomisation;
• treatment and control definitions;
• one primary outcome and relevant guardrails;
• eligibility rules and exposure timing;
• minimum sample size or precision target;
• duration sufficient to capture relevant cycles;
• analysis method and stopping rule; and
• privacy, safety and rollback procedures.

For a binary outcome, the observed treatment effect is:

where p_T and p_C are treatment and control conversion proportions. Relative lift is:

Absolute change and relative lift answer different questions. If conversion rises from 4% to 4.4%, the absolute increase is 0.4 percentage points and the relative lift is 10%.

A confidence interval quantifies sampling uncertainty under a specified statistical model. For two sufficiently large independent proportions, a simple approximate standard error for the difference is:

where n_T and n_C are group sizes. An approximate 95% interval is the observed difference ± 1.96 × SE. This approximation can be unsuitable for very small samples, extreme proportions, clustered observations or repeated observations from the same user. In those settings, use an appropriate exact, regression, cluster-robust or hierarchical method.

Statistical significance is not the same as commercial significance. A tiny statistically reliable improvement may not cover implementation cost, while a large practical effect may be inconclusive because the experiment is underpowered. Analysts should report effect size, uncertainty, costs, guardrails and the decision threshold.

Common errors include peeking repeatedly and stopping when a desired result appears, changing the primary metric after seeing outcomes, running overlapping experiments that interfere, ignoring novelty effects, and testing too many segments without adjustment or replication. A negative result may mean the intervention failed, the test lacked power, exposure was inadequate, or implementation was defective.

6. Translating evidence into managerial decisions

Decision-making should combine evidence with economics, strategic fit, operational feasibility and risk. A useful decision memo contains:

1. Decision: what will be changed, continued or stopped?
2. Evidence: which data and methods support the conclusion?
3. Uncertainty: what alternative explanations or limitations remain?
4. Economics: what is the expected incremental contribution and cost?
5. Guardrails: what customer, seller, legal or trust outcomes must not deteriorate?
6. Action and review: who acts, by when, and how will the effect be monitored?

For example, a redesign that increases checkout conversion but also increases refunds and support contacts should not be scaled solely on conversion. A campaign with high attributed revenue but negative contribution after discounts and fulfilment should be reconsidered. A seller incentive may improve supply acquisition but worsen late delivery or price quality. Evidence becomes managerial knowledge only when the decision rule is explicit.

Worked example: campaign and unit economics

An online retailer spends $12,000 on a campaign. It records 600 new customers and 1,000 orders. Net revenue is $50,000. Variable product, fulfilment, payment and service costs total $38,000.

The campaign CAC is:

Contribution before campaign spend is:

Contribution after campaign spend is $0. The campaign therefore breaks even on the observed period contribution, although it may be economically attractive if future repeat purchases generate positive contribution. The correct next step is not automatically to increase spend; it is to examine customer cohorts, retention, incremental orders and future margin under realistic scenarios.

7. Applied case: an illustrative marketplace experiment

Background

“MarketHub,” a fictional general-merchandise marketplace, experiences strong traffic growth but stagnant contribution profit. Management observes that mobile conversion is lower than desktop conversion and that seller cancellations have increased. The executive dashboard reports GMV and orders but not contribution by seller or customer cohort.

Problem

The proposed solution is a one-click checkout redesign. However, the analytics team identifies three risks: the redesign may increase low-margin orders, payment fraud may rise, and the apparent mobile gap may be partly caused by slower delivery promises in mobile traffic regions.

Analysis

The team first audits event definitions and discovers that cancelled orders were excluded from the historical conversion denominator. It rebuilds the funnel using eligible sessions and separates successful orders from cancelled orders. It then compares cohorts by acquisition month and device. Mobile users show lower conversion, but the difference narrows substantially after controlling for category mix and delivery region.

Seller analysis shows that 8% of sellers generate 62% of GMV, creating concentration risk. These sellers have high sales but materially higher cancellation rates. The platform therefore adds seller cancellation and on-time fulfilment as guardrails.

Approach and solution

MarketHub randomises eligible mobile users to the existing checkout or the redesigned checkout. Randomisation occurs at user level, and users remain in the same group throughout the test. The primary metric is completed, non-cancelled orders per eligible user. Secondary metrics are contribution per user, payment failure, refund rate and checkout time. Fraud loss and seller cancellation are guardrails.

The test runs through two weekly cycles, with a pre-specified stopping rule. Results show a modest increase in completed orders, but the confidence interval includes zero. Contribution per user is slightly lower in treatment, while checkout time improves. The team does not scale the redesign. Instead, it investigates payment routing and seller reliability, then proposes a second experiment limited to categories with reliable fulfilment.

Outcome

The company avoids a premature rollout that might have increased low-quality orders. It also changes the executive dashboard to show contribution per eligible user, cancellation-adjusted conversion, cohort repeat contribution and seller service risk.

Learning points

• Correct metric definitions can change the apparent diagnosis.
• Conversion should be adjusted for order quality and cancellation.
• A test must identify the unit of value, not merely the easiest event to count.
• Guardrails protect customer trust and platform economics.
• An inconclusive test can still produce useful operational learning.

8. Limitations, governance and contemporary developments

Analytics is vulnerable to selection bias, survivorship bias, missing data, identity errors, seasonality, confounding promotions and feedback loops. Personalisation can create a self-fulfilling pattern: users shown a product buy it more often, but this does not prove they would not have bought an alternative. Marketplace ranking can also amplify incumbent sellers and suppress new entrants.

Privacy governance requires purpose limitation, data minimisation, retention controls, access management, security, lawful processing and transparent communication. Organisations should avoid using sensitive or proxy variables to discriminate in pricing, ranking, credit, seller admission or service access. Consent and legal compliance do not eliminate the need for ethical judgement.

AI and machine learning can support demand forecasting, anomaly detection, recommendation, fraud detection, segmentation and natural-language analysis of reviews. However, models require representative data, monitoring for drift, explainability proportionate to risk, human oversight and a clear appeal process. Automated decisions affecting sellers or customers should be auditable. Generative AI can assist with analytical summaries, but its outputs must be checked against source data; it should not invent causal claims or expose confidential information.

Contemporary analytics is moving toward server-side and first-party measurement, privacy-enhancing technologies, incrementality testing, real-time decisioning and unified customer-data platforms. These developments improve some forms of measurement but do not remove fundamental questions about causality, data quality or commercial relevance.

Best-practice checklist

• Begin with a business decision and define the value outcome.
• Document every KPI, denominator, time window and exclusion.
• Separate descriptive, predictive and causal claims.
• Reconcile operational dashboards with finance and accounting records.
• Use contribution and quality guardrails, not only volume metrics.
• Segment by meaningful customer, product, device, geography and seller dimensions.
• Use cohorts to study retention and attribution cautiously.
• Pre-register experiment hypotheses, primary metrics and stopping rules.
• Report effect size, uncertainty, cost and operational consequences.
• Monitor privacy, fairness, security, fraud and seller concentration.
• Treat dashboards as governed decision products with named owners.

Chapter Summary

E-commerce analytics connects data collection to commercial decisions. A robust KPI architecture begins with strategic objectives and distinguishes outcome, driver, diagnostic and guardrail metrics. Core measures include traffic, conversion, AOV, retention, contribution margin, CAC, LTV, GMV, seller quality and marketplace liquidity. Definitions must be consistent and reconciled with finance.

Funnels identify stage-level loss, cohorts reveal behaviour over customer age, and attribution organises touchpoints but does not prove causal contribution. Incrementality experiments are stronger evidence for budget and product decisions when randomisation, primary outcomes, guardrails and stopping rules are properly designed. Statistical significance must be interpreted alongside effect size, uncertainty, economics and feasibility.

Dashboards should support action rather than display numbers passively. Marketplace decisions require attention to both sides of the market, seller service and trust. Privacy, bias, security, data quality and AI governance are integral to responsible analytics. The ultimate purpose of analytics is disciplined learning: measure what matters, test important changes, recognise uncertainty and act in ways that improve sustainable customer and business value.

Key Terms

KPI: A key performance indicator used to monitor progress toward an important objective.

North-star metric: A central measure representing sustained customer or marketplace value.

Conversion rate: The proportion of a defined population completing a specified action.

Average order value: Revenue divided by the number of completed orders.

GMV: Gross merchandise value transacted through a marketplace before specified deductions.

Contribution margin: Net revenue minus relevant variable costs.

CAC: Acquisition spend divided by newly acquired customers under a stated attribution rule.

LTV: Estimated discounted future contribution from a customer, usually net of acquisition cost.

Funnel: A sequence of stages through which users progress toward a desired outcome.

Cohort: A group sharing a defined starting event or characteristic.

Attribution: Allocation of conversion credit across marketing touchpoints.

Incrementality: The additional outcome caused by an intervention compared with its counterfactual.

A/B test: A randomised comparison of a control condition and a treatment condition.

Guardrail metric: A measure used to prevent optimisation from causing unacceptable harm.

Statistical significance: Evidence, under a specified model, that an observed result is unlikely under a null hypothesis.

Marketplace liquidity: The effectiveness with which platform supply and demand are matched.

Seller concentration: The degree to which marketplace activity depends on a small number of sellers.

Data governance: Policies and controls for data quality, access, security, privacy and responsible use.

Review Questions

Short Answer Questions

1. Why should a KPI framework begin with a business objective rather than available data?
2. Distinguish GMV from net revenue.
3. What is the difference between absolute conversion change and relative lift?
4. Why can last-touch attribution over-credit some marketing channels?
5. Name four guardrail metrics relevant to a marketplace.

Long Answer Questions

1. Explain how an e-commerce dashboard can be designed as a decision system rather than a reporting display.
2. Discuss the benefits and limitations of funnel, cohort and attribution analysis.
3. Explain the principal design requirements of a credible A/B test.
4. Analyse why contribution margin and CAC should be interpreted together with retention and customer quality.
5. Discuss privacy, bias and governance risks in AI-supported marketplace analytics.

Analytical / Application Questions

1. A campaign produces $80,000 of net revenue, $58,000 of variable costs and costs $15,000 in media spend. Calculate contribution after media spend and explain what additional evidence is needed before scaling the campaign.
2. A treatment group has 5,200 users and 260 completed orders; a control group has 5,000 users and 225 completed orders. Calculate each conversion rate, the absolute difference and the relative lift. Identify at least two limitations of interpreting the result without an uncertainty interval.
3. A marketplace reports rising conversion but also rising cancellations, refunds and seller complaints. Design a diagnostic analysis and recommend a decision rule for determining whether to retain, modify or stop the intervention.

Multiple Choice Questions with Answers

Question 1

Which measure is most directly a contribution-level profitability measure?

A. Page views

B. Gross merchandise value

C. Contribution margin

D. Session count

Correct Answer: C. Contribution margin

Question 2

If 500 orders arise from 10,000 qualified sessions, the session conversion rate is:

A. 0.5%

B. 5%

C. 10%

D. 20%

Correct Answer: B. 5%

Question 3

What is the main purpose of cohort analysis?

A. To allocate all revenue to the last marketing touchpoint

B. To compare customers at similar ages since a starting event

C. To remove all uncertainty from forecasts

D. To measure only current website traffic

Correct Answer: B. To compare customers at similar ages since a starting event

Question 4

In an A/B test, randomisation primarily helps to:

A. Guarantee a profitable result

B. Eliminate the need for a control group

C. Balance confounding characteristics between groups on average

D. Make every metric statistically significant

Correct Answer: C. Balance confounding characteristics between groups on average

Question 5

A conversion rate rises from 4% to 4.4%. What is the relative lift?

A. 0.4%

B. 4%

C. 10%

D. 40%

Correct Answer: C. 10%

Question 6

Which is an appropriate marketplace guardrail metric?

A. Seller cancellation rate

B. Raw page impressions only

C. Total visits without quality checks

D. Number of dashboard colours

Correct Answer: A. Seller cancellation rate

Question 7

Why is attribution not necessarily evidence of causation?

A. Touchpoints cannot be recorded digitally

B. Credit-assignment rules may not show what would have happened without the touchpoint

C. Customers never use more than one channel

D. All attribution models produce identical results

Correct Answer: B. Credit-assignment rules may not show what would have happened without the touchpoint

Question 8

If net revenue is $50,000 and variable costs are $38,000, contribution margin before fixed costs is:

A. $12,000

B. $38,000

C. $50,000

D. $88,000

Correct Answer: A. $12,000

Question 9

Which practice improves the credibility of an experiment?

A. Changing the primary metric after seeing the results

B. Stopping whenever the result first looks favourable

C. Pre-specifying the primary outcome and stopping rule

D. Ignoring refunds because they occur after purchase

Correct Answer: C. Pre-specifying the primary outcome and stopping rule

Question 10

A dashboard metric suddenly improves after an event-tracking change. The first appropriate response is to:

A. Immediately increase the marketing budget

B. Assume customer behaviour has permanently improved

C. Audit the definition, denominator and data pipeline

D. Delete the metric from the dashboard

Correct Answer: C. Audit the definition, denominator and data pipeline