Data Preparation, Analysis and Interpretation

Introduction

Market research creates value only when raw observations are transformed into trustworthy evidence for decisions. Survey responses, interview records, transaction data, web analytics and competitor databases commonly contain missing values, inconsistent categories, duplicate cases and measurement errors. Analysis cannot compensate for fundamentally poor data. This chapter presents a disciplined workflow for preparing, analysing and interpreting market-research data. It covers coding, data quality, descriptive statistics, visualisation, confidence intervals, hypothesis testing, correlation and regression, while emphasising the distinction between association and causation.

For entrepreneurs, the purpose of analysis is not to produce impressive tables. It is to answer commercially relevant questions such as: Who is the target customer? How large is a segment? Which product attributes influence preference? Is an observed difference likely to reflect a real market pattern? What uncertainty should be communicated to investors or managers?

Learning Objectives

By the end of this chapter, you should be able to:

• construct a data dictionary and prepare a defensible analysis file;
• identify and manage missing, inconsistent, duplicate and implausible observations;
• code categorical and qualitative responses appropriately;
• select suitable descriptive statistics and visualisations;
• calculate and interpret confidence intervals and hypothesis tests;
• distinguish correlation from causation;
• use and interpret simple regression responsibly;
• communicate findings with appropriate uncertainty, limitations and ethical safeguards.

1. The Data-to-Decision Workflow

A professional workflow normally includes the following stages:

1. Define the decision and research questions.
2. Identify the unit of analysis: person, household, organisation, transaction, product or market.
3. Inspect the data structure and create a data dictionary.
4. Validate, clean and document the data.
5. Code variables and construct justified derived measures.
6. Explore distributions and relationships visually and numerically.
7. Apply inferential methods where sampling and design justify them.
8. Interpret results in relation to the research question and business context.
9. Report uncertainty, limitations and implications.
10. Preserve an auditable record of transformations and decisions.

The unit of analysis matters. If a survey contains several responses from each customer, treating every response as an independent customer observation can understate uncertainty. Similarly, analysing product reviews as though they were randomly sampled consumers may produce misleading generalisations.

A data dictionary should record variable name, label, type, coding, unit, permissible values, source, missing-value convention and transformation history.

2. Data Preparation and Quality Assurance

2.1 Structural checks

Begin by checking the number of rows and columns, variable types, identifiers, date formats and expected ranges. A numeric field imported as text can prevent correct calculations. Dates should be stored consistently, and currency values should identify the currency and time period. Unique identifiers help detect duplicated records, but two records with the same customer identifier may be legitimate repeated observations rather than errors.

2.2 Missing data

Missingness may be:

• structural, when a question does not apply;
• unit non-response, when an entire case is absent;
• item non-response, when a respondent omits one item;
• unknown, when the reason is not recorded.

Do not automatically code missing values as zero. A missing income response does not mean zero income. Common approaches include complete-case analysis, reporting missingness as a category, simple imputation and model-based multiple imputation. The appropriate choice depends on the mechanism and amount of missingness. If respondents with low trust systematically omit a satisfaction question, the missing data may bias results.

Report the proportion missing for important variables. Where imputation is used, document the method and test whether conclusions change under plausible alternatives.

2.3 Invalid, inconsistent and duplicate records

Validation rules may identify ages outside the sampling frame, impossible dates, negative quantities or responses that violate skip logic. Correction should use original source records where possible. Do not silently overwrite questionable values; preserve the raw file and create a documented cleaned version.

Duplicates can arise from repeated uploads, multiple submissions or legitimate longitudinal observations. Deduplication requires a rule based on research design, such as retaining the most complete record or the earliest verified submission. Every exclusion should be logged.

2.4 Outliers

An outlier is an observation unusually distant from the rest. It may be an error, a rare but genuine customer or a consequence of a skewed distribution. Outliers should not be removed merely because they weaken a preferred conclusion. Investigate source accuracy, use robust summaries, segment the analysis or apply a justified transformation. A boxplot can flag potential outliers, but it does not prove that they are erroneous.

2.5 Coding and measurement

Nominal categories have no inherent order, such as payment method. Ordinal categories have order but not necessarily equal intervals, such as satisfaction ratings. Interval measures have meaningful differences but no true zero, while ratio measures have meaningful differences and a true zero, such as units sold.

Binary coding is useful for analysis, but labels must remain clear. Coding “yes” as 1 and “no” as 0 is conventional; it does not make a binary variable continuous in a substantive sense. For nominal variables with more than two categories, regression commonly uses indicator variables, with one category serving as a reference.

A composite scale may combine several items intended to measure a construct such as perceived value. Before averaging items, confirm that they represent the same construct, align their direction and evaluate reliability. A commonly used internal-consistency measure is Cronbach’s alpha:

Here, k is the number of items, sᵢ² is the variance of item i and s_T² is the variance of the total score. Alpha is not proof of validity or unidimensionality; a high value can result from redundant items. Its interpretation depends on the construct, number of items and research purpose.

3. Descriptive Analysis

Descriptive statistics summarise the observed sample; they do not automatically describe the entire market. The arithmetic mean is:

where xᵢ is observation i and n is the number of non-missing observations. The mean is sensitive to extreme values. The median is the middle ordered value and is often more appropriate for skewed income, spending or waiting-time data. The mode is the most frequent category or value.

The sample variance and standard deviation are:

The denominator n − 1 is used for the usual unbiased estimate of population variance under random sampling. Standard deviation is expressed in the same units as x. The interquartile range, IQR = Q₃ − Q₁, describes the central 50% of observations and is less sensitive to extreme values.

For categorical variables, report counts and percentages. Percentages should state their denominator, especially when missing responses differ across items. For example, “42 of 100 valid responses (42%) selected option A” is clearer than “42%” alone.

Visualisation

Use a chart that matches the variable and question:

Good charts use informative titles, labelled axes, consistent units, readable scales and source notes. Avoid truncated axes when they exaggerate differences, three-dimensional effects and decorative elements that obscure values. Visualisation is exploratory: an apparent pattern should be checked against counts, measurement quality and sampling design.

4. Sampling, Estimation and Confidence Intervals

Inferential statistics use sample evidence to estimate population characteristics. A confidence interval communicates both an estimate and sampling uncertainty. For a population mean, when observations are independent and the population standard deviation is unknown, a commonly used interval is:

Here, x̄ is the sample mean, s is the sample standard deviation, n is sample size and t* is the critical value from the t distribution with n − 1 degrees of freedom. The formula assumes an appropriate sampling design and, for small samples, an approximately normal population or a method robust to non-normality.

For a proportion p̂, a large-sample approximate interval is:

where z* is the standard-normal critical value. This approximation can perform poorly with small samples or proportions near 0 or 1; Wilson or exact methods may be preferable.

A 95% confidence interval does not mean there is a 95% probability that the fixed population parameter lies in this particular interval. In repeated samples using the same valid procedure, approximately 95% of such intervals would contain the parameter. Practical interpretation should also consider effect size and business relevance.

Worked example: mean spending

Suppose 64 sampled customers report mean monthly spending of ₹1,200 with standard deviation ₹320. Using t* ≈ 2.00 for 63 degrees of freedom:

SE = 320/√64 = 40

The margin of error is 2.00 × 40 = ₹80. The 95% interval is therefore ₹1,120 to ₹1,280. This estimates the population mean only if the sampling process and measurement assumptions are defensible. A large sample cannot eliminate non-response or coverage bias.

5. Hypothesis Testing

A hypothesis test evaluates whether data are inconsistent with a specified null model. The null hypothesis, H₀, commonly states no difference or no association; the alternative hypothesis, H₁, states the pattern of interest. A p-value is the probability, assuming H₀ and the analysis assumptions are true, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true, nor a measure of practical importance.

For a one-sample mean compared with a benchmark μ₀:

The statistic is compared with a t distribution with n − 1 degrees of freedom under the usual assumptions. For two independent means, methods such as Welch’s t-test are generally preferred when variances may differ. Paired tests are appropriate when each observation in one condition is naturally matched to an observation in another, such as before-and-after measurements on the same customer.

A two-sided test examines deviations in either direction; a one-sided test requires a directional hypothesis specified before seeing the data. The significance level α is a decision threshold, often 0.05, not a universal standard of truth. Multiple tests increase the chance of false positives. Pre-specifying primary outcomes, controlling the false-discovery rate or adjusting significance levels can reduce this problem.

Statistical power is the probability of detecting an effect of a specified size when it exists. It increases with sample size, effect magnitude and measurement precision, and decreases with stricter significance thresholds. A non-significant result may reflect no meaningful effect or insufficient information. Report effect estimates and confidence intervals, not p-values alone.

6. Correlation and Association

The Pearson correlation coefficient measures linear association between two quantitative variables:

Here, xᵢ and yᵢ are paired observations, and x̄ and ȳ are their sample means. The value lies between −1 and +1. Positive values indicate that larger values of one variable tend to accompany larger values of the other; negative values indicate the opposite. Pearson correlation describes linear association and can be distorted by outliers, non-linearity, restriction of range or dependence among observations. Spearman’s rank correlation may be useful for monotonic relationships or ordinal data, but it also does not establish causation.

Correlation is not causation because of confounding, reverse causality, selection effects and common trends. For example, advertising expenditure and sales may be positively correlated because firms increase advertising during high-demand periods. The observed relationship does not by itself show that advertising caused all the additional sales.

7. Regression for Market Decisions

Simple linear regression models the conditional mean of an outcome Y using a predictor X:

Yᵢ is the outcome for observation i, Xᵢ is the predictor, β₀ is the intercept, β₁ is the expected change in Y for a one-unit increase in X, and εᵢ is the unexplained error. The fitted equation replaces unknown parameters with estimates:

The slope estimate is:

The intercept is b₀ = ȳ − b₁x̄. The slope has units of outcome per unit of predictor. It should not be interpreted outside the observed range without strong justification.

The coefficient of determination is:

where SSE is the sum of squared residuals and SST is the total sum of squared deviations from the mean. R² is the proportion of observed variation explained by the fitted linear relationship in the sample; it is not proof of causal influence or managerial usefulness.

Important diagnostic considerations include linearity, influential observations, unequal residual variance, dependence, omitted variables and measurement error. Multiple regression can adjust for measured covariates, but statistical adjustment does not automatically remove confounding. Causal claims require stronger designs, such as randomised experiments, credible natural experiments or carefully justified longitudinal methods.

8. Interpretation and Communication

A sound interpretation proceeds in four steps:

1. State the estimate or observed pattern.
2. Quantify uncertainty and practical magnitude.
3. Relate it to the research question and decision.
4. State assumptions, limitations and alternative explanations.

Weak statement: “The campaign increased sales because sales were higher after launch.”

Stronger statement: “Sales were higher after launch, but the before-after comparison cannot isolate the campaign effect because seasonality, competitor actions and broader demand may also have changed. A matched control market or randomised rollout would provide stronger evidence.”

Avoid selective reporting, p-value hunting, post-hoc subgroup claims and causal verbs unsupported by design. Clearly distinguish exploratory findings from confirmatory tests. When presenting segmentation results, show segment sizes and stability; a tiny high-spending segment may be commercially attractive but statistically unstable.

Applied Case: Illustrative FreshBite Subscription Study

Background

FreshBite, an illustrative meal-subscription venture, surveyed 420 adults in three cities before launching a student-oriented plan. Management wanted to know whether price concern differed between students and non-students and whether perceived convenience predicted purchase intention.

Preparation

The team removed 12 duplicate submissions after comparing respondent identifiers and timestamps. Fourteen ages outside the eligible range were checked against source records; three were corrected and eleven were retained as missing because verification was impossible. “Prefer not to say” was retained as a valid response for demographic analysis and excluded only from analyses requiring a defined age. A data dictionary documented all decisions.

Purchase intention was measured using three five-point items. One item was reverse-coded, and the items were combined only after checking that they represented the same construct. The team reported missingness and conducted a sensitivity analysis using complete cases and an imputed dataset.

Analysis

Among 394 valid cases, 61% were students. The mean purchase-intention score was 3.42 on a five-point scale, with a 95% confidence interval of 3.30 to 3.54. Students had a mean price-concern score of 4.1, compared with 3.6 for non-students. The difference was statistically detectable, but management also examined its magnitude and distribution rather than treating significance as proof of commercial importance.

A scatterplot showed a positive association between perceived convenience and purchase intention. A simple regression estimated that a one-point increase in convenience corresponded to a 0.48-point increase in predicted intention. The relationship was observational: people who value convenience may also have busier schedules, higher incomes or stronger interest in meal services. The report therefore used “associated with,” not “causes.”

Approach and recommendation

FreshBite piloted two prices through a randomised online experiment, assigning eligible participants to different offers while holding product descriptions constant. The experiment provided stronger evidence about price response than the original survey. The company also planned a city-level rollout with a comparison city and pre-specified outcome measures.

Learning points

The case demonstrates that cleaning decisions influence results, descriptive evidence should precede modelling, confidence intervals add useful context, and causal claims require design rather than statistical sophistication alone. It also shows why entrepreneurial decisions often benefit from sequential research: exploratory survey, controlled test and post-launch monitoring.

Ethical, Governance and Technological Considerations

Data preparation is an ethical activity. Researchers should obtain appropriate consent, minimise collection, protect identifiers, restrict access and retain data only as long as justified. Sensitive attributes should not be used for discriminatory targeting. Analysts should disclose sponsorship, exclusions, imputation and conflicts of interest.

Automated tools and AI can accelerate coding, anomaly detection, transcription and dashboard creation, but they can also reproduce biased classifications, invent plausible values or expose confidential data through insecure services. Human review, reproducible scripts, version control and access controls remain essential. Synthetic data may support development, but it cannot automatically substitute for representative market evidence. Governance should include documentation of data provenance, model purpose, validation, monitoring and accountability.

Best Practices Checklist

• Begin with a decision question, not a preferred statistical technique.
• Preserve raw data and maintain a transformation log.
• Define missing values and valid ranges before analysis.
• Report denominators, sample sizes and units.
• Inspect distributions before selecting summary measures.
• Use confidence intervals and effect sizes alongside p-values.
• Check assumptions and conduct sensitivity analyses.
• Treat correlations and regression coefficients as associations unless design supports causal inference.
• Avoid unnecessary subgrouping and multiple unplanned tests.
• Communicate limitations in language decision-makers can understand.

Chapter Summary

Reliable market analysis begins with a clearly defined unit of analysis, a documented data dictionary and systematic quality checks. Missing, duplicated, invalid and extreme observations require investigation rather than automatic deletion. Coding must reflect the measurement level and research purpose. Descriptive statistics and suitable visualisations reveal distributions, differences and potential relationships, while confidence intervals communicate sampling uncertainty.

Hypothesis tests assess compatibility with a null model; they do not establish importance or truth. Correlation measures association, and regression estimates conditional relationships under stated assumptions. Neither automatically establishes causality. Interpretation should combine effect magnitude, uncertainty, design, context and limitations. Ethical governance, reproducibility and transparent reporting are essential, particularly when automated or AI-enabled tools are used. For entrepreneurial decision-making, the strongest workflow connects cleaned evidence to controlled experimentation and ongoing market validation.

Key Terms

Data dictionary: A record of variable meanings, types, codes, units and sources.

Unit of analysis: The entity represented by one analytical observation.

Missing data: Values unavailable, inapplicable or not recorded.

Outlier: An observation unusually distant from the rest of a dataset.

Nominal variable: A categorical variable without intrinsic order.

Ordinal variable: A categorical variable with order but not necessarily equal intervals.

Composite scale: A measure formed by combining multiple items.

Mean: The arithmetic average of numerical observations.

Median: The middle ordered observation.

Standard deviation: A measure of dispersion around the mean.

Confidence interval: An interval produced by a procedure designed to capture a population parameter at a stated long-run rate.

Null hypothesis: A formal statement representing no specified effect or association.

p-value: The probability, under the null model, of results at least as extreme as those observed.

Statistical power: The probability of detecting a specified effect when it exists.

Correlation: A numerical measure of association, commonly linear association.

Regression: A model relating an outcome to one or more predictors.

Confounder: A variable related to both a predictor and an outcome that can distort their observed relationship.

Effect size: A quantitative measure of the magnitude of a difference or relationship.

Reproducibility: The ability to recreate analytical results using documented data and procedures.

Review Questions

Short Answer Questions

1. Why should a missing income response not automatically be coded as zero?
2. Distinguish nominal, ordinal and ratio variables.
3. What information does a confidence interval add to a point estimate?
4. Why is a p-value not the probability that the null hypothesis is true?
5. State two reasons why correlation does not establish causation.

Long Answer Questions

1. Explain a complete workflow for preparing survey data for market analysis, including documentation and quality assurance.
2. Compare mean, median, standard deviation and interquartile range, and explain when each is useful.
3. Explain the assumptions and interpretation of a confidence interval for a population mean.
4. Discuss how regression can support entrepreneurial decisions while still producing misleading causal interpretations.
5. Evaluate the ethical and governance requirements for using AI in market-data preparation and analysis.

Analytical / Application Questions

1. A start-up finds that customers exposed to more advertisements spend more, but advertising was targeted toward previously high-value customers. Identify the likely bias and propose a stronger research design.
2. A survey reports a statistically significant difference in satisfaction of 0.08 points on a five-point scale with a narrow confidence interval. How should management evaluate statistical versus practical significance?
3. A dataset contains 20% missing income values, concentrated among respondents with low trust in the company. Recommend an analysis and sensitivity strategy, explaining why complete-case analysis alone may be problematic.

Multiple Choice Questions with Answers

Question 1

Which practice is most appropriate when a respondent does not report income?

A. Code the value as zero

B. Replace it with the sample maximum

C. Record it using a documented missing-value code

D. Delete the entire dataset

Correct Answer: C. Record it using a documented missing-value code

Question 2

Which statistic is generally least sensitive to extreme values?

A. Mean

B. Median

C. Variance

D. Standard deviation

Correct Answer: B. Median

Question 3

A 95% confidence interval is best interpreted as:

A. A 95% probability that the fixed parameter is in this calculated interval

B. A range produced by a method that captures the parameter in about 95% of repeated samples

C. The range containing 95% of individual observations

D. Proof that the estimate is practically important

Correct Answer: B. A range produced by a method that captures the parameter in about 95% of repeated samples

Question 4

A p-value is:

A. The probability that the alternative hypothesis is true

B. The probability that the sample was collected without bias

C. The probability, under the null model, of results at least as extreme as observed

D. The size of the business effect

Correct Answer: C. The probability, under the null model, of results at least as extreme as observed

Question 5

Which visualisation is most suitable for examining the relationship between two numeric variables?

A. Scatterplot

B. Pie chart

C. Single-category bar chart

D. Frequency table only

Correct Answer: A. Scatterplot

Question 6

If two variables have a strong positive correlation, which conclusion is justified?

A. One variable definitely causes the other

B. They display a strong positive association in the observed data

C. There is no possibility of confounding

D. A randomised experiment has occurred

Correct Answer: B. They display a strong positive association in the observed data

Question 7

In a simple regression equation, the slope represents:

A. The predicted outcome when the predictor is zero in every context

B. The expected change in outcome for a one-unit increase in the predictor

C. The proportion of missing observations

D. The sample size

Correct Answer: B. The expected change in outcome for a one-unit increase in the predictor

Question 8

If n = 64 and the sample standard deviation is 320, what is the standard error of the sample mean?

A. 4

B. 20

C. 40

D. 384

Correct Answer: C. 40

Question 9

Which action best reduces the risk of misleading subgroup conclusions?

A. Create as many subgroups as possible after viewing results

B. Report subgroup sizes and pre-specify important comparisons

C. Remove all small subgroups without documentation

D. Report only the subgroup with the largest effect

Correct Answer: B. Report subgroup sizes and pre-specify important comparisons

Question 10

Why should raw data be preserved separately from cleaned data?

A. To make missing data disappear

B. To permit auditing and reproduction of cleaning decisions

C. To guarantee causal inference

D. To increase the sample size

Correct Answer: B. To permit auditing and reproduction of cleaning decisions