Data Preparation and Statistical Analysis

Introduction

Business research produces value only when raw observations are converted into trustworthy evidence. Survey responses, transaction records, interviews coded into categories, website analytics and experiment results are rarely ready for analysis when first collected. They may contain duplicate cases, inconsistent labels, missing values, extreme observations, invalid responses and measurement errors. Data preparation is therefore not a clerical preliminary; it is part of research design and affects the credibility of every conclusion.

Statistical analysis provides a structured way to describe data, estimate population characteristics, assess evidence for propositions and examine relationships among variables. For entrepreneurs, these activities support decisions such as identifying attractive customer segments, estimating demand, evaluating a marketing campaign, forecasting sales and testing whether a new product improves customer outcomes.

This chapter presents a practical workflow using spreadsheets or statistical software. It emphasizes not only calculation, but also interpretation: statistical significance does not automatically imply managerial importance, and a visually impressive relationship may be misleading if the data or research design is weak.

Learning Objectives

By the end of this chapter, you should be able to:

1. Prepare, document and code a business dataset.
2. Identify and manage missing values, duplicates and outliers.
3. Select appropriate tables, charts and descriptive statistics.
4. Construct and interpret confidence intervals and hypothesis tests.
5. Apply correlation, simple regression and cross-tabulation appropriately.
6. Distinguish statistical significance, effect size and practical significance.
7. Communicate analytical results ethically using spreadsheets or statistical software.

1. The Data-Analysis Workflow

A defensible workflow normally follows these stages:

1. Define the unit of analysis and variables.
2. Create a data dictionary and coding scheme.
3. Import and preserve the raw data.
4. Check structure, validity, duplicates and range errors.
5. Diagnose missing values and outliers.
6. Transform or recode variables when justified.
7. Explore distributions using tables and visualizations.
8. Conduct descriptive and inferential analyses.
9. Test assumptions and sensitivity of conclusions.
10. Report methods, results, limitations and decisions.

The unit of analysis is the entity represented by one row: for example, one customer, order, store or month. Confusing units can produce invalid conclusions. A dataset containing several purchases per customer cannot be analysed as if each row represented an independent customer unless the research design accounts for repeated observations.

A data dictionary should record the variable name, meaning, measurement level, coding, permitted values, missing-value code and source. A useful distinction is between:

• Nominal variables, which identify categories without order, such as industry.
• Ordinal variables, whose categories have order but unequal or unknown spacing, such as satisfaction from 1 to 5.
• Interval variables, where differences are meaningful but zero is not absolute, such as temperature in Celsius.
• Ratio variables, with meaningful differences and a true zero, such as revenue, units sold and age.

Measurement level influences the appropriate analysis. For example, a mean is usually informative for revenue, but a median or frequency distribution may better represent a highly skewed ordinal measure.

2. Cleaning and Coding Data

Protecting the raw data

Keep an unchanged, read-only copy of the original file. Perform cleaning in a separate working file or through a reproducible script. Record every change in a cleaning log: variable, original value, revised value, reason, date and responsible analyst. This supports auditability and prevents selective alteration of inconvenient observations.

Common quality checks

Check that:

• each record has a valid identifier;
• identifiers are unique where uniqueness is expected;
• dates use one format and fall within the study period;
• numerical values lie within plausible bounds;
• categories use consistent spelling and capitalization;
• skip patterns in questionnaires have been respected;
• calculated fields agree with source fields; and
• impossible combinations are investigated.

For instance, “United Kingdom,” “UK” and “U.K.” should be mapped to one documented category. A customer age of 240 is not automatically an outlier to delete; it is first an error to investigate.

Coding variables

Categories can be represented by labels or numerical codes. Coding “Yes” as 1 and “No” as 0 is useful, but the numbers are labels unless the variable has a meaningful quantitative interpretation. A five-point satisfaction scale may be coded 1–5 for analysis, but analysts should acknowledge that its intervals may not be exactly equal.

For a categorical variable with k groups, regression often uses k − 1 dummy variables. If “independent business” is the reference group, coefficients for “franchise” and “corporate chain” express differences from that reference, not absolute effects.

Composite scales require attention to direction. If high values indicate satisfaction on most items but dissatisfaction on one reverse-worded item, that item must be reverse-coded before calculating a scale score. For a 1–5 item, reverse coding is:

Here x is the original response and x′ is the reversed response. This transformation assumes the response scale runs from 1 through 5.

3. Missing Values and Outliers

Missing data

Missingness may be caused by refusal, questionnaire routing, technical failure or inapplicability. Do not automatically code all missing values as zero. A zero may mean “none,” whereas a blank may mean “unknown.” Use explicit codes such as NA or a system-missing value, and document them.

Three useful concepts are:

• MCAR: missing completely at random; missingness is unrelated to observed and unobserved values.
• MAR: missing at random conditional on observed variables.
• MNAR: missingness depends on the unobserved value or another unmeasured process.

The mechanism cannot usually be proven from the dataset alone. Compare missingness rates across important groups and variables. If only a few cases are missing and missingness is plausibly harmless, complete-case analysis may be reasonable. If missingness is substantial, consider multiple imputation or model-based methods rather than replacing every blank with the mean.

Mean imputation reduces variation and can distort correlations. Median imputation may be more robust for skewed variables but still understates uncertainty. Multiple imputation creates several plausible completed datasets, analyses each one and combines estimates; it is preferable when assumptions and sample size support it, but it is not a remedy for poorly measured variables.

Outliers

An outlier is an observation unusually distant from the rest. It may reflect an error, a legitimate unusual case or a different population. Investigate the source, units, entry process and business context before removing it.

For a variable, a common rule identifies potential outliers below Q₁ − 1.5 × IQR or above Q₃ + 1.5 × IQR, where IQR = Q₃ − Q₁. This is a screening rule, not proof of error. A z-score is:

where x_i is an observation, x̄ is the sample mean and s is the sample standard deviation. Values beyond approximately ±3 may warrant investigation, although the threshold depends on distribution and purpose.

Deleting observations solely because they weaken a desired conclusion is unethical. Report exclusions, compare results with and without influential cases, and consider robust statistics or transformations where justified.

4. Descriptive Statistics and Visualization

Descriptive analysis summarizes the observed sample; it does not by itself establish population claims. For n observations x₁, x₂, …, xₙ, the sample mean is:

The sample variance and standard deviation are:

The denominator n − 1 is used when estimating population variability from a sample. The median is the middle ordered value, and the mode is the most frequent value. The range is maximum minus minimum. Quartiles divide ordered data into four parts.

Use the mean and standard deviation for reasonably symmetric quantitative data. Use the median and interquartile range when data are skewed or contain influential extremes. Frequencies and percentages are generally appropriate for categorical variables.

A well-designed visualization matches the question:

Charts need clear units, denominators and labels. A truncated vertical axis can exaggerate differences in bar charts. Overlapping points, unequal group sizes and selective time windows should be disclosed.

5. Sampling, Estimation and Confidence Intervals

A sample statistic estimates a population parameter. A confidence interval communicates sampling uncertainty. For a sample mean, when the population standard deviation is unknown, the conventional interval is:

Here t* is the critical value from the t distribution for the chosen confidence level and degrees of freedom n − 1. The formula assumes independent observations and is most reliable when the sampling distribution of the mean is approximately normal; larger samples make this less restrictive.

For a proportion p̂, a large-sample interval is often written:

where z* is the standard-normal critical value. This approximation can be poor with small samples or proportions near 0 or 1; exact or improved interval methods may then be preferable.

A 95% confidence procedure means that, over repeated samples using the same method, approximately 95% of intervals would contain the fixed population parameter. It does not mean there is a 95% probability that this particular fixed parameter lies inside the calculated interval.

Worked example. Suppose 100 surveyed customers report an average monthly spend of $48, with s = $20. Using t* ≈ 1.984 for 99 degrees of freedom:

SE = 20/√100 = 2

The interval is 48 ± 1.984 × 2, or approximately $44.03 to $51.97. The estimate should be reported with the sampling design and response rate, because a narrow interval cannot correct for biased sampling.

6. Hypothesis Testing

A hypothesis test evaluates how compatible observed data are with a null model. The null hypothesis H₀ commonly represents no difference or no association; the alternative hypothesis H₁ represents the research claim. The p-value is the probability, assuming H₀ and the test assumptions are correct, of obtaining a result at least as extreme as the observed result. It is not the probability that H₀ is true.

A typical workflow is:

1. State H₀ and H₁ before examining the result.
2. Select the test and significance level α, often 0.05.
3. Check design and assumptions.
4. Calculate the test statistic and p-value.
5. Report an effect size and confidence interval.
6. Interpret statistical and practical importance.

For a one-sample mean test:

where μ₀ is the hypothesized population mean. The test assumes independent observations and an approximately normal sampling distribution.

A p-value below α is commonly described as statistically significant, leading to rejection of H₀ under the stated model. A p-value above α does not prove no effect; it indicates insufficient evidence under that test. Statistical decisions should never be based on p-values alone.

7. Comparing Groups and Measuring Effect Size

For two independent means, a difference can be reported as x̄₁ − x̄₂, with a confidence interval. Welch’s t-test is often preferable to the equal-variance version when group variances or sample sizes differ. For paired observations, such as each customer’s score before and after a campaign, analyse within-person differences rather than treating observations as independent.

For two proportions, compare p̂₁ − p̂₂ and provide a confidence interval. A chi-square test assesses whether two categorical variables are independent. Its statistic is:

Oᵢⱼ is an observed cell count and Eᵢⱼ is the expected count under independence. Expected counts are calculated from row and column totals. Very small expected counts can make the approximation unreliable; categories may need principled combination or an exact test.

Effect size describes magnitude. Cohen’s d for two means is approximately:

where s_p is the pooled standard deviation under the equal-variance assumption. For unequal variances, use a suitable standardized effect-size method. For chi-square tables, Cramér’s V is:

where r and c are the numbers of rows and columns. Interpretation depends on context, measurement and industry benchmarks; universal cutoffs can mislead.

8. Correlation and Regression

Correlation measures linear association, not causation. Pearson’s sample correlation is:

Its value ranges from −1 to +1. A scatter plot should precede interpretation because a nonlinear relationship, clusters or one influential point can make r misleading. Spearman’s rank correlation may be useful for ordinal data or monotonic but non-normal relationships, though it also does not establish causality.

Simple linear regression models an outcome Y as:

b₀ is the predicted outcome when X = 0, and b₁ is the expected change in Y for a one-unit increase in X. The model assumes an appropriate linear form, independent errors, constant error variance and, for conventional confidence tests, approximately normal residuals. Predictions outside the observed range are extrapolations and should be treated cautiously.

In multiple regression, additional predictors help adjust for measured differences:

The coefficient bⱼ is interpreted conditionally: the expected change in Y for a one-unit increase in Xⱼ while the other included predictors remain constant. This is not automatically a causal effect. Omitted variables, reverse causality, selection bias and measurement error may remain.

R² is the proportion of sample variation in Y explained by the fitted model, but a high R² does not prove the model is useful or causal. Examine residual plots, influential observations, multicollinearity and out-of-sample performance where prediction is the objective.

9. Cross-Tabulation and Segmentation

A cross-tabulation compares counts or percentages across two categorical variables. Percentages must be labelled as row, column or total percentages. For example, “30% of customers are repeat buyers” differs from “30% of repeat buyers use mobile ordering.”

A practical analysis might cross-tabulate customer segment by subscription adoption, then use a chi-square test to assess association. Analysts should inspect cell sizes and report absolute counts alongside percentages. Association may reflect confounding variables such as age, income or location, so cross-tabulation is descriptive unless the design supports stronger inference.

10. Statistical Versus Practical Significance

Statistical significance is influenced by effect magnitude, variability and sample size. With a very large sample, a trivial difference may have p < 0.05. With a small sample, a commercially important effect may fail to reach that threshold.

Practical significance asks whether the size of an effect matters for a decision. A two-percentage-point conversion improvement may justify investment for a high-volume platform but not for a small campaign with substantial implementation costs. Report the estimated effect, confidence interval, cost or benefit threshold and relevant benchmark. Pre-specifying a minimum practically important difference reduces post hoc rationalization.

11. Software Implementation and Reproducibility

Spreadsheets are useful for small, transparent datasets. Use separate sheets for raw data, cleaned data, codebook, calculations and outputs. Avoid hard-coded results, label formulas and use data validation. Statistical packages are preferable for larger datasets, complex models, repeated analyses and reproducible scripts.

AI-assisted tools can help generate code, identify inconsistent labels or suggest visualizations, but analysts must verify formulas, protect confidential data and retain human responsibility for decisions. Never upload personally identifiable or commercially sensitive data to an unapproved service. Reproducibility requires preserving the dataset version, code, software version, assumptions and output.

12. Illustrative Case: Evaluating a Digital Loyalty Campaign

Background. A fictional online food retailer introduced a loyalty message to a randomly selected group of 400 customers. Another 400 customers formed a comparison group. The outcome was whether a customer placed an order within 30 days.

Problem. Management wants to know whether the message improved ordering and whether the improvement justifies the campaign cost.

Analysis. The analyst first checked duplicate customer IDs, treatment assignment, eligibility dates and missing outcome records. The treatment group had 92 purchasers (23.0%); the comparison group had 72 (18.0%). The estimated difference was 5 percentage points. A two-proportion analysis should report a confidence interval and p-value, but interpretation must also consider randomization quality, non-delivery of messages and the campaign’s cost.

The analyst cross-tabulated treatment by purchase outcome and used a bar chart with identical axes. A logistic regression could additionally examine whether the estimated treatment effect differs by prior purchase frequency, but subgroup analyses should be planned or clearly labelled exploratory.

Approach and decision. Suppose the confidence interval for the difference is approximately −1.0 to 11.0 percentage points. The result is compatible with no improvement as well as a meaningful improvement. Management should not describe the campaign as definitively successful. It may conduct a larger, pre-registered experiment, estimate profit per incremental order and assess whether the upper and lower bounds support investment.

Learning points. Random assignment strengthens causal interpretation, but missing outcomes and non-compliance can weaken it. A statistically uncertain result can still inform a staged business decision. Reporting the effect and interval is more useful than reporting only “significant” or “not significant.”

13. Limitations, Ethics and Best Practices

Statistical analysis cannot repair biased sampling, vague constructs or poor measurement. Common threats include nonresponse bias, convenience sampling, common-method bias, multiple testing, p-hacking, selective reporting and data leakage. Analysts should distinguish exploratory findings from confirmatory tests, disclose all important exclusions and avoid claiming causation from observational association.

Privacy principles include data minimization, purpose limitation, access control, secure storage, retention limits and lawful processing. Remove direct identifiers when they are not needed, suppress small cells where re-identification is possible and obtain appropriate consent or institutional approval. In regulated contexts, follow applicable data-protection and sector requirements.

Best practice is to:

• formulate the decision and estimand before analysis;
• preserve raw data and maintain a cleaning log;
• inspect distributions before selecting tests;
• report denominators, missingness and assumptions;
• use confidence intervals and effect sizes;
• conduct sensitivity checks for outliers and alternative specifications;
• distinguish association from causation;
• make charts accessible and accurately scaled; and
• communicate uncertainty in language decision-makers can use.

Chapter Summary

Data preparation converts raw observations into analysable evidence through documented coding, validity checks, missing-data decisions and outlier investigation. Descriptive statistics and suitable charts summarize distributions and group differences. Confidence intervals express estimation uncertainty, while hypothesis tests evaluate evidence under explicit assumptions. Correlation describes linear association; regression models conditional relationships but does not automatically establish causation. Cross-tabulation and chi-square analysis support categorical comparisons. Sound interpretation combines p-values with effect sizes, confidence intervals and practical thresholds. Ethical analysis requires privacy protection, transparency, reproducibility and honest communication of limitations.

Key Terms

Data dictionary: A record of variable meanings, codes, units and permitted values.

Unit of analysis: The entity represented by one observation or row.

Nominal variable: A categorical variable without an inherent order.

Ordinal variable: A categorical variable with ordered levels.

Missing completely at random (MCAR): Missingness unrelated to observed or unobserved values.

Missing at random (MAR): Missingness explainable by observed information.

Outlier: An unusually distant observation requiring investigation.

Mean: The arithmetic average of observations.

Median: The middle ordered observation.

Standard deviation: A measure of dispersion around the mean.

Confidence interval: An interval produced by a method designed to capture a population parameter at a stated long-run rate.

Null hypothesis: A formal statement commonly representing no difference or association.

P-value: Probability of data at least as extreme as observed under the null model.

Effect size: A measure of the magnitude of a difference or association.

Correlation: A measure of linear association between quantitative variables.

Regression: A model relating an outcome to one or more predictors.

Cross-tabulation: A table showing joint frequencies of categorical variables.

Statistical significance: Evidence judged inconsistent with a null model at a chosen threshold.

Practical significance: Importance of an effect for a real decision or outcome.

Reproducibility: Ability to reproduce an analysis from preserved data, methods and code.

Review Questions

5 Short Answer Questions

1. Why should the raw dataset be preserved before cleaning?
2. Distinguish MCAR, MAR and MNAR missingness.
3. What information does a confidence interval provide?
4. Why does correlation not establish causation?
5. What is the purpose of a data dictionary?

5 Long Answer Questions

1. Explain a complete workflow for preparing and analysing a business dataset.
2. Compare mean imputation, complete-case analysis and multiple imputation, including their limitations.
3. Explain how descriptive statistics and visualizations should be selected for different measurement levels.
4. Discuss the assumptions and interpretation of a linear regression model.
5. Explain why effect size and practical significance should accompany p-values.

3 Analytical / Application Questions

1. A retailer finds that a campaign group has a higher average spend than a comparison group. Design an analysis that addresses data cleaning, confidence intervals, hypothesis testing, effect size and possible confounding.
2. A survey contains satisfaction scores from 1 to 5, with 18% missing responses and several extreme revenue values. Recommend a defensible preparation and analysis plan.
3. A cross-tabulation shows higher subscription adoption among urban customers. Explain how you would assess whether the association is reliable, practically important and potentially confounded.

Multiple Choice Questions with Answers

Question 1

Which practice most directly supports auditability during data preparation?

A. Deleting all unusual observations

B. Maintaining a cleaning log documenting changes and reasons

C. Replacing every missing value with zero

D. Rounding all numerical variables

Correct Answer: B. Maintaining a cleaning log documenting changes and reasons

Question 2

For a sample mean with unknown population standard deviation, the usual confidence interval uses which critical distribution?

A. t distribution

B. Chi-square distribution only

C. Uniform distribution

D. Binomial distribution

Correct Answer: A. t distribution

Question 3

A p-value is best described as the probability of observing a result at least as extreme as the sample result, assuming:

A. The alternative hypothesis is true

B. The null model and its assumptions are true

C. The sample mean equals the population mean

D. The effect is practically important

Correct Answer: B. The null model and its assumptions are true

Question 4

Which statistic measures linear association between two quantitative variables?

A. Cramér’s V only

B. Pearson’s correlation coefficient

C. The median

D. A frequency count

Correct Answer: B. Pearson’s correlation coefficient

Question 5

In the regression equation Ŷ = b₀ + b₁X, what does b₁ represent?

A. The predicted value when X is zero

B. The sample size

C. The expected change in Y for a one-unit increase in X

D. The residual standard deviation

Correct Answer: C. The expected change in Y for a one-unit increase in X

Question 6

Which chart is generally most appropriate for examining the relationship between two quantitative variables?

A. Scatter plot

B. Pie chart

C. Bar chart of one category

D. Frequency table only

Correct Answer: A. Scatter plot

Question 7

An observation above Q₃ + 1.5 × IQR should be treated as:

A. Automatically deleted

B. Proof of fraud

C. A potential outlier requiring investigation

D. A missing value

Correct Answer: C. A potential outlier requiring investigation

Question 8

What does a chi-square test of independence evaluate?

A. Whether two categorical variables are associated

B. Whether a mean equals zero in every population

C. Whether a regression slope is measured in dollars

D. Whether all observations are normally distributed

Correct Answer: A. Whether two categorical variables are associated

Question 9

Why can a statistically significant result have limited managerial value?

A. Significance always means the data are invalid

B. A very small effect can become significant in a large sample

C. Confidence intervals cannot be calculated

D. Statistical tests never use sample size

Correct Answer: B. A very small effect can become significant in a large sample

Question 10

Which action is most ethically appropriate when an outlier weakens a preferred conclusion?

A. Remove it without recording the change

B. Keep or exclude it based on documented substantive criteria and report sensitivity

C. Change its value to the sample mean

D. Publish only the analysis without the outlier

Correct Answer: B. Keep or exclude it based on documented substantive criteria and report sensitivity