Research Design and Information Sources

Introduction and context

Marketing research is the systematic process of defining a marketing problem, collecting and analysing evidence, and communicating findings to support decisions. For product managers, research is not an academic exercise separate from practice. It informs decisions about customer problems, product–market fit, positioning, pricing, promotion, adoption, retention and product improvement.

A research project begins with a decision, not with a questionnaire. The central question is: What decision must the evidence support? A product team deciding whether to launch a new feature may need exploratory interviews, behavioural data and a survey of prospective users. A team diagnosing falling retention may need internal usage data, customer interviews and controlled experiments. Selecting an appropriate design therefore requires alignment among the decision problem, research questions, available evidence, time, budget and ethical obligations.

This chapter explains qualitative, quantitative and mixed-method designs and evaluates primary, secondary, syndicated and internal data. It also introduces criteria for judging research quality: validity, reliability, bias, cost and feasibility.

Learning objectives

By the end of this chapter, you should be able to:

1. Distinguish management decision problems from research problems.
2. Select qualitative, quantitative or mixed-method designs for different product and advertising decisions.
3. Evaluate primary, secondary, syndicated and internal information sources.
4. Assess validity, reliability, bias, cost and feasibility.
5. Design a practical research plan that is ethically and methodologically defensible.
6. Interpret research evidence without overstating what the data can establish.

1. From business problem to research design

A management decision problem concerns an action, such as whether to launch, reposition, advertise or discontinue a product. A research problem specifies what must be learned to make that decision. For example:

• Decision problem: Should a meal-delivery service introduce a premium subscription?
• Research problem: Which customer segments value the proposed benefits, what price and usage conditions affect adoption, and what objections may prevent trial?

A useful research brief contains:

• the decision to be supported;
• the target population or market;
• research questions and, where appropriate, hypotheses;
• information required;
• proposed methods and data sources;
• timing, budget and responsibilities;
• ethical, privacy and governance requirements; and
• the intended decision rule or use of findings.

A research design is the overall plan for obtaining and analysing evidence. It includes the research approach, sampling strategy, measurement instruments, fieldwork procedures, analysis plan and reporting format. A design is not automatically good because it is sophisticated. It is good when it produces credible, relevant evidence at an acceptable cost and within the decision timeline.

Exploratory, descriptive and causal purposes

Research purpose strongly influences design.

Exploratory research investigates poorly understood problems. It seeks concepts, language, motivations and possible explanations rather than precise population estimates. Interviews, observation, social listening and literature reviews are common.

Descriptive research measures characteristics, behaviours or attitudes. A survey estimating the percentage of users aware of an advertisement is descriptive. It requires clear operational definitions and an appropriate sample.

Causal research examines whether a change in one factor produces a change in another. A controlled advertising experiment may compare conversion rates between two messages. Causal claims require temporal order, an interpretable intervention and credible control of alternative explanations.

2. Qualitative research design

Qualitative research produces non-numerical evidence such as words, images, observations and narratives. It is especially useful when the researcher needs to understand meaning, context, emotion, language or an emerging problem.

Common qualitative methods

In-depth interviews are one-to-one conversations guided by a flexible protocol. They are suitable for sensitive topics, complex journeys and expert participants. The interviewer can probe unexpected answers, but interviewer influence and small, non-random samples limit statistical generalisation.

Focus groups bring several participants together to discuss a topic under a moderator’s guidance. Interaction can reveal shared language, disagreement and social norms. However, dominant participants, conformity and group pressure can suppress minority views. Focus groups are usually inappropriate for highly sensitive subjects or when participants have strong status differences.

Observation and ethnography examine behaviour in natural or realistic settings. They can reveal workarounds and unspoken needs that respondents cannot easily articulate. They require careful consent, recording procedures and interpretation of context.

Diary studies ask participants to record experiences over time. They are useful for recurring behaviours, onboarding journeys and service problems. Participant fatigue and incomplete entries are common risks.

Usability testing observes users attempting specified tasks with a product or prototype. It identifies confusion, friction and unmet needs. A small usability study can be highly valuable for design improvement, but it does not estimate population-level satisfaction or market share.

Qualitative sampling and analysis

Qualitative sampling is often purposive: participants are selected because they possess relevant experience. Researchers may use maximum-variation sampling to include contrasting users, criterion sampling to require a specified experience, or theoretical sampling to pursue concepts emerging during analysis. Sample adequacy depends on information richness and the research purpose, not on a universal numerical rule.

Analysis commonly involves transcription, familiarisation, coding, category development, comparison and interpretation. A code is a label attached to a meaningful segment of data. Themes are broader patterns that explain recurring meanings or relationships. Quality improves when researchers document coding decisions, compare interpretations, search for disconfirming evidence and retain an audit trail.

Qualitative evidence can establish that a perception or experience exists and can explain how participants understand it. It should not automatically be reported as the percentage of all customers who hold that view.

3. Quantitative research design

Quantitative research represents observations numerically and analyses patterns across cases. It is appropriate when the researcher needs estimates, comparisons, relationships or tests of specified hypotheses.

Surveys

A survey uses standardised questions so that responses can be compared. Good survey design requires:

• a clearly defined population;
• appropriate sampling or transparent limitations;
• neutral, unambiguous wording;
• response options that are exhaustive and mutually exclusive where appropriate;
• sensible question order;
• measurement scales suited to the construct; and
• pilot testing.

A double-barrelled question, such as “How satisfied are you with the product’s price and quality?”, asks about two constructs at once. Leading wording, undefined time periods and forced choices can create measurement error.

Attitude scales may use agreement, satisfaction, importance or likelihood formats. A scale is not automatically interval-level merely because it has numbers. Treating ordinal responses as numerical averages requires a defensible analytical rationale.

Experiments and causal designs

In a randomised experiment, units are assigned to conditions so that treatment groups are comparable in expectation. For example, users may be randomly shown Advertisement A or Advertisement B, and conversion can be compared. Randomisation supports causal inference when implementation is consistent, outcomes are measured appropriately and attrition or interference is controlled.

Quasi-experiments are used when random assignment is unavailable. Examples include before-and-after comparisons with a control market, matched comparison groups and interrupted time-series designs. They can be useful, but causal interpretation depends on assumptions such as the comparability of groups and the absence of unmeasured concurrent changes.

Basic quantitative summaries

A sample mean is calculated as:

Here, xᵢ is observation i, n is the number of observations, and x̄ is the arithmetic average. The formula assumes numerical observations for which an arithmetic mean is meaningful; it is sensitive to extreme values.

A proportion is:

where x is the number of cases with the characteristic and n is the total number of valid cases. The result is a fraction between 0 and 1, often multiplied by 100 to express a percentage. Researchers must state the denominator and explain missing responses.

For a two-group experiment, an estimated difference in conversion rates is:

where p̂₁ and p̂₂ are the observed conversion proportions in groups 1 and 2. This difference describes the observed association between assignment and conversion; it supports a causal claim only when the experimental design and implementation justify it.

Sampling and sample size

A population is the complete set of units relevant to the research question. A sample is the subset observed. Probability sampling gives units a known, non-zero selection probability and supports statistical generalisation under appropriate procedures. Examples include simple random, stratified and cluster sampling.

Non-probability methods include convenience, quota, purposive and snowball sampling. They may be practical and useful for exploratory work, but sampling error cannot be assessed in the same way as with a probability sample, and estimates may not represent the target population.

Sample size should reflect the required precision, variability, confidence level, design, subgroup analysis and resources. A large biased sample does not solve coverage or measurement problems. In product research, a smaller well-defined sample may be more useful than a large panel that excludes important users.

4. Mixed-method research

Mixed-method research intentionally integrates qualitative and quantitative evidence in one study or programme. It is more than conducting an interview study and a survey independently; the methods must be connected through the research questions, sampling, instruments, analysis or interpretation.

Three practical designs are common:

1. Exploratory sequential design: qualitative exploration identifies concepts and language, followed by a quantitative survey or experiment. This is useful when constructs are unclear.
2. Explanatory sequential design: quantitative results are followed by qualitative investigation of surprising or important patterns. For example, interviews may explain why a high-value segment has unusually low adoption.
3. Convergent design: qualitative and quantitative data are collected in a similar period and compared. Agreement strengthens confidence; divergence prompts investigation rather than automatic rejection of one method.

Integration may occur by connecting samples, building one instrument from another method, merging datasets in a joint display, or interpreting findings together. Mixed methods can improve completeness and explanation, but they require more time, skill and coordination. It is not a cure for poor sampling or weak measurement.

5. Information sources

Primary data

Primary data are collected specifically for the current research purpose. Examples include interviews, surveys, experiments, usability sessions and newly collected observations.

Advantages include relevance, control over measurement and access to information not otherwise available. Disadvantages include cost, fieldwork time, recruitment difficulty and the possibility that the research team introduces bias. Primary data are not inherently more accurate than secondary data; quality depends on design and execution.

Secondary data

Secondary data were collected previously for another purpose. They include government statistics, academic studies, trade publications, public company reports, industry databases and published market reports.

Secondary research can define market context, identify trends, benchmark competitors and prevent duplication. Researchers should evaluate the original purpose, definitions, population, time period, collection method, missing data, sponsorship and permissions. A market category defined by one source may not match the product team’s category, making direct comparison misleading.

Syndicated data

Syndicated data are collected by a specialist provider and sold or licensed to multiple clients. Examples include audience measurement, retail scanning, media exposure and consumer panels. They may provide regular benchmarks and broad coverage at a lower cost than commissioning an equivalent system alone.

Limitations include standardised measures that may not fit the decision, restricted access to raw data, unknown panel quality and possible conflicts between commercial usefulness and transparency. Buyers should examine methodology, weighting, recruitment, refresh rates, definitions, error information and contractual restrictions.

Internal data

Internal data are generated within the organisation. Examples include transaction records, customer relationship management data, product analytics, customer support logs, campaign metrics, returns and subscription cancellations.

Internal data are often timely and closely connected to action. However, they may be structured for operations rather than research. Definitions can change, events may be missing, duplicate accounts may exist and users who do not interact with the system may be invisible. A dashboard can therefore be precise while still measuring the wrong construct.

Comparing sources

The best research programme often triangulates sources rather than treating one source as definitive.

6. Evaluating evidence quality

Validity

Validity concerns whether a measure or design supports the intended interpretation.

• Face validity: appears, on inspection, to measure the intended concept. It is a weak but useful initial check.
• Content validity: covers the relevant dimensions of a construct.
• Construct validity: behaves as theory predicts in relation to other measures. Convergent evidence shows association with related measures; discriminant evidence shows distinction from unrelated constructs.
• Criterion-related validity: corresponds to a relevant outcome or benchmark.
• Internal validity: confidence that an observed effect was produced by the studied factor rather than an alternative explanation.
• External validity: extent to which findings transfer to other people, settings, times or products.

A highly realistic study may have strong external validity but weak control over confounding factors. A tightly controlled experiment may have strong internal validity but limited generalisability.

Reliability

Reliability concerns consistency of measurement. A reliable measure tends to produce similar results when the underlying condition has not changed. Forms include test–retest reliability, inter-rater reliability and internal consistency. Reliability is necessary but not sufficient for validity: a consistently wrong measurement can be reliable but invalid.

For qualitative research, dependability is strengthened through transparent procedures, reflexivity, careful documentation, multiple analysts where appropriate and attention to contradictory cases.

Bias and error

Bias is a systematic tendency that distorts findings. Important forms include:

• coverage bias: relevant members of the population are excluded from the sampling frame;
• selection bias: participation or inclusion is related to the outcome of interest;
• non-response bias: respondents differ meaningfully from non-respondents;
• social desirability bias: participants give socially acceptable rather than accurate answers;
• recall bias: memory is incomplete or reconstructed;
• interviewer or moderator bias: researcher behaviour influences responses;
• question-order and wording effects: earlier or leading questions alter answers;
• survivorship bias: analysis includes only cases that remain visible or successful; and
• confirmation bias: researchers favour evidence supporting prior beliefs.

Bias cannot be removed merely by adding more cases. It must be anticipated, measured where possible and disclosed.

Cost and feasibility

A practical design balances decision value against cost, time, access, skills and risk. Relevant costs include recruitment, incentives, software, data licensing, fieldwork, translation, cleaning, analysis and opportunity cost. Feasibility also includes whether participants can be reached, whether the organisation can implement the intervention, and whether privacy or legal restrictions limit collection.

A useful decision matrix can rate alternatives against relevance, credibility, speed, cost, access and ethical risk. Ratings should be justified rather than presented as false precision.

7. A professional research workflow

A disciplined workflow is:

1. Clarify the decision and stakeholders.
2. Review existing internal, secondary and syndicated information.
3. Translate the decision into research questions.
4. Choose exploratory, descriptive, causal or mixed purposes.
5. Define the population, sampling approach and recruitment criteria.
6. Operationalise concepts into interview topics, measures or behavioural events.
7. Prepare a protocol, pilot it and revise ambiguous procedures.
8. Collect, document, secure and quality-check the data.
9. Analyse according to the design, including contradictory evidence.
10. Integrate evidence and state limitations.
11. Present implications, not merely statistics or quotations.
12. Record decisions, data lineage and recommended follow-up.

The final report should distinguish findings, interpretation, assumptions and recommendations. A recommendation may be decisive even when evidence is imperfect, but uncertainty must be visible.

8. Ethics, privacy and governance

Marketing research involves power over participants and often uses personal or behavioural data. Ethical practice requires informed consent where applicable, voluntary participation, the right to withdraw, minimisation of data collection, secure storage, controlled access and appropriate retention limits. Researchers must avoid deception unless it is justified, proportionate and reviewed according to applicable standards.

Personal data should be collected and used for a specified legitimate purpose. Sensitive data require stronger safeguards. Product teams should distinguish aggregated analytics from identifiable records and should not assume that data available internally are automatically permissible for every new use. Applicable privacy law, contractual terms, industry codes and institutional policies must be checked.

AI tools can assist with transcription, coding, survey drafting, synthetic stimulus generation and anomaly detection. They can also reproduce training-data bias, fabricate summaries, expose confidential information or obscure how conclusions were produced. Human review, data minimisation, validation against source material and disclosure of material AI use are essential. Synthetic respondents should not be treated as substitutes for real customers without validation.

9. Illustrative case: improving adoption of a digital budgeting feature

Background. A fictional mobile banking firm launched an automated budgeting feature. Initial usage was low, although product analytics showed that customers who activated it were more likely to retain the application.

Problem. The team first assumed that customers did not understand the feature. That assumption was converted into broader questions: How do customers currently manage budgets? What benefits and risks do they perceive? At which onboarding step do they abandon the process? Which messages increase activation without creating distrust?

Analysis and approach. The team reviewed internal event data, support tickets and prior campaign reports. It then conducted purposive interviews with active users, non-users and customers who abandoned onboarding. Interviews revealed that “automatic categorisation” sounded convenient to some participants but intrusive to others. A short usability study identified confusing permission language.

Next, the team developed a survey to measure the prevalence of these concerns across customer segments. It used neutral wording and distinguished privacy concern, perceived usefulness and understanding. Finally, an experiment randomly assigned eligible users to one of three onboarding messages: a control message, a convenience-focused message and a transparency-focused message. The primary outcome was feature activation within fourteen days; secondary outcomes included permission completion and subsequent use.

Outcome. The transparency-focused message produced the strongest completion among privacy-sensitive users, while the convenience message performed better among users already comfortable with automated tools. The team therefore adopted segment-specific onboarding, simplified permissions and continued monitoring. The findings did not prove that messaging alone caused long-term retention; retention required a longer follow-up and controls for other product changes.

Learning points. Internal data diagnosed the behavioural pattern but not its meaning. Qualitative research explained motivations and language. The survey estimated the distribution of concerns. The experiment tested a decision intervention. Together, the designs were more useful than any single source, while each retained clear limitations.

Challenges and best practices

Common failures include beginning with a preferred method, using a convenient sample without disclosure, confusing correlation with causation, relying on a single dashboard metric, asking participants what they would do instead of observing behaviour, and reporting statistical precision without measurement validity.

Best practice is to use the simplest design that can answer the decision question, pilot instruments, preregister or document key decisions where feasible, separate exploratory from confirmatory analysis, triangulate important claims, protect participants, and make limitations decision-relevant. Product managers should also plan how evidence will change an action; research that cannot influence a decision may be interesting but inefficient.

Chapter Summary

Research design connects a marketing or product decision to credible evidence. Exploratory qualitative methods reveal meanings, motivations and context; quantitative methods estimate patterns, compare groups and, under suitable conditions, test causal effects; mixed-method designs integrate breadth with explanation. Primary data are tailored but costly, secondary data are efficient but may not fit, syndicated data provide benchmark coverage with limited customisation, and internal data are actionable but often incomplete or operationally defined. Validity concerns the defensibility of interpretation, while reliability concerns consistency. Bias can arise at every stage and is not solved by sample size alone. Effective researchers balance relevance, credibility, cost, feasibility, privacy and time, and communicate uncertainty honestly.

Key Terms

Research design: The plan linking research questions, data collection, sampling, measurement and analysis.

Management decision problem: The action or choice facing an organisation.

Research problem: The information problem that must be addressed to support a decision.

Exploratory research: Research used to clarify poorly understood issues and generate concepts.

Descriptive research: Research that measures characteristics, behaviours or attitudes.

Causal research: Research examining whether an intervention produces an outcome.

Qualitative research: Research using words, observations or other non-numerical evidence.

Quantitative research: Research representing observations numerically for comparison or analysis.

Mixed-method research: Deliberate integration of qualitative and quantitative approaches.

Primary data: Data collected specifically for the current research purpose.

Secondary data: Existing data collected previously for another purpose.

Syndicated data: Commercial data collected by a provider and supplied to multiple clients.

Internal data: Data generated within an organisation’s operations or systems.

Validity: The extent to which evidence supports the intended interpretation.

Reliability: The consistency of a measurement or procedure.

Sampling frame: The operational list or source from which a sample is selected.

Triangulation: Comparing evidence from different methods or sources.

Selection bias: Systematic distortion caused by how cases enter a study.

Internal validity: Confidence that a studied factor caused an observed effect.

External validity: The transferability of findings to other contexts.

Review Questions

Short Answer Questions

1. Distinguish a management decision problem from a research problem.
2. State two strengths and two limitations of qualitative research.
3. What is the difference between primary and secondary data?
4. Explain why reliability is necessary but not sufficient for validity.
5. Identify two forms of sampling bias.

Long Answer Questions

1. Compare exploratory, descriptive and causal research, giving a product-management example of each.
2. Explain how an exploratory sequential mixed-method design can be used to develop an advertising survey.
3. Evaluate the benefits and limitations of internal, syndicated and secondary data for market analysis.
4. Discuss how questionnaire wording and question order can affect validity.
5. Describe an end-to-end research workflow for testing a new product feature.

Analytical / Application Questions

1. A product team has falling retention but does not know why. Design a mixed-method research programme and justify the sequence of methods.
2. A survey reports that 82% of respondents intend to purchase, but the sample came from a brand’s social-media followers. Assess the credibility and likely biases of the finding.
3. An advertising experiment shows higher conversion for a new message, but the treatment group was exposed during a major competitor outage. Explain the threat to causal inference and propose a stronger design.

Multiple Choice Questions with Answers

Question 1

Which research purpose is primarily concerned with understanding an unfamiliar problem and generating possible explanations?

A. Exploratory research

B. Descriptive research

C. Causal research

D. Audit research

Correct Answer: A. Exploratory research

Question 2

Which method is most suitable for observing users attempting tasks with a product prototype?

A. Syndicated panel analysis

B. Usability testing

C. Census estimation

D. Financial forecasting

Correct Answer: B. Usability testing

Question 3

What is a major limitation of a convenience sample?

A. It always requires random assignment

B. It cannot contain qualitative information

C. It may not represent the target population

D. It prevents all forms of measurement

Correct Answer: C. It may not represent the target population

Question 4

Which source is collected specifically for the current research purpose?

A. Primary data

B. Secondary data

C. Syndicated data

D. Archived administrative data

Correct Answer: A. Primary data

Question 5

A measure gives similar results repeatedly but does not measure the intended construct. It is best described as:

A. Valid but unreliable

B. Reliable but potentially invalid

C. Both invalid and unreliable by definition

D. Externally valid but causal

Correct Answer: B. Reliable but potentially invalid

Question 6

In a randomised advertising experiment, random assignment primarily helps researchers:

A. Guarantee a large sample

B. Eliminate all non-response

C. Create comparable groups in expectation

D. Make the survey instrument valid

Correct Answer: C. Create comparable groups in expectation

Question 7

Which design begins with qualitative exploration and then uses a survey to assess how widespread the findings are?

A. Exploratory sequential mixed-method design

B. Explanatory sequential mixed-method design

C. Convergent design only

D. Post-test-only archival design

Correct Answer: A. Exploratory sequential mixed-method design

Question 8

Which problem occurs when relevant members of the target population are absent from the sampling frame?

A. Recall bias

B. Coverage bias

C. Interviewer bias

D. Survivorship bias

Correct Answer: B. Coverage bias

Question 9

In the expression p̂ = x ÷ n, what does x represent?

A. The total population size

B. The number of cases with the characteristic

C. The sample mean

D. The sampling error

Correct Answer: B. The number of cases with the characteristic

Question 10

Why should internal product-analytics data be evaluated before use in research?

A. Internal data are never numerical

B. Internal data cannot describe behaviour

C. Definitions, missing events and visibility limitations may distort interpretation

D. Internal data automatically violate privacy law

Correct Answer: C. Definitions, missing events and visibility limitations may distort interpretation