Opportunity Discovery and Customer Insight

Introduction

Product development begins before a product is designed. It begins with deciding which customer problem is worth solving. Opportunity discovery is the disciplined process of identifying meaningful unmet or poorly served needs, understanding the circumstances in which they arise, and assessing whether they justify product investment. Customer insight is the interpretation of evidence about customers’ behaviours, motivations, constraints and desired outcomes.

A product team should not treat customer research as a search for attractive ideas or as a procedure for confirming an executive’s preferred solution. Its purpose is to reduce uncertainty. Research cannot eliminate uncertainty because customers may change, markets evolve and observed behaviour is always incomplete. However, systematic research can improve the quality of product decisions by replacing assumptions with evidence and by making remaining assumptions visible.

This chapter explains how market research, observation, interviews, jobs-to-be-done, personas, journey maps and problem framing can be combined into an opportunity-discovery process. It also examines research bias, data limitations, ethical responsibilities and the implications of artificial intelligence for customer insight.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish markets, customer needs, opportunities, problems and solutions.
2. Plan and combine secondary research, observation and customer interviews.
3. Apply the jobs-to-be-done perspective to understand desired progress.
4. Construct evidence-based personas and customer journey maps.
5. Frame problems without prematurely prescribing solutions.
6. Recognise sampling, response, interpretation and confirmation biases.
7. Assess the quality and limitations of qualitative and quantitative evidence.
8. Translate customer insight into prioritised product opportunities.

1. From Market Signals to Product Opportunities

A market is a group of present or potential exchanges involving customers, providers and alternatives. A segment is a subset of that market with relatively similar characteristics or needs. A customer is the person or organisation that receives value, while a user is the person who interacts directly with the product. In business-to-business contexts, the buyer, user, administrator, approver and economic decision-maker may all be different people.

A need is a condition or outcome that matters to a person. A problem is an undesirable gap between a current situation and a desired situation. An opportunity is a promising, evidence-supported possibility for creating value by addressing a problem or need. A solution is a proposed response, such as a feature, service, process or business model.

These concepts should not be collapsed. For example, “customers need a mobile application” is usually a solution hypothesis. A more useful formulation might be “commuters need to know whether a delayed service will affect their arrival time, without repeatedly checking several information sources.” The second statement identifies a situation and desired outcome while leaving multiple solutions possible.

A practical opportunity statement normally identifies:

• a clearly described customer or user group;
• a situation, trigger or context;
• an important unmet or poorly served need;
• evidence that the problem occurs and matters;
• existing alternatives and their shortcomings; and
• a plausible basis for creating value.

Opportunity discovery is therefore both expansive and disciplined. Teams explore many possible needs but apply evidence and strategic criteria before committing resources.

2. The Opportunity-Discovery Process

A useful process is iterative rather than strictly linear:

1. Define the decision context. Clarify what decision research must support, such as entering a market, improving onboarding or selecting a target segment.
2. Map assumptions. Record beliefs about customers, needs, behaviour, competition and feasibility. Classify each assumption by importance and uncertainty.
3. Explore the context. Conduct secondary research, market scanning and stakeholder conversations.
4. Observe and listen. Study behaviour in natural settings and interview relevant participants.
5. Synthesize evidence. Identify recurring patterns, contradictions, workarounds and significant moments.
6. Frame opportunities. Convert findings into problem statements and opportunity hypotheses.
7. Prioritise. Compare opportunities using customer importance, frequency, severity, strategic fit and ability to act.
8. Test learning. Use prototypes, experiments or additional research to test the riskiest assumptions.

The process is cyclical. A surprising interview may require new observation; an apparently attractive opportunity may be rejected after testing willingness to change; and a journey map may reveal a stakeholder who was omitted from the original research.

3. Market Research and Secondary Evidence

Secondary research uses information already collected by others. Sources include government statistics, industry reports, academic studies, regulatory publications, company reports, competitor websites, app-store reviews, support tickets, search trends and professional communities.

Secondary research is valuable because it provides context quickly and may reveal market size, demographic change, regulations, adoption patterns and established alternatives. It can also prevent a team from asking customers questions that published evidence already answers.

However, secondary sources must be evaluated rather than accepted uncritically. Consider:

Quantitative market data can indicate scale but often explains little about causes. A report may show declining subscription renewal without revealing whether customers object to price, lack necessary functionality, experience poor support or have changed jobs. Qualitative research is then required to understand meaning and context.

Market size estimates also require caution. A theoretical population is not equivalent to an immediately reachable market. Product managers should distinguish the total population with a relevant characteristic from the subset that can be served and the subset likely to adopt a particular offering.

4. Observation: Studying Behaviour in Context

Observation examines what people do, where they do it and what surrounds the behaviour. It is especially useful when people cannot accurately recall routine actions, when behaviour is socially sensitive, or when stated intentions differ from actual conduct.

Observation may be:

• Structured or unstructured: using a predefined checklist or exploring openly.
• Participant or non-participant: joining the activity or remaining separate.
• Overt or covert: participants know they are being observed or do not. Covert observation raises substantial ethical concerns and is generally inappropriate without strong justification and governance.
• Naturalistic or task-based: examining normal work or asking participants to complete a realistic task.

A good observation plan specifies the setting, participants, time period, behaviours of interest and recording method. Notes should distinguish description from interpretation. “The employee opened three systems and copied an order number” is an observation. “The employee is frustrated by inefficient software” is an interpretation that requires supporting evidence.

Useful observation targets include workarounds, interruptions, hand-offs, queues, errors, repeated searches, physical artefacts, informal tools and points at which people ask for help. These often reveal needs that customers do not spontaneously mention because they have normalised them.

Researchers must minimise disruption and protect privacy. In workplaces, permission from an individual may not be sufficient if confidential records, customers or colleagues are visible. Sensitive data should not be copied unnecessarily, and notes should be anonymised.

5. Customer Interviews

Interviews are purposeful conversations designed to understand experiences, decisions, meanings and constraints. They are not sales demonstrations and should not be treated as votes on a proposed feature.

Planning an Interview

Before recruiting, define the research questions. For example:

• What triggers the customer to seek a solution?
• What did the customer do most recently?
• Which alternatives were considered?
• What was difficult, costly or risky?
• How was success judged?
• Who else influenced the decision?

Recruit for relevance rather than convenience. Include people who recently experienced the situation, not only enthusiastic existing customers. In a B2B study, recruit across roles because the user’s experience may differ from that of the purchaser or administrator.

Conducting an Interview

Begin with consent, purpose, confidentiality limits and permission to record. Use open questions and ask for concrete recent examples. “Tell me about the last time you submitted an expense claim” is generally more useful than “Would you use an automated expense feature?” Follow up with probes such as “What happened next?”, “How did you decide?”, “What did you do when that failed?” and “Why did that matter?”

Avoid leading questions, double-barrelled questions, jargon and hypothetical claims. People are poor predictors of future behaviour and may provide socially desirable answers. A statement such as “I would definitely use it” is weaker evidence than repeated recent behaviour involving time, money or a workaround.

A semi-structured guide balances consistency with discovery. The same core topics are addressed across interviews, while the researcher follows relevant unexpected information. Interviewers should record uncertainty, contradictions and their own assumptions.

Analysing Interviews

Analysis may involve transcription, coding and comparison. Codes can describe topics, behaviours, contexts, emotions, barriers and outcomes. Teams should search for patterns across participants while preserving exceptions. Frequency alone does not determine importance: a rare problem may be critical, while a common minor irritation may not justify investment.

A simple evidence record can include the finding, supporting quotations or observations, participant characteristics, confidence level and implications. This prevents memorable anecdotes from being presented as universal truths.

6. Jobs-to-be-Done

The jobs-to-be-done (JTBD) perspective focuses on the progress a person is trying to make in a particular circumstance. Customers do not simply buy product attributes; they “hire” products or services to achieve functional, emotional or social outcomes.

A job statement usually includes:

When [situation], I want to [motivation or progress], so that [desired outcome].

For example: “When preparing for a complex client meeting, I want to find the latest approved project information quickly, so that I can answer questions confidently.” This statement does not assume a specific interface or technology.

JTBD analysis commonly examines:

• Functional job: the practical task to accomplish.
• Emotional job: how the person wants to feel.
• Social job: how the person wants to be perceived.
• Circumstances and triggers: events that activate the job.
• Desired outcomes: measures of improved performance.
• Competing solutions: products, services, people or improvised methods already used.

The “job” is not always identical to a customer’s stated task. A person may say that they want a budgeting application, while the broader job is to feel in control before an uncertain financial event. Understanding the broader job can reveal alternative opportunities, but researchers must avoid inventing psychological explanations unsupported by evidence.

JTBD is particularly useful for identifying non-consumption: situations in which people do nothing, use a manual workaround or delay action because current solutions are too costly, complex or inaccessible.

7. Personas as Research-Based Representations

A persona is a concise representation of a meaningful user pattern. It combines relevant behaviours, goals, contexts, constraints and needs into a form that supports design and prioritisation.

A credible persona is based on evidence, not on an invented biography. It should normally include:

• a descriptive name or role rather than a misleading stereotype;
• goals and desired outcomes;
• behaviours and routines;
• situations and triggers;
• constraints, anxieties and accessibility needs;
• current alternatives and workarounds;
• relevant digital confidence or organisational context; and
• evidence sources and confidence level.

Personas are not a substitute for segmentation. A market segment may be defined by industry, geography or purchasing characteristics, whereas a persona emphasises behavioural patterns relevant to product use. One individual may resemble different personas in different contexts.

Avoid demographic caricatures. Age, gender or job title should be included only when they help explain behaviour or access. Personas can cause harm when they become rigid labels, exclude minority users or are used to justify decisions without revisiting the evidence. Teams should document who is absent from the research and update personas when new evidence contradicts them.

8. Customer Journey Mapping

A customer journey map represents stages in an experience and the customer’s actions, thoughts, emotions, interactions and pain points. It may cover the journey before, during and after product use.

A journey map normally identifies:

1. Stages: for example, recognise need, search, evaluate, purchase, use and obtain support.
2. Customer actions: what the person does at each stage.
3. Touchpoints: interactions with people, systems, channels or artefacts.
4. Questions and expectations: what the customer needs to know.
5. Emotional experience: confidence, uncertainty, effort or frustration.
6. Pain points and workarounds: obstacles and compensating behaviour.
7. Internal ownership: teams or partners responsible for the experience.
8. Evidence and opportunities: what is known and what requires testing.

A journey map should represent a specific persona and scenario. A map that attempts to describe every customer and every possible path becomes too general to guide action. The map should also distinguish the customer’s journey from the organisation’s internal process. These may differ substantially: an organisation may view an issue as “resolved” when the customer is still uncertain.

Journey maps are hypotheses informed by evidence, not objective photographs of reality. Annotating each claim with its source and confidence improves responsible use.

9. Problem Framing and Opportunity Statements

Problem framing converts research into a focus for action without prematurely selecting a solution. A strong problem statement describes who is affected, in what situation, what difficulty occurs and why it matters.

A useful structure is:

[User] needs a way to [desired progress] when [context], because [evidence-based difficulty or consequence].

An opportunity question can then be expressed as:

How might we help [user] achieve [outcome] in [context] while respecting [important constraint]?

The “How might we” form is useful only when it remains specific. “How might we improve everything about education?” is too broad. “How might we help first-year students identify the next credible study task after receiving confusing assignment feedback?” provides a more actionable boundary.

Good framing separates the problem from the proposed solution. “Users need a dashboard” is solution-led; “team leaders cannot detect delayed work until deadlines are missed” is problem-led. Problem framing should also avoid blaming users. Behaviour that appears irrational may be a reasonable response to incentives, information gaps or organisational constraints.

Teams can prioritise opportunities using a qualitative matrix or a scoring model. A simple weighted score may be written as:

Here, P is the priority score; I is customer importance; F is frequency; S is severity or consequence; A is strategic alignment; C is implementation cost or difficulty; and w₁ through w₅ are weights. All criteria should be scored on a defined common scale, such as 1 to 5. The equation is a decision aid, not an objective truth. It assumes that the criteria are sufficiently comparable and that the team’s weights reflect its explicit priorities. Scores should be accompanied by evidence and confidence ratings.

10. Bias, Validity and Data Limitations

Research quality depends not only on the number of participants but also on the fit between evidence and decision. Important sources of bias include:

• Sampling bias: recruited participants differ systematically from the target population.
• Self-selection bias: people with strong opinions are more likely to participate.
• Non-response bias: those who decline may differ from respondents.
• Recall bias: participants misremember past events.
• Social desirability bias: people give answers that appear acceptable.
• Interviewer bias: wording, tone or expectations influence responses.
• Confirmation bias: researchers notice evidence supporting existing beliefs.
• Survivorship bias: successful users are studied while abandoned or failed cases are ignored.
• Availability bias: memorable incidents are treated as typical.
• Platform or instrumentation bias: data reflects what a system records, not all behaviour.
• Algorithmic bias: automated systems reproduce or amplify unequal representation in their training or usage data.

Triangulation improves confidence by comparing different methods, sources or researchers. For example, interview claims about repeated manual work can be compared with observation, support tickets and workflow data. Triangulation does not guarantee truth; sources may share the same underlying bias. Contradictions should be investigated rather than averaged away.

Quantitative data also has limitations. Correlation does not establish causation. A relationship between usage and retention may reflect a third factor, such as customer size or implementation support. Dashboards can create false precision when definitions are unstable or missing data is ignored. Teams should document the population, time period, definitions, exclusions and collection process for every important metric.

Qualitative research has different standards. Its strength is depth, context and meaning rather than statistical representation. A small study can reveal a mechanism or important edge case, but it cannot automatically estimate prevalence. Claims should therefore match the evidence: “some participants described…” is different from “most customers…” and both differ from “the market needs…”.

11. Ethics, Privacy and Governance

Customer research involves power, privacy and potential harm. Researchers should obtain informed consent, explain participation and recording, allow withdrawal where practicable and avoid deception unless a properly approved method requires it. Participants should know how data will be used, stored and shared.

Collect only data necessary for the research purpose. Sensitive information should be minimised, access-controlled and de-identified where possible. Researchers must comply with applicable privacy and data-protection requirements, organisational policies and sector regulations. Children, employees, patients and financially vulnerable people may require additional safeguards.

Compensation should recognise time without becoming coercive. Recruitment should avoid exploiting people who feel unable to refuse. Findings should not expose an identifiable participant or lead to unfair exclusion. Accessibility should be considered in recruitment, communication, interview format and research materials.

When research is conducted inside a workplace, the organisation paying for research may not be the person whose interests are most at risk. Product managers should consider whether employees can speak freely, whether monitoring is excessive and whether findings could be used for disciplinary purposes.

12. Technology and AI in Customer Insight

Digital analytics, session recordings, speech transcription, survey platforms and text analysis can make research faster and more scalable. AI can help cluster feedback, summarise transcripts, identify recurring terms and generate alternative interview probes. These tools are aids to analysis, not replacements for judgement.

Risks include inaccurate transcription, fabricated summaries, loss of context, exposure of confidential information, demographic bias and overconfident interpretation. A product team should establish data-access controls, verify AI-generated outputs against source material, retain an audit trail and disclose relevant automated processing where required. Sensitive customer data should not be uploaded to an external model without authorisation and appropriate safeguards.

Automated sentiment analysis is particularly limited. The same words may express different meanings across cultures and contexts, and neutral or polite language may conceal serious problems. AI-generated personas can also create a false sense of evidence if they are based on stereotypes rather than observed customers.

13. Applied Case: Improving Appointment Attendance

The following is an illustrative case, not a report of an actual organisation.

Background

A community health service reported that many patients did not attend scheduled appointments. Management proposed sending more text-message reminders and asked the product team to design a reminder application.

Problem and Research Approach

The team first reframed the decision: understand why appointments were missed and identify opportunities to reduce avoidable non-attendance. It reviewed attendance records and support logs, then interviewed patients, reception staff and clinicians. Researchers observed the booking and reminder process, with consent and appropriate privacy controls.

The evidence did not support a single explanation. Some patients forgot appointments, but others received reminders at a time when they were working night shifts. Some could not interpret changing appointment details, some lacked reliable transport and some avoided calling because rescheduling was embarrassing or difficult. Staff used several disconnected systems and occasionally entered changes late.

Synthesis

A journey map showed pain points across booking, preparation, travel and rescheduling. The JTBD statement was: “When my circumstances change after booking, I want to understand my options and change the appointment without a stressful phone call, so that I can receive care at a feasible time.” This revealed that attendance was not simply a memory problem.

The team developed two personas based on behavioural patterns: a patient with unstable work schedules and a patient with transport and accessibility constraints. Demographic details were included only where they affected access. The team also documented non-users and patients who had stopped attending, rather than relying only on regular attendees.

Approach and Outcome

Instead of immediately building a new application, the service tested several low-cost changes: clearer appointment messages, an accessible rescheduling link, multiple reminder timing options and a staff process for rapidly updating changes. A small pilot compared outcomes with a similar service period, while recognising that seasonal and service differences limited causal certainty.

Early results suggested fewer failed appointments and fewer avoidable calls, but the team did not claim that reminders alone caused the improvement. It continued monitoring groups who might be disadvantaged by digital-only communication and retained telephone and in-person alternatives.

Learning Points

The case demonstrates that an executive’s proposed feature may be a response to a symptom rather than the underlying problem. Combining records, observation, interviews and journey mapping exposed system-level causes. It also illustrates the importance of accessibility, alternative channels, cautious interpretation of pilot results and attention to people who are absent from conventional customer feedback.

Best Practices for Product Teams

• Begin with a decision and assumptions, not a preferred feature.
• Recruit for recent, relevant experience and include non-users where possible.
• Ask about actual past behaviour before discussing future concepts.
• Observe workarounds and system interactions, not only verbal opinions.
• Separate evidence, interpretation and hypothesis in research records.
• Use personas and journey maps as revisable tools rather than permanent facts.
• Triangulate methods and actively search for disconfirming evidence.
• State uncertainty, missing data and limits of generalisation.
• Protect participants through consent, privacy, accessibility and fair treatment.
• Prioritise opportunities before designing solutions, then test the riskiest assumptions.

Chapter Summary

Opportunity discovery is the evidence-informed search for meaningful customer problems and valuable areas for product action. Market research provides context and scale, while observation reveals behaviour in context and interviews explore experiences, decisions and meanings. Jobs-to-be-done focuses on the progress customers seek; personas represent recurring behavioural patterns; journey maps show experiences across stages and touchpoints; and problem framing converts evidence into an actionable focus without prescribing a solution.

Research is never neutral or complete. Sampling, recall, social desirability, interviewer expectations, confirmation bias, survivorship and algorithmic bias can distort findings. Qualitative evidence provides depth but does not automatically establish prevalence; quantitative evidence can reveal patterns but does not by itself explain causation. Triangulation, transparent documentation, explicit confidence levels and active attention to missing perspectives improve decision quality. Ethical research requires consent, privacy, accessibility, data minimisation and responsible use of technology. The professional product manager turns uncertain evidence into testable opportunity hypotheses rather than presenting assumptions as facts.

Key Terms

Opportunity discovery: The process of identifying and evaluating promising customer problems or unmet needs.

Customer insight: An evidence-based interpretation of customer behaviour, motivation, context and constraints.

Market segment: A group within a broader market sharing relevant characteristics or needs.

User: A person who directly interacts with a product or service.

Unmet need: A meaningful need that is absent, inadequately served or served with excessive effort or cost.

Secondary research: Analysis of information previously collected by another person or organisation.

Observation: Systematic study of behaviour in a natural or realistic setting.

Semi-structured interview: An interview using consistent topics while allowing adaptive follow-up.

Jobs-to-be-done: A perspective that examines the progress a person seeks in a particular circumstance.

Functional job: The practical task a customer is trying to accomplish.

Persona: An evidence-based representation of a recurring user pattern.

Customer journey map: A representation of customer actions, experiences and touchpoints across a scenario.

Touchpoint: An interaction between a customer and an organisation, product, person or channel.

Problem framing: Defining a customer difficulty and its context without prematurely choosing a solution.

Opportunity statement: A concise description of a promising, evidence-supported area for action.

Triangulation: Comparing different sources, methods or researchers to strengthen interpretation.

Sampling bias: Systematic distortion caused by a sample that does not represent the relevant population.

Confirmation bias: The tendency to seek or interpret evidence as support for existing beliefs.

Data limitation: A condition that restricts the accuracy, completeness, interpretation or generalisation of evidence.

Review Questions

5 Short Answer Questions

1. Distinguish a customer problem from a proposed product solution.
2. State two advantages and two limitations of customer interviews.
3. What is the purpose of a jobs-to-be-done statement?
4. Identify three components commonly included in a customer journey map.
5. Explain why a persona should be based on behavioural evidence rather than stereotypes.

5 Long Answer Questions

1. Explain how secondary research, observation and interviews can be combined during opportunity discovery.
2. Discuss the value and limitations of personas and journey maps in product decision-making.
3. Describe major sources of bias in customer research and propose ways to reduce their effects.
4. Explain how problem framing prevents premature commitment to a solution.
5. Discuss the ethical responsibilities of a product team collecting and analysing customer data.

3 Analytical / Application Questions

1. A team believes that customers need a mobile application because support calls are increasing. Design a research plan that tests this assumption without presupposing the solution.
2. Using the illustrative appointment-attendance case, identify the relevant users, jobs, touchpoints and opportunity statements. Explain which evidence would be needed before prioritising a solution.
3. A survey reports that 80% of respondents want an advanced feature, but product usage data shows that only 10% use the related basic function. Analyse at least four possible explanations and recommend further research.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an opportunity from a solution?

A. An opportunity describes a promising customer problem or need; a solution describes a proposed response.

B. An opportunity is always a product feature; a solution is always a market segment.

C. An opportunity is a confirmed sales forecast; a solution is a research method.

D. An opportunity is a customer demographic; a solution is a business regulation.

Correct Answer: A. An opportunity describes a promising customer problem or need; a solution describes a proposed response.

Question 2

Which interview question is most likely to produce evidence about actual behaviour?

A. Would you use an automated expense feature?

B. Do you think expense claims are important?

C. Tell me about the last time you submitted an expense claim.

D. Would you recommend our proposed application?

Correct Answer: C. Tell me about the last time you submitted an expense claim.

Question 3

What is the central focus of the jobs-to-be-done perspective?

A. The age and income of a target customer

B. The progress a person seeks in a particular circumstance

C. The number of features offered by competitors

D. The visual design of a product interface

Correct Answer: B. The progress a person seeks in a particular circumstance

Question 4

Which practice best improves the credibility of a persona?

A. Adding an elaborate fictional biography

B. Using only demographic stereotypes

C. Basing it on recurring behavioural evidence and documenting its sources

D. Making it represent every possible customer

Correct Answer: C. Basing it on recurring behavioural evidence and documenting its sources

Question 5

Why should researchers distinguish observation from interpretation in field notes?

A. To ensure that all notes are numerical

B. To prevent interpretations from being mistaken for directly observed evidence

C. To eliminate the need for analysis

D. To make interviews unnecessary

Correct Answer: B. To prevent interpretations from being mistaken for directly observed evidence

Question 6

Which is an example of survivorship bias?

A. Studying only successful users while ignoring people who abandoned the product

B. Recruiting participants from two different regions

C. Comparing interview findings with support-ticket data

D. Asking participants about a recent experience

Correct Answer: A. Studying only successful users while ignoring people who abandoned the product

Question 7

A journey map is most useful when it:

A. Describes every customer and every possible experience at once

B. Represents a specific persona and scenario using evidence about stages and touchpoints

C. Lists only the organisation’s internal departments

D. Contains proposed features but no customer evidence

Correct Answer: B. Represents a specific persona and scenario using evidence about stages and touchpoints

Question 8

Which statement about qualitative research is most accurate?

A. A small qualitative study automatically estimates market prevalence.

B. Qualitative research is useful for depth and context but does not automatically establish frequency.

C. Qualitative research is free from researcher interpretation.

D. Qualitative findings cannot identify product opportunities.

Correct Answer: B. Qualitative research is useful for depth and context but does not automatically establish frequency.

Question 9

Which action reflects responsible use of AI in customer research?

A. Accepting an AI-generated summary without checking the transcript

B. Uploading confidential customer data to any public model

C. Using AI to assist analysis while verifying outputs against source material

D. Treating automated sentiment scores as customer truth

Correct Answer: C. Using AI to assist analysis while verifying outputs against source material

Question 10

Which statement is the best problem frame?

A. Users need a dashboard.

B. The company should build a reminder application.

C. Team leaders cannot detect delayed work until deadlines are missed.

D. Customers want more advanced technology.

Correct Answer: C. Team leaders cannot detect delayed work until deadlines are missed.