Keyword and Search Demand Research

Introduction

Keyword and search demand research is the systematic process of discovering, evaluating, grouping, and prioritising the language people use when seeking information, products, services, or solutions through search engines. It is a foundational activity in search engine optimisation (SEO) and search advertising because it connects audience needs with content, website architecture, landing pages, and campaign decisions.

A keyword is not merely a word entered into a search box. It is evidence of a possible information need, problem, preference, or commercial intention. Effective research therefore asks several questions at once: What does the searcher want? How often is the query used? How competitive is the result page? What type of content satisfies the query? Is the searcher ready to act? Can the organisation provide a credible answer?

Search demand research should not be reduced to selecting the keywords with the largest volume. High-volume terms may be broad, expensive, competitive, or poorly aligned with organisational goals. Conversely, a specific long-tail query may have modest volume but a much higher probability of producing a qualified lead or sale. Professional practice combines quantitative evidence from tools with qualitative interpretation of search results and first-party business knowledge.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain seed terms, keyword variants, long-tail queries, entities, topics, and search intent.
2. Use keyword tools, search engine results pages (SERPs), competitor research, trend data, and first-party insights responsibly.
3. Evaluate demand, seasonality, competition, difficulty, relevance, and commercial value.
4. Create keyword clusters based on meaning, intent, and likely ranking-page requirements.
5. Map clusters to pages and content types without creating unnecessary duplicate pages.
6. Recognise limitations, ethical issues, privacy requirements, and the effects of changing search technologies.

1. From Keywords to Search Behaviour

1.1 Seed terms and keyword expansion

A seed term is a broad starting expression from which related queries are developed. For a sustainable clothing retailer, seeds might include “organic clothing,” “linen shirts,” “recycled fabric,” and “ethical fashion.” Seeds can be obtained from product catalogues, customer-service records, sales conversations, internal site search, industry vocabulary, and interviews with subject-matter experts.

Seed terms are not final targets. They are inputs to a research process. Expansion may produce:

• Synonyms: “running shoes” and “jogging shoes.”
• Modifiers: “women’s,” “waterproof,” “near me,” “for beginners,” or “under £100.”
• Questions: “how to wash linen” or “which running shoes prevent shin splints?”
• Comparisons: “Brand A versus Brand B.”
• Local variants: “dentist in Bristol.”
• Transactional variants: “buy,” “price,” “quote,” “subscription,” or “discount.”
• Problem-led variants: “why are my shoes causing heel pain?”

Expansion must preserve meaning. Automatically generated lists often contain irrelevant, duplicated, or misleading phrases. Human review remains essential.

1.2 Queries, topics, and entities

A query is an individual search expression. A topic is a broader subject that may be represented by many queries. An entity is a distinct person, place, organisation, product, concept, or thing that search systems can identify and relate to other entities. For example, “best laptops for graphic design” is a query; “graphic-design laptops” is a topic; and a particular laptop model is a product entity.

This distinction matters because modern search systems evaluate topical coverage, relationships, context, and usefulness rather than relying only on exact-match phrases. A page about “home solar panels” may need to discuss installation, electricity generation, warranties, roof suitability, maintenance, and costs even if those terms were not in the original seed list.

1.3 Search intent

Search intent is the likely purpose behind a query. A practical classification includes:

Intent is inferred, not directly observed. A query may have mixed intent, and intent can change over time. The SERP is one of the strongest practical indicators. Product pages, advertisements, maps, videos, definitions, forums, and editorial guides each signal different expectations.

2. Evidence Sources for Demand Research

2.1 Keyword platforms

Keyword tools commonly provide estimated search volume, related terms, trends, cost-per-click (CPC), paid competition, and sometimes an SEO difficulty score. Examples include search-engine planning tools, specialist SEO platforms, and suites supplied by advertising networks.

These values are estimates, not measurements of every search. Definitions differ between providers. “Volume” may represent an average over a period, a grouped set of close variants, or a rounded range. CPC reflects advertising-market conditions and is not the same as organic ranking value. Difficulty scores are proprietary indexes and should be used comparatively within one tool rather than treated as universal scientific measurements.

2.2 SERP analysis

SERP analysis examines the actual results for a query. Record:

• The dominant content format: guides, category pages, product pages, videos, maps, or forums.
• The apparent intent and stage of the customer journey.
• Recurring subtopics and questions.
• Brand strength and authority of ranking domains.
• Features such as featured snippets, product results, local packs, “People also ask,” and AI-generated summaries.
• Evidence of freshness requirements, such as current prices, regulations, or software versions.

SERP analysis prevents a common error: creating a page that is well optimised for a phrase but incompatible with what searchers and search engines expect.

2.3 Competitor research

Competitor research should include direct business competitors and search competitors. A search competitor is any site that repeatedly appears for relevant queries, even if it sells a different product. Examine competitors’ page types, information depth, internal linking, titles, headings, structured data, author credentials, reviews, and unanswered questions. The aim is not to copy content. It is to identify the level of usefulness required and opportunities for legitimate differentiation.

Competitor rankings do not prove that their content is accurate or suitable. Their visibility may result from domain history, links, brand demand, technical performance, or temporary algorithmic conditions.

2.4 Trends and seasonality

Trend data reveals how interest changes over time and across locations. It is useful for identifying seasonal peaks, emerging terminology, declining phrases, and regional differences. Relative trend indexes should not be confused with absolute search volume: a rising index indicates increasing share of observed interest, not a specific number of searches.

A retailer might compare searches for “winter coats” across months and plan content before demand rises. A tax adviser may identify a recurring annual peak before filing deadlines. Trend interpretation should consider external causes such as news, weather, regulation, product launches, and changes in vocabulary.

2.5 First-party evidence

First-party data is often more relevant than generic tool estimates. Useful sources include:

• Search Console queries and landing-page performance.
• Internal site-search terms.
• CRM and sales-call language.
• Customer-service tickets and chat transcripts.
• On-site searches that return no results.
• Product reviews and survey responses.
• Conversion paths and assisted conversions.
• Geographic, device, and audience data collected with appropriate consent.

First-party data should be interpreted carefully. A query with few impressions may still represent a valuable specialist need, while a high-impression query may produce little business value.

3. Evaluating Keyword Opportunity

3.1 Relevance, demand, competition, and value

A useful evaluation considers four broad dimensions:

1. Relevance: Does the query match the organisation’s offer, expertise, audience, and geographic scope?
2. Demand: Is there evidence of recurring or growing interest?
3. Competition: How difficult will it be to earn visibility and satisfy the SERP standard?
4. Value: If the organisation attracts the searcher, is there a realistic path to engagement, lead generation, revenue, or another objective?

These dimensions should be documented rather than collapsed into an unexplained score. A simple prioritisation model may help teams organise work, but it is a decision aid, not an objective truth.

For example, a team may define a weighted priority index:

Here, P is a prioritisation index; R, D, V, and C are normalised scores for relevance, demand, value, and competition; and w_R, w_D, w_V, and w_C are non-negative weights. Scores should use the same defined scale, such as 0 to 10, and the weights should be agreed before ranking opportunities. The model is applicable only as a transparent internal framework; it does not predict rankings or revenue.

3.2 Search volume and click-through rate

Search volume alone does not equal traffic. Click-through rate (CTR) is commonly calculated as:

Clicks are visits attributed to the result or advertisement, and impressions are occasions when it was displayed. Both must refer to the same channel, period, query set, and reporting definition. CTR varies with position, SERP features, device, brand familiarity, query intent, and competition. A featured answer or local pack may reduce clicks to traditional organic results even when impressions remain high.

Potential traffic is therefore better treated as a range than as a guaranteed outcome. Estimates should state their assumptions about ranking position, impressions, CTR, seasonality, and cannibalisation by other pages.

3.3 Difficulty and commercial value

Difficulty may be estimated from the authority of ranking domains, link profiles, content quality, SERP features, and the number of capable competitors. It is not a fixed property of a keyword. A new competitor, algorithm change, or change in search intent can alter the competitive environment.

Commercial value can be assessed through CPC, conversion data, lead quality, average order value, margin, customer lifetime value, and strategic importance. CPC is useful evidence because advertisers often pay more for queries associated with valuable actions, but it is affected by bidding, industry, geography, and seasonality. It should not be used as a substitute for first-party conversion evidence.

4. Building Keyword Clusters

A keyword cluster is a set of queries that can be served effectively by one page or one closely related content asset. Clustering should consider semantic similarity, intent, audience, search-result overlap, and the depth of answer required.

A practical workflow is:

1. Collect and clean candidate terms.
2. Remove irrelevant, duplicated, and impossible-to-serve queries.
3. Label intent, audience, location, product stage, and freshness.
4. Review SERPs for the most important variants.
5. Group queries whose search results and user needs substantially overlap.
6. Separate queries that require different page types or distinct answers.
7. Assign a primary topic and supporting subtopics to each cluster.
8. Validate the proposed page against business owners and users.

For example, “best waterproof hiking boots,” “waterproof hiking boots for beginners,” and “hiking boots comparison” may belong to a commercial-investigation cluster. “Buy waterproof hiking boots size 9” is more transactional and may belong to a category or product experience. “How to waterproof hiking boots” requires instructional content and should not be forced onto a product page merely because the words overlap.

Clustering prevents two opposite problems. Keyword stuffing places many unrelated phrases on one page. Keyword cannibalisation occurs when several pages compete for substantially the same intent without a clear reason for their existence. However, not every repeated phrase is cannibalisation: different locations, audiences, products, or stages may justify separate pages.

5. Keyword Mapping and Information Architecture

Keyword mapping assigns each priority cluster to a URL or proposed content asset. A useful map contains:

Mapping should reflect the customer journey. Informational content can introduce a problem; comparison content can help evaluation; category and product pages can support transactions; post-purchase content can reduce support demand and encourage retention. Internal links should express meaningful relationships, such as a buying guide linking to relevant categories and a product page linking to care instructions.

A map is not a permanent contract. Search behaviour, products, regulations, and organisational priorities change. Review it after major site changes, product launches, algorithmic shifts, and significant changes in Search Console data.

6. Worked Example: A Local Bicycle-Service Business

The following is an illustrative case.

Background

A bicycle-service business offers repairs, annual servicing, e-bike diagnostics, and bicycle fitting in one metropolitan area. Its existing website has a home page and a generic “services” page but receives few organic enquiries.

Problem

The owner believes that the business should target “bike repair,” a broad and competitive phrase. The phrase does not reveal whether the searcher needs an emergency repair, a price estimate, a specific service, or a nearby provider.

Analysis

Research combines customer-service language, Search Console data, keyword-tool suggestions, trend data, and SERP inspection. The resulting groups include:

SERP analysis shows that local queries display map results, opening hours, reviews, and location evidence. Informational queries display guides and videos. Therefore, one generic page cannot efficiently satisfy all identified needs.

Approach and solution

The business creates a local service page with accurate address and contact information, a separate e-bike diagnostics page, a transparent pricing page, and a maintenance guide. Each page has a distinct purpose, clear calls to action, evidence of expertise, and internal links. The business also improves its local profile, requests reviews ethically from genuine customers, and adds service-area information without creating thin pages for every nearby town.

Outcome

After implementation, success is assessed using qualified calls, booking forms, direction requests, non-branded impressions, and completed jobs rather than rankings alone. The business expects different pages to perform different roles: the guide may generate awareness, while local and specialist pages may generate direct enquiries.

Learning points

The case demonstrates that keyword research is a business and user research activity. Intent, geography, page type, and conversion path are as important as volume. A small set of highly relevant clusters can be more valuable than a large undifferentiated keyword list.

7. Technology, AI, and Changing Search Results

Automation can accelerate collection, deduplication, tagging, clustering, and anomaly detection. Natural-language tools can identify recurring questions in reviews or support transcripts. Generative AI can suggest seed expansions and draft taxonomies, but it may invent phrases, merge distinct intents, reproduce bias, or misunderstand specialist terminology. Outputs require human validation against actual SERPs, first-party evidence, and organisational expertise.

Search increasingly includes visual results, voice interfaces, shopping modules, local results, answer interfaces, and AI-generated summaries. This increases the importance of clear entities, structured information, factual accuracy, distinctive expertise, and content that deserves citation or referral. It does not eliminate traditional research: users still express needs, search systems still interpret intent, and organisations still need appropriate destinations.

8. Ethics, Privacy, and Governance

Keyword research can reveal sensitive interests, including health, finance, sexuality, religion, or political views. Teams must avoid treating sensitive queries as permission to profile individuals. Personal data from analytics, CRM systems, or transcripts should be collected and used according to applicable privacy law, organisational policy, purpose limitation, access controls, retention rules, and consent requirements where relevant.

Ethical practice also requires:

• Avoiding deceptive titles and content that promise answers the page does not provide.
• Not creating doorway pages or large sets of near-duplicate local pages.
• Representing products, prices, qualifications, and reviews accurately.
• Disclosing commercial relationships where recommendations are monetised.
• Using inclusive language and testing whether keyword choices exclude important audiences.
• Keeping regulated advice, such as medical or financial information, subject to qualified review.

Governance should record data sources, collection dates, assumptions, owners, approval status, and review dates. This creates an audit trail and helps distinguish evidence from opinion.

9. Limitations and Best Practices

Keyword tools cannot observe every search, intent classification is uncertain, and volume estimates may be rounded or delayed. Competitor data may be incomplete, trends may be distorted by news events, and rankings fluctuate. Conversion data may be affected by attribution models, brand demand, offline interactions, and tracking restrictions.

Best practice is therefore triangulation: compare tools, inspect SERPs, consult customers and experts, use first-party performance data, and test assumptions through content and campaign results. Prioritise relevance and usefulness before scale. Establish a repeatable process, but allow professional judgement where evidence conflicts.

A concise operating checklist is:

1. Start with business goals and audience problems.
2. Generate seeds from diverse evidence sources.
3. Expand, clean, and classify candidates.
4. Inspect SERPs rather than trusting tool labels alone.
5. Evaluate relevance, demand, competition, value, and seasonality.
6. Cluster by intent and page requirement.
7. Map clusters to purposeful pages.
8. Implement, measure qualified outcomes, and revise.

Chapter Summary

Keyword and search demand research connects audience language with SEO and search-marketing decisions. Seed terms provide starting points, while expansion reveals variants, questions, entities, and long-tail needs. Search intent determines the experience a page should provide, and SERP analysis is essential for validating that intent. Keyword tools offer useful estimates of volume, trends, CPC, and difficulty, but their values require interpretation.

Effective research triangulates keyword platforms, competitor evidence, trend data, SERPs, and first-party insights. Opportunity depends on relevance, demand, competition, and commercial or strategic value. Keyword clusters group queries that can be satisfied by a coherent page; keyword maps connect those clusters to URLs, content types, objectives, and owners. Ethical practice protects privacy, avoids manipulation, and maintains accuracy. Because search behaviour and interfaces change, research must be reviewed continuously and evaluated through meaningful business and user outcomes.

Key Terms

Seed term: A broad starting expression used to generate related keyword ideas.

Keyword: A word or phrase representing a search query or search demand.

Long-tail query: A relatively specific query, often consisting of several words and expressing a narrower need.

Search intent: The likely purpose or desired outcome behind a query.

Entity: A distinct person, place, organisation, product, concept, or thing identifiable in a knowledge system.

Topic: A broad subject represented by related queries and concepts.

SERP: Search engine results page displayed for a query.

Search volume: An estimate of how often a query or query group is searched during a defined period.

Click-through rate: The percentage of impressions that result in clicks.

CPC: Cost per click in an advertising market.

Keyword difficulty: An estimated level of competition for organic visibility.

Search competitor: A site competing for visibility, whether or not it is a direct business competitor.

Seasonality: Recurring variation in demand associated with time, events, or conditions.

Keyword cluster: A group of related queries that can be served by one coherent content asset.

Keyword map: A planning document connecting clusters to pages, URLs, and objectives.

Cannibalisation: Unhelpful competition among a site’s own pages for substantially similar intent.

First-party data: Information collected directly through an organisation’s own interactions and systems.

Commercial intent: The likelihood that a searcher is investigating or completing a valuable business action.

Review Questions

Short Answer Questions

1. What is a seed term, and why is it not normally a final keyword target?
2. State four common categories of search intent.
3. Why should CPC not be treated as a direct measure of organic ranking value?
4. What is the purpose of a keyword map?
5. Name two limitations of keyword-tool data.

Long Answer Questions

1. Explain how SERP analysis can validate or challenge a keyword-tool classification of search intent.
2. Describe a complete workflow for developing keyword clusters from seed terms.
3. Discuss how first-party data can improve the quality of keyword research.
4. Explain the relationship between search volume, CTR, and potential organic traffic.
5. Discuss ethical and privacy considerations when using customer data and sensitive queries.

Analytical / Application Questions

1. A university ranks for “online marketing course” but receives few enrolments. Design a research process to determine whether the problem is intent mismatch, weak SERP alignment, poor commercial value, or an ineffective landing page.
2. A retailer has separate pages for “women’s waterproof hiking boots,” “waterproof hiking boots for women,” and “women’s hiking boots for rain.” Explain what evidence you would examine before consolidating or retaining the pages.
3. Using the prioritisation framework P = w_R R + w_D D + w_V V − w_C C, propose reasonable scoring criteria and weights for a local service business, and explain why the resulting ranking should be treated as a decision aid rather than a prediction.

Multiple Choice Questions with Answers

Question 1

Which activity is the best example of SERP analysis?

A. Counting the number of words in a company’s annual report

B. Examining the content formats and features appearing for a target query

C. Changing a page’s font size without reviewing search results

D. Purchasing a domain name that contains a keyword

Correct Answer: B. Examining the content formats and features appearing for a target query

Question 2

What is the primary purpose of a seed term?

A. To serve as a broad starting point for generating related queries

B. To guarantee a first-place ranking

C. To replace the need for competitor research

D. To calculate an exact conversion rate

Correct Answer: A. To serve as a broad starting point for generating related queries

Question 3

Which query most clearly expresses transactional intent?

A. How do solar panels work

B. History of solar power

C. Buy solar panels in Manchester

D. Are solar panels environmentally friendly

Correct Answer: C. Buy solar panels in Manchester

Question 4

Which statement about search volume is most accurate?

A. It guarantees the number of clicks a page will receive

B. It is always an exact count of unique searchers

C. It is an estimate that must be interpreted with other evidence

D. It measures conversion value directly

Correct Answer: C. It is an estimate that must be interpreted with other evidence

Question 5

What does CTR measure?

A. Conversions divided by revenue

B. Clicks divided by impressions, expressed as a percentage

C. Impressions divided by clicks, expressed as a percentage

D. Search volume divided by CPC

Correct Answer: B. Clicks divided by impressions, expressed as a percentage

Question 6

Which factor is most important when deciding whether two queries belong in one cluster?

A. Whether they have exactly the same number of words

B. Whether they can be satisfied by a coherent page serving a similar intent

C. Whether they begin with the same letter

D. Whether they come from different keyword tools

Correct Answer: B. Whether they can be satisfied by a coherent page serving a similar intent

Question 7

Why can a high-volume keyword have low business value?

A. High-volume terms never produce clicks

B. The term may be poorly relevant or have weak commercial intent

C. Search engines ignore all high-volume queries

D. Volume and relevance are identical measures

Correct Answer: B. The term may be poorly relevant or have weak commercial intent

Question 8

Which is an example of first-party insight?

A. A competitor’s estimated domain score

B. A generic industry keyword list

C. Queries recorded in a company’s internal site search

D. A random phrase generated without evidence

Correct Answer: C. Queries recorded in a company’s internal site search

Question 9

What is keyword cannibalisation?

A. A paid advertisement appearing above an organic result

B. Unhelpful competition among a site’s own pages for similar intent

C. A search engine removing all duplicate words

D. A customer using a long-tail query

Correct Answer: B. Unhelpful competition among a site’s own pages for similar intent

Question 10

Which practice is most consistent with ethical keyword research?

A. Creating misleading pages for every nearby town

B. Using sensitive customer queries to profile individuals without safeguards

C. Copying a competitor’s content and presenting it as original

D. Using accurate claims, privacy safeguards, and purposeful pages

Correct Answer: D. Using accurate claims, privacy safeguards, and purposeful pages