Unsupervised Discovery and Topic Modeling

Introduction

Text and social media data contain large-scale, rapidly changing evidence about customers, citizens, employees, brands and markets. Much of this evidence is unlabelled: organisations may possess millions of posts, reviews or service transcripts without a predefined category for each document. Unsupervised learning addresses this problem by discovering structure from the data itself. In text analytics, the principal approaches include clustering, latent semantic analysis (LSA), probabilistic topic models and embedding-based topic discovery.

These methods do not simply “find the truth” in a corpus. They produce representations that depend on preprocessing, vocabulary, model assumptions, parameter choices and interpretation. A topic may represent a genuine business theme, a writing style, a demographic group or an artefact of data collection. Consequently, effective topic discovery requires both quantitative evaluation and substantive managerial judgement.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how clustering and topic modeling differ.
2. Construct and interpret a document–term matrix and TF–IDF representation.
3. Apply and evaluate LSA, probabilistic topic models and embedding-based methods.
4. Assess topic coherence, diversity, stability and business usefulness.
5. Diagnose common failure modes, ethical risks and governance requirements.
6. Translate discovered topics into managerial actions without overstating causal meaning.

1. The Discovery Problem in Text Analytics

A corpus is a collection of documents, such as product reviews, posts or call transcripts. Let there be D documents and V vocabulary terms. A document–term matrix X records term counts:

Here x_dv is the count of term v in document d. The matrix is typically sparse because each document contains only a small fraction of the full vocabulary. Tokenisation, lowercasing, lemmatisation, phrase detection, removal of unsuitable stop words and treatment of hashtags or emojis can substantially affect the resulting matrix.

A discovery system should be designed around the analytical unit. A tweet may be too short for reliable document-level topics, whereas aggregating posts by user, week, campaign or product may provide sufficient context. Aggregation improves statistical signal but can conceal variation and create privacy risks.

Count and TF–IDF Representations

Raw counts favour long documents and frequent terms. A common alternative is term frequency–inverse document frequency:

tf_dv is a chosen term-frequency measure, df_v is the number of documents containing term v, and D is the number of documents. The smoothed expression avoids division by zero and assigns lower weight to terms appearing in many documents. TF–IDF is useful for similarity and clustering, but it is not a generative model and does not itself identify topics.

Cosine similarity is frequently used for document vectors:

It measures the angle between vectors and is insensitive to overall document magnitude after normalisation. It is appropriate when direction in term space is more meaningful than absolute length; it is undefined for a zero vector and should not be interpreted as semantic understanding.

2. Clustering Text Documents

Clustering partitions or groups documents so that documents within a group are more similar than documents in different groups. In k-means, each document is assigned to one of k clusters by minimising within-cluster squared Euclidean distance:

μ_r is the centroid of cluster r. The method assumes a numerical vector space, requires k in advance and is sensitive to scaling, initialisation and outliers. Because text vectors are sparse and high-dimensional, cosine distance or spherical k-means is often more suitable than ordinary Euclidean distance.

Hierarchical clustering creates a tree of nested groups. Agglomerative clustering begins with each document as a singleton and repeatedly merges the closest groups according to a linkage rule, such as average or complete linkage. It does not require a single final number of clusters and supports dendrogram-based exploration, although the result can be computationally expensive and sensitive to the linkage choice.

Density-based methods, including DBSCAN and HDBSCAN, identify dense regions and can label isolated documents as noise. They are useful when clusters have unequal sizes or when anomalous posts are substantively meaningful. However, sparse text spaces and changing density can make parameter selection difficult.

Clustering and topic modeling are related but not equivalent. Clustering normally assigns a document to one or more groups based on similarity. Topic models represent documents as mixtures of latent themes and can therefore describe a review discussing both delivery and product quality. A cluster is a grouping of documents; a topic is an interpretable pattern of word usage under a particular model.

3. Latent Semantic Analysis

Latent Semantic Analysis uses singular value decomposition (SVD) to reduce a document–term matrix to a lower-dimensional representation:

U_k, Σ_k and V_k retain the first k singular dimensions. The original matrix may be TF–IDF weighted rather than a count matrix. Documents represented in the reduced space can become similar even when they do not share exact words, because words occurring in related distributional contexts contribute to the same latent dimensions.

The singular values indicate the amount of matrix variation captured by each dimension. Selecting k is not determined by a universal rule: analysts may inspect explained reconstruction error, downstream performance, topic interpretability and stability. LSA dimensions are linear combinations with positive and negative weights. Their interpretation is therefore less direct than that of probabilistic topics. A dimension associated with both “battery” and “charging” may also contain opposing or unrelated terms because signs and weights are algebraic rather than probabilistic.

LSA is valuable for information retrieval, dimensionality reduction and exploratory analysis. Its limitations include dependence on the chosen weighting scheme, inability to express a clear document-level probability distribution over topics, and sensitivity to vocabulary and corpus composition.

4. Probabilistic Topic Models

4.1 Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) assumes that each document is a mixture of topics and each topic is a probability distribution over words. The generative process is:

1. For each topic k, draw a word distribution φ_k from a Dirichlet distribution with parameter β.
2. For each document d, draw a topic mixture θ_d from a Dirichlet distribution with parameter α.
3. For each token, draw a topic from θ_d and then draw a word from that topic’s φ_k.

The joint model can be written as:

w_dn is the observed word token, z_dn its latent topic, θ_d the topic proportions for document d, and φ_k the word probabilities for topic k. α and β control sparsity. Lower values generally encourage documents to use fewer topics and topics to use fewer words, although exact effects depend on parameterisation and inference.

LDA is a probabilistic model, not a claim that real authors consciously generate text through these steps. The process is a useful approximation that provides interpretable parameters. In practice, analysts estimate posterior topic assignments using variational inference or collapsed Gibbs sampling. Results can vary across random seeds and implementations.

4.2 Interpreting LDA

A topic is commonly inspected through its highest-probability words, representative documents and salient words. Ranking only by probability can produce generic terms. A relevance score may combine probability and lift, where lift compares a word’s probability within a topic with its overall corpus probability. Human review remains essential: a topic labelled “delivery problems” should be supported by documents containing meaningful delivery-related context, not merely by the words “time”, “order” and “customer”.

The number of topics K is a modeling choice. Too few topics merge distinct managerial issues; too many split coherent themes or produce idiosyncratic topics. Analysts should compare several values of K, evaluate multiple criteria and explain the final selection.

4.3 Extensions

Structural Topic Models allow topic prevalence or word use to vary with document metadata, such as region, time or customer segment. Dynamic topic models examine topic evolution over time. Correlated topic models relax LDA’s assumptions about topic independence. Non-negative matrix factorisation (NMF), although not generative in the LDA sense, often produces sparse and practically interpretable topics from TF–IDF matrices.

5. Embedding-Based Topic Discovery

Word embeddings represent words as dense vectors learned from contextual co-occurrence. Sentence-transformer models extend this idea by producing document or sentence embeddings. A typical pipeline is:

1. Generate embeddings for documents or sentences.
2. Reduce dimensionality, often with a method such as UMAP for exploration.
3. Cluster the vectors using a method such as HDBSCAN.
4. Represent each cluster with class-based TF–IDF or another term-ranking method.
5. Inspect documents and merge, split or label topics substantively.

Embeddings can capture semantic similarity when documents use different words, such as “refund delayed” and “money has not come back”. They are especially useful for short texts, multilingual variation and paraphrase-rich social media. However, embedding similarity reflects the training data and model objective. It may encode demographic or cultural bias, flatten minority meanings and confuse topical similarity with sentiment, authorship or conversational style.

Dimensionality reduction is not neutral. UMAP and similar methods preserve selected local or global relationships for visualisation, but a two-dimensional plot should not be treated as a faithful map of the original semantic space. Clustering should preferably be evaluated in the original embedding space or a carefully justified reduced space.

6. Evaluating Topic Quality

No single metric establishes that a topic model is good. Evaluation should combine statistical, semantic, operational and ethical criteria.

6.1 Perplexity

For a held-out corpus, perplexity is commonly defined as:

N is the number of held-out tokens and p(w_i) is the model probability assigned to token i. Lower perplexity indicates better predictive fit under the model and preprocessing choices. It does not guarantee human interpretability or managerial usefulness; models with lower perplexity can produce less coherent topics.

6.2 Coherence and Diversity

Topic coherence measures whether top words tend to co-occur or have related semantic associations. Measures such as UMass and normalized pointwise mutual information use different reference corpora and assumptions, so their values are not automatically comparable across studies. Topic diversity can be expressed as:

K is the number of topics and M the number of displayed words per topic. High diversity indicates less repetition across topics, but excessive diversity may reflect fragmented or noisy topics.

6.3 Stability

A stable model gives similar topic structure under reasonable perturbations, such as bootstrap samples, time windows, preprocessing alternatives or random seeds. Topic labels are not directly comparable because topic numbers are arbitrary. Topics must be aligned using similarity between their word distributions or document prevalence vectors. A model that changes dramatically under small perturbations should not support high-stakes decisions without further investigation.

6.4 Human and Managerial Evaluation

Human judges can rate topic coherence, exclusivity, interpretability and document fit. Inter-rater agreement is useful when labels are assigned systematically. Managerial usefulness asks whether topics improve a decision: for example, routing complaints, prioritising product defects or identifying emerging risks. A topic can have moderate statistical coherence but high value if it reveals a previously invisible issue. Conversely, a mathematically elegant topic may have little operational relevance.

A practical evaluation scorecard should record:

7. Worked Example: Choosing Between Models

Suppose an organisation analyses 10,000 product reviews. It compares three candidate systems. Model A is LDA with 10 topics, Model B is LDA with 20 topics, and Model C is embedding clustering with 14 topics. After held-out evaluation and human review, the results are:

The values are illustrative and are not comparable across different coherence definitions without a common evaluation protocol. Model B has the highest coherence and diversity but lower stability and actionability. Model C best supports the operational objective, provided its clusters are adequately covered and ethically reviewed. The defensible conclusion is not that C is universally superior, but that it is preferable for this stated use case after validation.

8. Illustrative Case: Social Listening for a Retailer

Background

A national retailer receives online reviews, public posts and customer-service transcripts. Senior managers know that satisfaction scores are falling but lack a detailed, current view of the causes.

Problem

The existing dashboard uses fixed categories such as “price”, “quality” and “delivery”. These categories are too broad, do not capture emerging issues and cannot distinguish a late parcel from a failed refund.

Analysis and approach

The analytics team removes duplicated posts, preserves negations, detects phrases such as “still waiting” and separates bot-like content. It analyses reviews using LDA and short social posts using sentence embeddings plus density-based clustering. Analysts examine topic prevalence by week, product line and channel. They validate candidate topics through representative documents and workshops with customer operations, logistics and compliance teams.

The models identify six useful themes: delayed delivery, refund not received, confusing mobile checkout, damaged packaging, misleading promotion and positive staff assistance. A seemingly coherent “premium customer” topic is rejected because it mainly reflects a loyalty programme’s campaign language rather than a customer problem.

Outcome

The retailer routes refund-related posts to a payment operations team, investigates a packaging supplier and redesigns the mobile checkout page. Topic prevalence is monitored as an early-warning indicator, but managers are cautioned that increased discussion may reflect a campaign or change in platform usage rather than increased incidence.

Learning points

First, discovery should supplement—not replace—existing categories. Second, topic labels require domain review. Third, trend interpretation needs denominators, such as posts per 10,000 orders, rather than raw volume alone. Finally, operational deployment requires monitoring for drift, sampling bias and changes in platform language.

9. Managerial Interpretation and Governance

Topics are descriptive patterns, not causal explanations. If “delivery complaints” rise after a policy change, the association warrants investigation but does not prove that the policy caused the increase. Topic prevalence can be calculated as the mean document-level topic proportion:

For a time period or segment, the same calculation should use a clearly defined subset. Comparisons should account for changing document volume, channel composition and exposure. Confidence intervals or bootstrap uncertainty are appropriate when prevalence is used for formal comparison.

Governance should address data minimisation, lawful processing, retention, access controls and re-identification risk. Public availability does not automatically eliminate privacy expectations. Sensitive attributes should not be inferred or used for adverse decisions without a clear legal and ethical basis. Analysts should document corpus construction, preprocessing, model version, hyperparameters, random seeds, evaluation results and human overrides.

Topic systems can reproduce linguistic inequity. A model trained mainly on formal English may misrepresent dialects, code-switching, emojis or minority languages. Translation can erase culturally important distinctions. Human review panels should include relevant linguistic and cultural expertise, and high-impact uses should include appeal and audit mechanisms.

10. Best Practices and Contemporary Developments

A robust workflow begins with a decision question rather than a preferred algorithm. Define the unit of analysis, sampling frame and success criteria. Establish a transparent preprocessing pipeline and retain raw data securely for audit where permitted. Use multiple candidate models, compare them on held-out or temporally separated data, and inspect representative documents rather than relying on top words alone.

For deployment, version models and embeddings, monitor topic drift, review changes in topic prevalence and maintain a human escalation path. Retrieval-augmented and generative AI systems can help summarise documents associated with a topic, but generated labels and summaries must be grounded in source evidence and checked for hallucination. Large language models may also be used to classify or label discovered topics, but their prompts, model versions, costs, privacy implications and reproducibility must be governed.

The strongest practice is triangulation: combine lexical models, embedding models, metadata, human judgement and business outcomes. Disagreement between methods is not merely an inconvenience; it may reveal ambiguity, polysemy, sampling problems or multiple valid structures in the corpus.

Chapter Summary

Unsupervised discovery reveals structure in unlabelled text but does not automatically produce objective or causal knowledge. Clustering groups documents according to a distance measure, whereas topic models describe latent mixtures of word distributions. LSA uses low-rank matrix factorisation; LDA uses probabilistic document–topic and topic–word distributions; embedding-based approaches capture contextual similarity and commonly combine vector representations with clustering.

Topic quality must be assessed through coherence, diversity, stability, coverage, human interpretability, actionability and fairness. Perplexity evaluates predictive fit but is not a sufficient quality criterion. The choice of topic number, preprocessing, embedding model and clustering algorithm can materially change conclusions. Managerial use requires careful denominator selection, domain validation, documentation, privacy protection and monitoring for drift. The most defensible systems combine quantitative diagnostics with substantive interpretation and governance.

Key Terms

Corpus: A collection of documents analysed together.

Document–term matrix: A matrix recording term values for documents.

TF–IDF: A weighting scheme that increases the importance of document-specific terms.

Cosine similarity: Similarity based on the angle between two vectors.

Clustering: Grouping observations according to similarity.

Centroid: The mean vector representing a cluster.

LSA: A low-rank semantic representation based on singular value decomposition.

SVD: Matrix factorisation into singular vectors and singular values.

Topic model: A model representing documents through latent themes.

LDA: A probabilistic model in which documents mix topics and topics mix words.

Topic prevalence: The proportion of a corpus associated with a topic.

Topic coherence: The extent to which topic words form a meaningful semantic group.

Topic diversity: The extent to which top words differ across topics.

Stability: Consistency of discovered topics under perturbations.

Embedding: A dense numerical representation of linguistic content.

HDBSCAN: A density-based clustering method that can identify noise.

Perplexity: A held-out predictive-fit measure for probabilistic language models.

Topic drift: Change in topic meaning or prevalence over time.

Semantic bias: Systematic representational distortion associated with language, data or model assumptions.

Review Questions

5 Short Answer Questions

1. Why can TF–IDF be useful for text clustering?
2. State the main generative assumptions of LDA.
3. How does LSA differ from LDA?
4. What does topic stability measure?
5. Why should raw topic volume be interpreted cautiously over time?

5 Long Answer Questions

1. Explain the complete workflow for embedding-based topic discovery and discuss the role of dimensionality reduction.
2. Compare k-means, hierarchical clustering and density-based clustering for social media text.
3. Explain why perplexity and coherence may lead to different model selections.
4. Discuss how preprocessing choices can alter discovered topics and managerial conclusions.
5. Evaluate the privacy, fairness and governance issues involved in social listening.

3 Analytical / Application Questions

1. An organisation obtains high coherence from a 30-topic model but low stability across random seeds. Recommend an evaluation and selection strategy.
2. A topic’s raw post count doubles while the number of active users and total posts also doubles. How should an analyst assess whether the underlying issue has increased?
3. Design a validated topic-discovery system for customer complaints, specifying data preparation, candidate models, evaluation metrics, human review and deployment controls.

Multiple Choice Questions with Answers

Question 1

Which representation gives lower weight to terms appearing in many documents?

A. Raw document length

B. TF–IDF

C. One-hot document IDs

D. Cluster labels

Correct Answer: B. TF–IDF

Question 2

What is a central assumption of LDA?

A. Each document belongs to exactly one topic

B. Topics are distributions over documents only

C. Each document is a mixture of latent topics

D. All topics have identical word probabilities

Correct Answer: C. Each document is a mixture of latent topics

Question 3

What does SVD provide in LSA?

A. A low-rank factorisation of the document–term matrix

B. A causal estimate of topic effects

C. A legal basis for data processing

D. A guaranteed optimal number of topics

Correct Answer: A. A low-rank factorisation of the document–term matrix

Question 4

Which measure evaluates predictive fit on held-out tokens in a probabilistic topic model?

A. Perplexity

B. Topic diversity

C. Cosine distance

D. Cluster count

Correct Answer: A. Perplexity

Question 5

Why can embedding-based topic discovery help with short social media posts?

A. It eliminates all bias

B. It captures semantic similarity beyond exact word overlap

C. It guarantees causal interpretation

D. It requires no validation

Correct Answer: B. It captures semantic similarity beyond exact word overlap

Question 6

What does high topic diversity generally indicate?

A. Top words are less repetitive across topics

B. All documents have equal length

C. The model has perfect predictive fit

D. Every topic is causally independent

Correct Answer: A. Top words are less repetitive across topics

Question 7

Which result most directly indicates model instability?

A. Similar topics across random seeds

B. Stable prevalence across samples

C. Large changes in topics after small perturbations

D. High-quality representative documents

Correct Answer: C. Large changes in topics after small perturbations

Question 8

Why should topic labels not be based only on the highest-probability words?

A. Top words may be generic without showing contextual meaning

B. Probability cannot be computed for words

C. Topics contain no words

D. Labels are always produced by clustering

Correct Answer: A. Top words may be generic without showing contextual meaning

Question 9

In the prevalence formula, what does θ_dk represent?

A. The number of vocabulary terms

B. Document d’s proportion associated with topic k

C. The cosine distance between documents

D. The number of clusters selected

Correct Answer: B. Document d’s proportion associated with topic k

Question 10

Which is an appropriate governance practice for deployed topic models?

A. Discarding model versions after deployment

B. Using public posts without considering privacy

C. Monitoring drift and documenting preprocessing and parameters

D. Treating topics as proven causes

Correct Answer: C. Monitoring drift and documenting preprocessing and parameters