Distributed Representations and Semantic Analytics

Introduction

Natural language is expressed as a sequence of discrete symbols, but machine-learning systems require numerical representations. A vocabulary item such as “bank”, a sentence such as “The bank approved the loan”, and a document such as a news article must therefore be mapped to vectors that preserve useful linguistic and semantic information. This process is called representation learning.

Earlier natural-language systems represented text using one-hot vectors, word counts, or manually designed linguistic features. These methods are interpretable but usually sparse and unable to express graded similarity. Distributed representations instead encode meaning across many dimensions. A word is represented not by one isolated feature, but by a pattern of values distributed over a vector. Similar contexts can therefore produce similar representations.

This chapter examines word, sentence, and document embeddings; similarity and distance measures; clustering and topic structure; semantic search; evaluation; and representation bias. The emphasis is both conceptual and practical: a good embedding system must not only achieve high benchmark scores but also support reliable decisions in a specific application.

Learning Objectives

After completing this chapter, a student should be able to:

1. Explain the motivation and assumptions of distributed representations.
2. Distinguish count-based, prediction-based, contextual, sentence, and document embeddings.
3. Calculate and interpret vector similarity measures.
4. Select suitable methods for clustering, topic discovery, and semantic search.
5. Evaluate embeddings using intrinsic and downstream measures.
6. Identify limitations, bias, privacy risks, and governance requirements.
7. Design a defensible embedding workflow for an applied NLP problem.

1. From Symbolic Tokens to Distributed Representations

A one-hot representation assigns each vocabulary item a vector of length |V|, where |V| is the vocabulary size. Exactly one coordinate is 1 and all others are 0. For a vocabulary containing “cat”, “dog”, and “fish”, the vector for “cat” might be [1, 0, 0]. This representation identifies tokens but does not express that “cat” and “dog” are semantically more related than “cat” and “finance”.

A distributed embedding maps an item to a dense vector in ℝᵈ, where d is usually much smaller than |V|. The coordinates generally do not have individually interpretable meanings. Meaning is represented by the complete pattern of values. The central linguistic assumption is the distributional hypothesis: words occurring in similar contexts tend to have similar meanings.

This assumption is powerful but incomplete. It captures usage patterns rather than truth, reference, intention, or physical experience. Consequently, embeddings may encode stereotypes, historical inequalities, or associations caused by the data itself.

Important terminology

• Token: an individual unit produced by tokenisation.
• Vocabulary: the set of tokens recognised by a model.
• Embedding: a learned numerical vector representing an item.
• Dimension: one coordinate of an embedding vector.
• Context window: neighbouring tokens used to define context.
• Static embedding: one vector per vocabulary item.
• Contextual embedding: a representation that changes with surrounding text.
• Encoder: a model that maps input text to a representation.
• Semantic space: the geometric space in which representations are compared.

2. Major Families of Embeddings

2.1 Count-based representations

Count-based methods construct a term-context matrix X. Entry xᵢⱼ records how often word i occurs with context j, possibly using a distance-weighted count. Raw counts are dominated by frequent words, so transformations such as term frequency–inverse document frequency (TF–IDF), logarithmic scaling, or positive pointwise mutual information (PPMI) are commonly used.

For a corpus of N documents, the inverse document frequency of term t is commonly defined as:

where df(t) is the number of documents containing t. A basic TF–IDF weight is:

Here TF(t,d) is the frequency of term t in document d. Smoothing may be required when a term occurs in every or no document. TF–IDF is useful for information retrieval, but it usually represents documents in a very high-dimensional sparse space and does not reliably capture synonymy.

Matrix factorisation methods, such as singular value decomposition (SVD), reduce a count matrix to a dense space. If X is approximated by UΣVᵀ, retaining the first k singular components gives a lower-dimensional representation. The choice of k controls the trade-off between compression and information retention.

2.2 Prediction-based static embeddings

Word2Vec learns vectors by predicting words from contexts or contexts from words. Its two principal training objectives are continuous bag of words (CBOW) and skip-gram. In skip-gram with negative sampling, a centre word is trained to distinguish observed context words from randomly sampled negative words. The objective for a positive pair (w,c) and negative samples n₁,…,nₖ can be written as:

where v_w is the centre-word vector, v′_c and v′_n are output vectors, σ is the logistic sigmoid, and k is the number of negative samples. Training maximises this objective across corpus windows. The negative-sampling distribution and context-window design affect the resulting geometry.

GloVe learns vectors from global word co-occurrence statistics. FastText represents a word through character n-grams, allowing it to handle rare, morphologically complex, or previously unseen words more effectively than ordinary word-level models.

Static embeddings are efficient and useful for many classification and similarity tasks. However, “bank” has one vector whether the sentence concerns finance or a river. They also inherit frequency and social biases from their training corpora.

2.3 Contextual embeddings

Transformer encoders generate a representation conditioned on the entire input sequence. Thus the representation of “bank” differs in “deposit money at the bank” and “sit beside the river bank”. Self-attention permits each token to incorporate information from other tokens, while multiple layers create increasingly abstract representations.

Contextual embeddings support named-entity recognition, question answering, classification, and semantic retrieval. Their limitations include computational cost, sensitivity to prompts or surrounding text, and possible instability across model versions. A contextual vector is not automatically a complete sentence meaning; pooling and task-specific training remain important.

2.4 Sentence and document embeddings

A sentence embedding is a fixed-length vector intended to represent the meaning of a sentence or passage. Simple approaches average word vectors. More advanced approaches use transformer encoders trained with contrastive or natural-language-inference objectives. Sentence-transformer models, for example, train semantically related texts to be close and unrelated texts to be separated.

Document embeddings may be obtained by averaging sentence vectors, using a dedicated document encoder, or pooling token representations. Long documents create a length and coherence problem: one vector may obscure sections, contradictions, or multiple topics. Chunking followed by hierarchical aggregation is often more reliable.

Pooling is a design decision. Mean pooling is stable and inexpensive; maximum pooling highlights strong features; attention pooling learns which tokens matter. The correct choice depends on the training objective and the intended application.

3. Geometry: Similarity and Distance

The most common measure for comparing embeddings is cosine similarity:

Vectors x and y must be non-zero. The result lies between −1 and 1, although practical embedding distributions often occupy a narrower range. Cosine similarity compares orientation and ignores vector magnitude. If vectors are L2-normalised, cosine similarity equals their dot product.

Euclidean distance is:

It is appropriate when magnitude is meaningful and the embedding space has been trained or calibrated for Euclidean geometry. Manhattan distance sums absolute coordinate differences. No distance measure is universally best; it should match model training and validation evidence.

Worked example

Let x = [1, 2] and y = [2, 1]. Their dot product is 1×2 + 2×1 = 4. Each vector has norm √5. Therefore:

Their Euclidean distance is √[(1−2)² + (2−1)²] = √2 ≈ 1.414. The vectors are fairly similar in direction, but they are not identical.

Similarity scores must not be interpreted as probabilities. A score of 0.8 does not mean an 80% probability that two texts have the same meaning. Thresholds must be selected using labelled validation data and the costs of false positives and false negatives.

4. Clustering and Topic Structure

Clustering groups representations according to a chosen distance measure. k-means partitions n observations into k clusters by minimising within-cluster squared Euclidean distance:

Here Cⱼ is cluster j and μⱼ is its centroid. The method assumes that clusters are reasonably compact and that a meaningful value of k is supplied. It is sensitive to scaling, initialisation, outliers, and non-spherical structures.

Hierarchical clustering creates a tree of nested groups and is useful when analysts want to inspect several levels of granularity. Density-based methods such as DBSCAN can identify irregular clusters and outliers, but their parameters become difficult in high-dimensional spaces.

Topic modelling differs from ordinary clustering. Latent Dirichlet allocation (LDA) represents documents as mixtures of topics and topics as distributions over words. A simplified generative interpretation is:

1. For each document, draw a topic-mixture vector.
2. For each token, draw a topic from that mixture.
3. Draw the token from the selected topic’s word distribution.

LDA provides interpretable word distributions but relies on a bag-of-words assumption and may struggle with short texts or semantic synonyms. Neural topic models and embedding-based topic methods can use semantic information, but their topics may be less stable or less interpretable.

Topic quality should be assessed using coherence, stability across runs or samples, human interpretability, and usefulness for the decision at hand. A mathematically coherent topic is not necessarily operationally meaningful.

5. Semantic Search and Retrieval

Keyword search requires exact or near-exact lexical overlap. Semantic search instead embeds a query and candidate documents in a common space, then retrieves candidates with high similarity. A practical architecture is:

1. Clean and segment documents into passages.
2. Compute and store passage embeddings.
3. Embed the incoming query.
4. Retrieve nearest passages using an approximate nearest-neighbour index.
5. Optionally rerank candidates with a cross-encoder.
6. Present evidence and citations to the user.

Approximate nearest-neighbour methods trade a small amount of recall for substantial speed and lower memory cost. Common families include inverted-file indexes, product quantisation, and graph-based indexes. Index configuration must be tested rather than assumed to be optimal.

Retrieval evaluation should include recall@k, precision@k, mean reciprocal rank (MRR), and normalised discounted cumulative gain (nDCG). For a query whose first relevant result occurs at rank r, reciprocal rank is 1/r. MRR averages reciprocal rank over queries. nDCG gives greater credit to highly ranked relevant results and can represent graded relevance.

A retrieval system can return a semantically similar but factually incorrect passage. Therefore semantic retrieval should be separated from answer generation, and generated answers should be grounded in retrieved evidence. Access controls, document freshness, deletion requirements, and confidential-data handling must be implemented at indexing time, not added only at the user interface.

6. Evaluation and Diagnostics

Intrinsic evaluation

Intrinsic evaluation tests properties of the representation itself. Word similarity datasets compare model rankings with human judgements, often using Spearman’s rank correlation. Analogy tests examine relations such as capital-city or grammatical transformations, but analogy performance can be affected by memorisation and dataset artifacts. Sentence-embedding evaluations may use semantic textual similarity, entailment, or paraphrase classification.

Extrinsic evaluation

Extrinsic evaluation measures performance in a real task: classification F1, retrieval recall, clustering utility, recommendation quality, or human decision accuracy. It is usually more relevant than a single generic benchmark.

Statistical and operational diagnostics

A professional evaluation should use held-out data, confidence intervals or repeated samples where feasible, subgroup analysis, error inspection, and temporal validation. Data leakage occurs when information from evaluation documents or future events influences training. Near-duplicate documents can create inflated scores. Embedding spaces may also be anisotropic, meaning vectors are concentrated in a narrow region; centring, whitening, or model-specific normalisation may help, but such transformations require validation.

Evaluation should examine:

• semantic relevance and factual usefulness;
• robustness to spelling, dialect, domain shift, and paraphrase;
• performance across languages and demographic groups;
• latency, memory, throughput, and energy consumption;
• behaviour when documents are missing, stale, ambiguous, or adversarial.

7. Representation Bias, Ethics, and Governance

Embeddings learn from data. If a corpus associates occupations with particular genders, or treats one dialect as less authoritative, the representation may reproduce those associations. Bias can be allocational, representational, linguistic, cultural, or intersectional. Removing an explicitly sensitive word rarely removes the information encoded in correlated features.

Risk assessment should begin with the intended use, affected groups, data provenance, and possible harms. Useful practices include dataset documentation, model cards, subgroup evaluation, adversarial testing, human review, access controls, and an appeal process. Debiasing methods can reduce particular measured associations, but they may remove legitimate information, fail under distribution shift, or create a false impression of fairness.

Privacy is another concern. Embedding vectors may reveal sensitive properties or enable membership-inference and reconstruction attacks. Organisations should minimise collected data, restrict retention, encrypt stores, document model access, and follow applicable privacy and sectoral regulations. Personally identifiable information should not be embedded merely because it is technically possible.

Fairness cannot be reduced to one universal metric. The appropriate criterion depends on the application, legal setting, and harm model. High aggregate accuracy may conceal severe subgroup failures. Human oversight must be meaningful: reviewers need authority, time, evidence, and a way to correct system outputs.

8. Applied Case: University Policy Semantic Search

Background

This illustrative case concerns a university with 18,000 policy documents, regulations, forms, and committee minutes. Students and staff report that keyword search misses relevant passages because the same concept is described using different terminology.

Problem

The university needs a search system that retrieves authoritative passages for queries such as “How do I appeal an academic misconduct decision?” It must respect document permissions, distinguish current from superseded policies, and provide evidence rather than unsupported answers.

Analysis

A sample of 1,200 historical queries is labelled by librarians for relevance at passage level. Analysis shows that exact keyword search performs poorly for paraphrases, while long documents contain several unrelated topics. Sensitive records must not be included in a general index.

Approach and solution

Documents are permission-filtered before processing. Current policy versions are identified using metadata. Each document is split into coherent passages with titles, section headings, effective dates, and source identifiers retained. A sentence-transformer encoder creates normalised passage vectors. An approximate nearest-neighbour index retrieves 30 candidates, and a cross-encoder reranks the top candidates. The interface displays the top passages, policy dates, and links to original documents.

The team evaluates recall@10, MRR, subgroup query performance, latency, and librarian judgements. Queries are tested using synonyms, spelling errors, abbreviations, and deliberately ambiguous wording. A scheduled re-indexing process handles policy changes and deletion requests.

Outcome

The pilot improves paraphrase retrieval and reduces time spent locating policies. However, some queries return semantically related but obsolete passages. Adding effective-date filtering and mandatory source display resolves much of this problem. The university therefore keeps keyword filters and metadata search alongside embeddings rather than replacing them entirely.

Learning points

The case demonstrates that embeddings are one component of a retrieval system. Data governance, chunking, metadata, evaluation, reranking, and human-centred presentation are equally important. A high similarity score alone is insufficient evidence of authority or correctness.

9. Best-Practice Design Workflow

1. Define the decision or user task before selecting a model.
2. Establish a representative, permission-aware corpus.
3. Record tokenisation, chunking, model version, dimensions, and distance metric.
4. Create labelled validation and test sets that reflect real queries and affected groups.
5. Compare a strong lexical baseline with embedding alternatives.
6. Tune thresholds and retrieval parameters on validation data only.
7. Inspect errors, especially false matches, omissions, and subgroup failures.
8. Monitor drift, freshness, latency, cost, and access violations after deployment.
9. Version embeddings and indexes so results can be reproduced.
10. Provide provenance, human escalation, and a rollback mechanism.

Contemporary Developments

Current systems increasingly combine sparse lexical retrieval with dense retrieval, a method often called hybrid search. Multilingual and cross-lingual encoders support retrieval across languages, but quality may vary by language and domain. Instruction-tuned embedding models adapt representations to task descriptions, while multimodal models align text with images, audio, or video.

Large language models can produce useful embeddings, but their representations may be sensitive to prompting, pooling strategy, and model updates. Retrieval-augmented generation combines semantic search with generation, yet it introduces additional risks: retrieval errors can be amplified into fluent misinformation, and citations may not guarantee faithful use of evidence. Consequently, evaluation must cover both retrieval and generated responses.

Chapter Summary

Distributed representations encode words, sentences, and documents as vectors whose relationships can be analysed geometrically. Count-based methods preserve co-occurrence information, while prediction-based and contextual models learn representations from language tasks. Static vectors are efficient but cannot naturally resolve word-sense variation; contextual encoders adapt to surrounding text. Sentence and document embeddings require careful pooling, chunking, and task alignment.

Cosine similarity, Euclidean distance, clustering, topic modelling, and nearest-neighbour retrieval provide complementary analytical tools. Semantic search is most reliable when dense retrieval is combined with lexical and metadata constraints, reranking, evidence display, and continuous evaluation. Intrinsic benchmark scores do not replace downstream testing. Bias, privacy, access control, temporal drift, and explainability are central governance concerns. The most defensible embedding workflow begins with a clearly defined task and ends with monitoring, human oversight, and reproducible documentation.

Key Terms

Distributed representation: A numerical representation in which information is spread across multiple vector dimensions.

Embedding: A learned dense vector representing a token, sentence, document, or other object.

Distributional hypothesis: The principle that similar contexts indicate related meaning.

Static embedding: A representation assigning one vector to each vocabulary item.

Contextual embedding: A representation whose value depends on surrounding text.

TF–IDF: A weighting method combining term frequency with inverse document frequency.

Skip-gram: A prediction-based method that predicts context words from a centre word.

Negative sampling: Training with observed examples and sampled non-observed examples.

Cosine similarity: The cosine of the angle between two non-zero vectors.

Euclidean distance: The straight-line distance between vectors.

Pooling: Combining token-level vectors into a fixed-length sentence or document vector.

Clustering: Grouping observations according to similarity or distance.

Topic model: A model representing documents through latent themes or topics.

Semantic search: Retrieval based on meaning-related representations rather than exact terms alone.

Reranking: Reordering initially retrieved candidates using a more detailed model.

Recall@k: The proportion of relevant items retrieved within the first k results.

Embedding bias: Systematic associations or performance differences inherited by representations.

Data leakage: Unintended use of evaluation or future information during model development.

Anisotropy: Concentration of vectors in a limited region or set of directions.

Review Questions

Short Answer Questions

1. What is the distributional hypothesis?
2. Distinguish static and contextual embeddings.
3. Why is cosine similarity frequently used for embeddings?
4. What is the purpose of negative sampling?
5. Name two risks associated with embedding-based semantic search.

Long Answer Questions

1. Compare count-based, prediction-based, and contextual representation methods.
2. Explain how sentence and document embeddings can be constructed and why pooling matters.
3. Describe a complete evaluation strategy for an embedding-based retrieval system.
4. Explain how clustering differs from topic modelling.
5. Discuss the ethical and governance challenges of representation bias.

Analytical / Application Questions

1. Design an embedding-based search system for a hospital policy repository. Specify chunking, metadata, retrieval, evaluation, and privacy controls.
2. A model has high average retrieval recall but poor performance for one language group. Analyse possible causes and propose a diagnostic and remediation plan.
3. Given three candidate embedding models with different accuracy, latency, memory, and bias profiles, develop a decision framework for selecting one for a high-stakes application.

Multiple Choice Questions with Answers

Question 1

Which principle most directly motivates distributed word representations?

A. Words with similar contexts tend to have similar meanings.

B. Every word must have a unique binary code.

C. Longer documents always have greater semantic value.

D. Frequency and meaning are always identical.

Correct Answer: A. Words with similar contexts tend to have similar meanings.

Question 2

What does cosine similarity primarily compare?

A. The number of tokens in two documents

B. The angle or orientation between two non-zero vectors

C. The alphabetical order of vocabulary items

D. The training time of two models

Correct Answer: B. The angle or orientation between two non-zero vectors

Question 3

For x = [1,2] and y = [2,1], what is their cosine similarity?

A. 0

B. 0.5

C. 0.8

D. 1.414

Correct Answer: C. 0.8

Question 4

What is a principal limitation of static word embeddings?

A. They cannot be stored digitally.

B. They assign the same vector to a word in different contexts.

C. They require no training data.

D. They only represent punctuation.

Correct Answer: B. They assign the same vector to a word in different contexts.

Question 5

In k-means clustering, what does the centroid represent?

A. The mean vector of a cluster

B. The least frequent document

C. The longest document in the corpus

D. A randomly selected vocabulary item only

Correct Answer: A. The mean vector of a cluster

Question 6

Which metric measures whether relevant results appear within the first k retrieved positions?

A. Recall@k

B. Vocabulary size

C. Token frequency

D. Embedding dimension

Correct Answer: A. Recall@k

Question 7

Why can a semantic-search result be unsuitable even when its similarity score is high?

A. Similarity is always a probability of truth.

B. The passage may be obsolete, unauthorized, or factually inappropriate.

C. High similarity proves that the passage is current.

D. Dense vectors cannot represent text.

Correct Answer: B. The passage may be obsolete, unauthorized, or factually inappropriate.

Question 8

What is the main purpose of reranking in retrieval systems?

A. To delete all retrieved passages

B. To reorder initial candidates using a more detailed model

C. To convert documents into one-hot vectors

D. To remove document metadata

Correct Answer: B. To reorder initial candidates using a more detailed model

Question 9

Which practice best addresses embedding bias?

A. Evaluate only aggregate accuracy.

B. Remove all metadata without testing.

C. Use subgroup evaluation, documentation, and human review.

D. Assume a larger vector dimension eliminates bias.

Correct Answer: C. Use subgroup evaluation, documentation, and human review.

Question 10

Why should document permissions be applied before indexing sensitive material?

A. To make vectors shorter

B. To prevent unauthorised content from becoming retrievable

C. To increase the number of duplicate passages

D. To guarantee perfect semantic similarity

Correct Answer: B. To prevent unauthorised content from becoming retrievable