Text Representation and Semantic Models

Introduction

Text analytics begins with a deceptively difficult task: converting language into a form that computational systems can store, compare, model and retrieve. Human language is symbolic, contextual and socially situated. Computers, however, operate on numerical structures. A representation therefore determines what information a model can observe and what information it will ignore.

A representation based only on word counts may be useful for identifying topics, but it generally misses word order and ambiguity. A representation based on dense embeddings may capture semantic similarity, but it can obscure interpretability and reproduce biases in the data used to train it. In business analytics, representation is consequently not a minor preprocessing choice; it is a modelling decision with consequences for prediction, search, customer intelligence, risk analysis and managerial action.

This chapter compares representations that encode lexical, syntactic and semantic information: bag-of-words, n-grams, TF–IDF, engineered features, static embeddings, contextual embeddings and vector databases. It also explains how these methods fit into practical analytical systems and how their limitations should be governed.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain how text is transformed into numerical representations.
2. Compare count-based, weighted, feature-engineered and embedding-based representations.
3. Calculate and interpret TF–IDF values.
4. Distinguish lexical, syntactic and semantic information.
5. Select an appropriate representation for classification, clustering, search and social-media analytics.
6. Explain the role of vector databases in retrieval and semantic applications.
7. Evaluate representation quality, bias, interpretability, computational cost and governance risks.

1. What Does a Text Representation Capture?

A document representation is a mapping from a text object to a numerical structure. A document may be a customer review, social-media post, support ticket, news article or conversation. The representation can be:

• Sparse, with many zero values, as in a document–term matrix.
• Dense, with relatively few numerical dimensions, as in an embedding vector.
• Static, where a word has one vector regardless of context.
• Contextual, where the representation changes according to surrounding words.
• Interpretable, where dimensions have clear meanings, or distributed, where meaning is spread across many dimensions.

Three kinds of information are particularly important:

No representation captures all three perfectly. The best choice depends on the task, language, data volume, latency requirements and need for explanation.

2. Preparing Text Before Representation

Representation normally follows a pipeline involving collection, cleaning, segmentation and feature construction. Tokenization divides text into words, subwords or characters. Lowercasing can reduce vocabulary size, but it may remove useful distinctions such as product codes or proper names. Stop-word removal can reduce noise in topic classification, but words such as “not” may be essential for sentiment. Stemming truncates words to crude roots, whereas lemmatization uses linguistic information to identify dictionary forms.

Social-media text requires additional decisions. Hashtags may contain topical information; emojis can express sentiment; mentions identify actors; URLs may be useful for source analysis; repeated characters can indicate emphasis; and code-switching can make language-specific processing unreliable. A good pipeline preserves information that is relevant to the business question rather than applying cleaning rules mechanically.

Preprocessing must also prevent data leakage. For example, vocabulary construction, feature selection and document-frequency calculations should be fitted on the training set only when evaluating a predictive model. Otherwise, information from the test set can influence the representation and produce overly optimistic results.

3. Bag-of-Words and Document–Term Matrices

The bag-of-words (BoW) model represents a document by the words it contains and their frequencies, while ignoring grammatical order. Suppose the vocabulary is [“analytics”, “customer”, “risk”]. A document containing “customer risk risk” is represented by the count vector [0, 1, 2]. For a collection of documents, these vectors form a document–term matrix.

BoW is attractive because it is simple, fast and often highly effective for linear classification. It is also interpretable: a large positive coefficient for a term can be examined by an analyst. However, BoW treats “customer praised service” and “service praised customer” as equivalent if they contain the same words. It also treats synonyms as unrelated features and cannot naturally resolve polysemy: “bank” in a financial report and “bank” in a river description share one feature.

Binary presence can sometimes be preferable to counts. In a binary representation, a feature is 1 if a term occurs and 0 otherwise. Binary features are useful when repeated mentions do not indicate proportionally greater evidence, such as detecting whether a document contains a legal disclaimer.

The main assumptions of BoW are that the selected vocabulary is adequate and that word occurrence provides sufficient evidence for the task. These assumptions are often reasonable for keyword monitoring, spam detection and interpretable baseline models, but weaker for semantic search and nuanced language understanding.

4. N-Grams: Introducing Local Order

An n-gram is a contiguous sequence of n tokens. Unigrams contain one token, bigrams contain two and trigrams contain three. The phrase “poor delivery experience” produces bigrams “poor delivery” and “delivery experience”. N-grams preserve limited local order and can distinguish “not useful” from “useful”.

Character n-grams are especially valuable for misspellings, hashtags, morphology and noisy social text. A character model may recognize similarities among “connect”, “connected” and “connection” without requiring each complete word to be in the vocabulary.

N-grams increase dimensionality. If a vocabulary contains V tokens, the theoretical number of ordered bigrams is V², although only a small subset occurs. Sparse storage and frequency thresholds are therefore important. Longer n-grams may improve specificity but become rare and brittle. In practice, unigram–bigram combinations are often a strong baseline, while trigrams should be introduced when sufficient data and a clear task justification exist.

N-grams encode local syntax, not complete meaning. They can identify phrases such as “credit card” or “machine learning”, but they do not generally understand that two differently worded sentences express the same idea.

5. TF–IDF Weighting

Raw counts give frequent terms substantial influence. A term that occurs in nearly every document, however, may be poor at distinguishing documents. TF–IDF reduces the influence of common terms and emphasizes terms that are frequent in one document but uncommon across the collection.

A common term-frequency definition is normalized frequency:

Here, f(t,d) is the count of term t in document d, V is the vocabulary and the denominator is the total number of term occurrences in d. The inverse-document-frequency component is commonly defined as:

N is the number of documents and df(t) is the number of documents containing t. The TF–IDF weight is:

The logarithm may use any consistent base; changing the base rescales values but does not change their ranking. Implementations often use smoothed variants, such as log((N + 1)/(df(t) + 1)) + 1, to avoid zero values and handle edge cases. The chosen convention must be documented because numerical values are not interchangeable across implementations.

Worked Example

Assume N = 100 documents. In document d, the term “refund” occurs 2 times in a document containing 200 total tokens. It occurs in df(t) = 5 documents. Using natural logarithms:

TF = 2/200 = 0.01.

IDF = ln(100/5) = ln(20) ≈ 2.996.

Therefore:

TF–IDF ≈ 0.01 × 2.996 = 0.02996.

The result is dimensionless because both TF and IDF are ratios or logarithms of ratios. The value is meaningful only relative to the same weighting convention and corpus. A term occurring in every document has IDF = ln(1) = 0, although smoothing may prevent an exact zero.

TF–IDF is effective for document similarity, information retrieval, topic indicators and linear predictive models. It remains sparse and largely lexical: it does not understand that “reimbursement” and “refund” may be semantically related unless both are connected through additional features or training.

6. Feature Engineering for Text and Social Media

Feature engineering adds structured evidence to lexical representations. Examples include:

• document length and sentence count;
• counts of positive, negative or domain-specific terms;
• punctuation, capitalization and repeated-character indicators;
• emoji and hashtag counts;
• readability measures;
• time, author, language and source features;
• reply, repost, mention and interaction counts;
• named entities, topics and extracted products;
• sentiment scores and uncertainty markers.

These features can improve performance when the business construct is partly behavioural or structural. For example, fraud detection may benefit from account age, posting frequency and coordination patterns in addition to text. Customer-service escalation may depend on message sentiment, response delay and the number of previous contacts.

Feature engineering requires construct validity. A feature should have a defensible relationship to the target, not merely correlate accidentally in one sample. Analysts should examine missingness, stability over time, subgroup effects and possible leakage. A feature such as “presence of the word cancellation” may be predictive, but it can become invalid if a policy change alters how customers phrase requests.

7. Static Word Embeddings

Static embeddings represent each vocabulary item as a dense vector. Word2Vec, GloVe and FastText are influential examples. They learn geometric relationships from distributional evidence: words appearing in similar contexts tend to have similar representations. Similarity is often measured using cosine similarity:

x and y are vectors, x · y is their dot product and ||x||₂ is the Euclidean norm. The value lies between −1 and 1 for non-zero vectors, although practical embedding distributions often occupy a narrower range. Cosine similarity is scale-invariant and is appropriate when direction represents similarity.

A document vector can be formed by averaging word vectors, sometimes weighted by TF–IDF. This is computationally efficient, but averaging loses word order and may blur important distinctions. Static embeddings also assign one vector to a word, so “light” has the same representation in “light traffic” and “light meal”. FastText partially addresses rare and misspelled words through character subword information, but it remains fundamentally static.

Embeddings are not automatically neutral. They can encode social stereotypes, historical inequities and domain-specific associations. A vector relation that appears mathematically coherent may be socially inappropriate or operationally harmful.

8. Contextual Embeddings and Transformer Models

Contextual models produce representations dependent on surrounding text. In “the bank approved the loan” and “the bank was beside the river”, the representation of “bank” can differ. Transformer architectures use attention mechanisms to model relationships among tokens, often with bidirectional context in encoder models such as BERT-like systems. Decoder models are optimized for generation, while encoder and encoder–decoder models are commonly used for understanding and transformation tasks.

Subword tokenization helps models handle rare words, compounds and misspellings. A sentence is converted into tokens, processed through multiple layers and mapped to hidden vectors. These vectors may be used directly for similarity, pooled for classification, or passed to a task-specific head through fine-tuning.

Contextual representations generally capture richer syntax and semantics than BoW or static embeddings, but they require greater computational resources and are less transparent. Their performance can degrade under domain shift, such as applying general web-trained language models to clinical, financial or multilingual social-media data. Fine-tuning may improve domain performance but can overfit small datasets and amplify annotation bias.

The distinction between a model’s internal representation and a useful business representation is important. A high-dimensional hidden state is not automatically a reliable measure of sentiment, intent or truth. Task-specific evaluation remains necessary.

9. Vector Databases and Semantic Retrieval

A vector database stores embedding vectors together with metadata and supports nearest-neighbour search. A typical semantic-search workflow is:

1. Split source documents into meaningful chunks.
2. Generate an embedding for each chunk.
3. Store the vector, text, source, date, access permissions and other metadata.
4. Embed a user query using the same or a compatible model.
5. Retrieve vectors with high similarity.
6. Filter by metadata and optionally rerank the results.
7. Present evidence or pass retrieved text to a downstream model.

Approximate nearest-neighbour (ANN) methods, including graph-based and inverted-index approaches, trade a small amount of exactness for speed and scale. The choice of index, distance function, vector dimension, update process and hardware affects latency and recall. A vector database does not create semantic understanding by itself; it provides infrastructure for storing and retrieving representations.

Hybrid retrieval combines lexical search, such as BM25 or TF–IDF, with vector search. Lexical retrieval is strong for exact identifiers, legal phrases and product codes. Vector retrieval is strong for paraphrases and conceptually related expressions. A hybrid ranker can therefore be more robust than either method alone.

Metadata is essential for governance. Access controls, document version, publication date, tenant identity and deletion status should be stored and enforced. Embeddings can also leak information through membership inference or similarity attacks, so sensitive data requires careful protection.

10. Comparative Evaluation

Evaluation should be task-specific. Classification may use precision, recall, F1 or calibration. Ranking may use mean reciprocal rank, precision at k or normalized discounted cumulative gain. Clustering requires both internal measures and substantive inspection. Representation quality should also be tested for robustness across time, language, demographic groups, domains and adversarial wording.

A useful experimental design begins with an interpretable baseline, such as TF–IDF plus logistic regression. More complex embeddings should be justified by measurable gains, acceptable latency and operational value. Statistical comparison should use consistent splits and, where possible, confidence intervals or repeated evaluation rather than a single score.

11. Illustrative Case: Customer-Service Semantic Search

Background. A telecommunications company receives 200,000 support messages per month. Agents search a knowledge base using exact keywords, but customers use varied language: “my handset keeps dropping calls” and “voice connection repeatedly fails” may describe the same issue.

Problem. Keyword search returns irrelevant results when the query is paraphrased. However, purely semantic search sometimes retrieves documents from the wrong product line or outdated tariff.

Analysis. The analytics team creates a benchmark containing representative queries, relevant documents, product metadata and time-validity labels. A TF–IDF baseline performs well for model numbers and exact error codes but poorly for paraphrases. A general-purpose embedding model improves paraphrase retrieval but confuses similar concepts across products.

Approach. The team implements hybrid retrieval. TF–IDF or BM25 supplies lexical candidates, while a domain-adapted embedding model supplies semantic candidates. Results are filtered by product and region metadata, then reranked using a cross-encoder. Documents are chunked by procedure rather than arbitrary character length. Evaluation uses precision at 5, response latency and agent resolution rate. Access permissions and document expiry are enforced at retrieval time.

Outcome. The hybrid system improves relevant results for paraphrased queries while retaining exact-code performance. Agents report fewer searches, but the company keeps the lexical baseline as a fallback and monitors drift after product launches.

Learning points. First, representation should follow the retrieval problem rather than fashion. Second, metadata and freshness can be as important as vector similarity. Third, human evaluation is necessary because a mathematically close document may still be operationally wrong. Finally, retrieval systems should expose source evidence so agents can verify recommendations.

12. Ethical, Governance and Managerial Considerations

Text representations can affect people even when they are not visible to decision-makers. Sentiment systems may interpret dialect, sarcasm or code-switching unevenly. Toxicity models can over-flag identity terms. Embeddings trained on historical customer interactions may encode unequal service patterns. Organizations should document training data, intended use, known limitations and evaluation populations.

Privacy is especially important for social-media and customer text. Collection should have a lawful and legitimate purpose; retention should be limited; access should be controlled; and sensitive attributes should not be inferred or used without justification. Vectorization does not anonymize text: an embedding may still facilitate reconstruction, linkage or sensitive similarity inference.

For high-impact applications, governance should include human oversight, appeal mechanisms, model monitoring and clear accountability. Automated classifications should not be treated as facts merely because they are expressed numerically. Explainability may involve showing influential terms, retrieved evidence, comparable examples or model confidence, while acknowledging that post-hoc explanations can be incomplete.

13. Best-Practice Decision Framework

Choose BoW or TF–IDF when data are limited, interpretability matters, exact terms are important or a strong baseline is needed. Add word or character n-grams when phrases, negation, spelling variation or local syntax matter. Engineer structured features when metadata and behaviour are substantively relevant.

Choose static embeddings when low latency, compact storage or limited compute is important and contextual ambiguity is manageable. Choose contextual embeddings when paraphrase, polysemy, long-range context or nuanced meaning materially affects the task. Use vector databases when embeddings must be searched repeatedly at scale, but combine them with metadata filtering, access controls, versioning and, where appropriate, lexical retrieval.

A mature architecture often uses several representations rather than one. For example, a classifier may combine TF–IDF, character n-grams and behavioural features, while an enterprise search system may combine lexical retrieval, dense retrieval and a reranker. Complexity should be earned through validated improvement.

Contemporary Developments

Current practice increasingly combines retrieval with generative language models. In retrieval-augmented generation, relevant chunks are retrieved before a model produces an answer. This can improve grounding, but it does not eliminate errors: poor chunking, stale indexes, inadequate filters and irrelevant retrieval can still produce unsupported answers.

Multimodal and multilingual embeddings are expanding search across text, images, audio and languages. Domain adaptation, parameter-efficient fine-tuning and model distillation can reduce cost. At the same time, evaluation is moving beyond accuracy toward robustness, fairness, calibration, energy use, provenance and resistance to prompt or retrieval manipulation.

Chapter Summary

Text representation is the foundation of text and social-media analytics. Bag-of-words and TF–IDF provide transparent, sparse lexical representations; n-grams add local order and robustness to phrases or spelling variation; engineered features incorporate domain, behavioural and structural evidence. Static embeddings provide compact distributional semantics but assign one vector to each word. Contextual embeddings adapt meaning to surrounding text and generally capture richer linguistic information at greater computational and governance cost. Vector databases operationalize embedding-based retrieval, but their usefulness depends on embedding quality, metadata, access control, freshness and evaluation. Effective business systems usually compare an interpretable baseline with more complex alternatives and select representations according to the task, data, risks and operational constraints.

Key Terms

Representation: A numerical encoding of text used by an analytical system.

Vocabulary: The set of tokens or features recognized by a representation.

Bag-of-words: A representation based on term occurrence or frequency without word order.

Document–term matrix: A matrix whose rows are documents and columns are terms or features.

Tokenization: Dividing text into computational units.

N-gram: A contiguous sequence of n tokens.

TF–IDF: A weighting scheme combining within-document frequency and corpus rarity.

Sparse vector: A vector containing mostly zero values.

Embedding: A dense numerical vector intended to encode relationships or meaning.

Static embedding: A word representation that does not change with context.

Contextual embedding: A representation whose value depends on surrounding text.

Cosine similarity: A directional similarity measure based on the angle between vectors.

Polysemy: The presence of multiple related meanings for one word form.

Feature engineering: Designing informative variables from raw data and domain knowledge.

Vector database: A system for storing vectors and performing similarity retrieval.

Approximate nearest neighbour: A fast search method that may trade exactness for efficiency.

Hybrid retrieval: Retrieval combining lexical and semantic methods.

Data leakage: The unintended use of information unavailable at prediction time.

Embedding drift: Change in representation behaviour caused by changes in data, models or domains.

Retrieval-augmented generation: Generation supported by retrieved external evidence.

Review Questions

Short Answer Questions

1. Why does bag-of-words fail to represent word order?
2. What is the purpose of inverse document frequency?
3. Distinguish static and contextual embeddings.
4. Why are character n-grams useful for social-media text?
5. What role does metadata play in vector retrieval?

Long Answer Questions

1. Compare BoW, TF–IDF and n-gram representations in terms of information captured, interpretability and computational cost.
2. Explain the TF–IDF formula, including its assumptions and common implementation variations.
3. Discuss the advantages and limitations of static and contextual embeddings for business analytics.
4. Explain how a hybrid semantic-search architecture works and why it may outperform a single retrieval method.
5. Critically evaluate the ethical and governance risks of embedding-based text analytics.

Analytical / Application Questions

1. Design a representation pipeline for classifying urgent customer-service messages from noisy social-media posts. Justify preprocessing, features, model choice and evaluation.
2. A company has poor search performance for paraphrased queries but excellent performance for product codes. Propose an architecture and an experiment to test it.
3. Develop a monitoring plan for detecting representation drift and subgroup performance differences in a multilingual sentiment system.

Multiple Choice Questions with Answers

Question 1

Which information is directly ignored by a standard bag-of-words representation?

A. Whether a term occurs

B. The frequency of a term

C. The order of terms

D. The vocabulary used

Correct Answer: C. The order of terms

Question 2

In TF–IDF, a term appearing in every document has an unsmoothed IDF of:

A. 0

B. 1

C. The number of documents

D. The term frequency

Correct Answer: A. 0

Question 3

What is the principal advantage of a word bigram over a unigram representation?

A. It removes all sparsity

B. It captures local word order

C. It guarantees semantic understanding

D. It requires no vocabulary

Correct Answer: B. It captures local word order

Question 4

If a term occurs twice in a 200-token document, its normalized term frequency is:

A. 0.001

B. 0.01

C. 0.02

D. 100

Correct Answer: B. 0.01

Question 5

Which representation gives different vectors to a word according to its surrounding context?

A. Binary BoW

B. Static embedding

C. Contextual embedding

D. Document frequency

Correct Answer: C. Contextual embedding

Question 6

Cosine similarity primarily compares the:

A. Length of two documents

B. Angle between two vectors

C. Number of vocabulary terms

D. Number of database records

Correct Answer: B. Angle between two vectors

Question 7

Why can hybrid retrieval be valuable?

A. It eliminates the need for metadata

B. It combines exact lexical matching with semantic similarity

C. It guarantees correct generated answers

D. It makes embeddings unnecessary

Correct Answer: B. It combines exact lexical matching with semantic similarity

Question 8

Which is an example of feature leakage?

A. Using training documents to fit a vocabulary

B. Using a feature created from information available at prediction time

C. Using character n-grams for misspellings

D. Using metadata filters in search

Correct Answer: B. Using a feature created from information available at prediction time

Question 9

What is a central function of a vector database?

A. Tokenizing every language perfectly

B. Generating labels without training data

C. Storing vectors and supporting similarity search

D. Replacing access-control policies

Correct Answer: C. Storing vectors and supporting similarity search

Question 10

Which representation is generally most interpretable for identifying influential individual terms in a linear classifier?

A. TF–IDF features

B. An unnamed transformer hidden state

C. An approximate-neighbour graph

D. A pooled multimodal embedding

Correct Answer: A. TF–IDF features