Text Representation and Language Features

Introduction and Context

Natural-language text is rich in meaning but is not directly usable by most machine-learning algorithms. A learning algorithm requires numerical inputs, so text must be transformed into vectors, matrices, or sequences of vectors. This transformation is called text representation or feature representation.

The choice of representation affects what a model can learn. A representation may preserve word identity, document frequency, local word order, semantic similarity, grammatical context, or several of these properties at once. It also affects memory usage, training speed, interpretability, fairness, and performance on new data.

This chapter compares six important families of representations:

• one-hot encoding;
• bag-of-words;
• n-grams;
• TF–IDF;
• static word embeddings; and
• contextual embeddings.

The central professional question is not which representation is universally best. It is: which representation preserves the information required by the task at an acceptable computational and operational cost?

Learning Objectives

By the end of this chapter, you should be able to:

1. explain how text is converted into numerical features;
2. distinguish one-hot, count-based, weighted, static, and contextual representations;
3. calculate term frequency, inverse document frequency, TF–IDF, and cosine similarity;
4. evaluate representations using semantic, computational, and practical criteria;
5. select a suitable representation for a text-classification or language-processing task; and
6. identify limitations involving ambiguity, vocabulary coverage, privacy, bias, and interpretability.

1. From Text to Numerical Features

A text-processing system commonly follows this pipeline:

raw text → normalization and tokenization → vocabulary → numerical representation → model

Tokenization divides text into units called tokens. A token may be a word, punctuation mark, subword unit, or character. For example, the sentence “Deep learning works!” might be tokenized as [“Deep”, “learning”, “works”, “!”]. Lowercasing, punctuation removal, stemming, and stop-word removal are optional preprocessing operations. They should be used only when they support the task; removing negation words such as “not” can damage sentiment analysis.

A vocabulary is the set of recognized tokens. If the vocabulary contains V tokens, a document-level representation may have V dimensions, or more if n-grams are included. A large vocabulary increases memory and computation, particularly for sparse methods.

Important terminology includes:

• Sparse vector: a vector containing mostly zero values.
• Dense vector: a vector in which most dimensions have non-zero values.
• Feature: a measurable input supplied to a model.
• Lexical information: information about the actual words or symbols used.
• Semantic information: information about meaning and relationships.
• Context: surrounding words that influence interpretation.

2. One-Hot Encoding

One-hot encoding represents one token using a vector with a single 1 and zeros elsewhere. Suppose the vocabulary is [cat, dog, fish]. Then:

• cat → [1, 0, 0]
• dog → [0, 1, 0]
• fish → [0, 0, 1]

Formally, token t is represented by a vector e(t) of length V, where the component corresponding to t is 1 and every other component is 0.

One-hot vectors are useful for representing categorical identity and are easy to interpret. However, they have serious limitations:

1. No similarity: cat and dog are equally distant from each other as cat and fish. Using Euclidean distance, any two different one-hot vectors have distance √2.
2. High dimensionality: a vocabulary of one million tokens produces one-million-dimensional vectors.
3. Sparsity: almost all entries are zero.
4. No compositional meaning: the vector does not indicate relationships such as synonymy or topical similarity.
5. Out-of-vocabulary problem: an unseen token has no vector unless an unknown-token category is provided.

One-hot encoding is suitable for small categorical vocabularies, lookup tables, and the initial representation used by an embedding layer. It is generally not an adequate standalone representation for semantic NLP tasks.

3. Bag-of-Words Representation

The bag-of-words (BoW) model represents a document by the number of times each vocabulary term occurs. It ignores word order.

Consider the vocabulary [data, deep, model, text]. The document “deep text model text” becomes [0, 1, 1, 2]. This is a count vector, not a one-hot vector, because a document can contain a term multiple times.

A binary BoW variant records only presence or absence. The same document becomes [0, 1, 1, 1]. A count representation preserves repetition; a binary representation can be more robust when repetition is not meaningful.

BoW is effective for many document-classification tasks because terms such as “refund”, “fraud”, or “urgent” can be strong indicators of a class. It is also highly interpretable: a feature corresponds to an identifiable word. Linear classifiers, including logistic regression and linear support vector machines, often perform well with BoW features.

Its main weakness is that word order and meaning are lost. “The dog chased the cat” and “The cat chased the dog” have the same unigram BoW representation. BoW also produces high-dimensional sparse vectors and does not naturally recognize that “automobile” and “car” are related.

4. N-Grams and Local Language Features

An n-gram is a consecutive sequence of n tokens.

• A unigram contains one token.
• A bigram contains two consecutive tokens.
• A trigram contains three consecutive tokens.

For “machine learning is useful”, the bigrams are “machine learning”, “learning is”, and “is useful”. A unigram-plus-bigram representation can distinguish “not good” from “good”, which a unigram-only model may not do reliably.

N-grams add local order and phrase information. They are useful for sentiment analysis, spam detection, authorship analysis, and information retrieval. Character n-grams can handle misspellings, morphological variation, and previously unseen words. For example, character n-grams can relate “organize” and “organised” more effectively than exact word matching.

The principal cost is feature growth. If there are V possible tokens, the theoretical number of possible bigrams is V², although only a small proportion appears in a corpus. Trigrams are even more expensive. Feature selection, frequency thresholds, hashing, and regularization are common controls.

N-grams remain local. A phrase occurring several sentences away is not represented as a direct relationship. They also do not automatically provide general semantic similarity.

5. TF–IDF: Weighting Informative Terms

Raw counts give high weight to frequent terms, but a word appearing in nearly every document is often not useful for distinguishing documents. Term frequency–inverse document frequency (TF–IDF) reduces the influence of terms that occur widely across a collection.

Let:

• D be the number of documents;
• tf(t,d) be the term frequency of term t in document d; and
• df(t) be the number of documents containing t.

A common logarithmic inverse-document-frequency definition is:

The basic TF–IDF weight is:

This definition assumes df(t) > 0 and uses document frequency rather than total token frequency. If a term occurs in every document, df(t) = D and idf(t) = log(1) = 0. The term therefore receives no discriminating weight. Different software libraries use smoothing, such as log((D + 1)/(df(t) + 1)) + 1, and sometimes normalize term frequency. The exact convention must be documented.

Worked Example: TF–IDF

Suppose a collection contains D = 100 documents. The term “invoice” occurs in 10 documents, and it occurs twice in document d. Using natural logarithms:

With raw term frequency tf(invoice,d) = 2:

The value has no physical unit. It is a relative feature weight. The calculation is meaningful only under the stated TF–IDF convention; another convention may produce a different numerical value.

TF–IDF vectors are often normalized before similarity calculation. For vectors x and y, cosine similarity is:

Here x · y is the dot product and ||x||₂ is the Euclidean norm, defined by ||x||₂ = √(∑ᵢ xᵢ²). For non-zero vectors, cosine similarity ranges from −1 to 1 in general; non-negative TF–IDF vectors produce values from 0 to 1. Cosine similarity measures orientation rather than document length and is widely used in information retrieval.

TF–IDF is strong when exact terms matter, training data is limited, and interpretability is important. It is weaker when paraphrases, word order, polysemy, or long-range dependencies are central.

6. Static Word Embeddings

A static word embedding maps each vocabulary word to a dense vector, commonly with tens or hundreds of dimensions. Words used in similar contexts tend to have nearby vectors. Similarity is often measured with cosine similarity.

Methods such as Word2Vec, GloVe, and fastText learn embeddings from large text collections. Word2Vec includes predictive approaches such as continuous bag-of-words and skip-gram. GloVe uses global co-occurrence statistics. fastText represents words partly through character n-grams, improving handling of rare and morphologically related words.

Unlike one-hot vectors, embeddings can encode graded relationships. For example, vectors for “doctor” and “nurse” may be closer than vectors for “doctor” and “volcano”, depending on the training data. However, this relationship is statistical rather than a guaranteed definition.

Static embeddings have three important properties:

1. Dense and compact: a 300-dimensional vector can replace a vocabulary-sized sparse vector.
2. Transferable: pretrained vectors can provide useful information when labelled data is scarce.
3. One vector per word type: the word “bank” has the same vector in “river bank” and “bank loan”.

The third property is a major limitation. Static embeddings do not resolve word meaning using sentence context. They may also encode social stereotypes and domain-specific errors inherited from training data. Fine-tuning or domain adaptation may be necessary.

7. Contextual Embeddings

A contextual embedding assigns a representation based on the surrounding sequence. Thus, “bank” can receive different vectors in “river bank” and “bank loan”. Modern contextual representations are usually produced by neural language models, especially Transformer-based models such as BERT-like encoders and generative language models.

A Transformer uses attention mechanisms to allow tokens to incorporate information from other tokens. A simplified scaled dot-product attention operation is:

Here Q, K, and V are query, key, and value matrices; dₖ is the key-vector dimension; and softmax converts scores into non-negative weights that sum to 1 across the attended positions. The scaling factor helps control the magnitude of dot products. This formula describes one attention head and does not by itself describe a complete Transformer model.

Contextual embeddings can capture syntax, semantic relations, negation, and long-range dependencies more effectively than BoW or static embeddings. They support tasks such as named-entity recognition, question answering, summarization, sentiment analysis, and semantic search.

Their costs include:

• greater memory and computational requirements;
• increased latency, especially for long sequences;
• more difficult interpretation;
• dependence on model and tokenizer choices;
• possible privacy risks when sensitive text is sent to external services; and
• inherited bias, hallucination risk in generative systems, and uncertain behaviour outside the training distribution.

A contextual embedding is not automatically a complete solution. A model may still fail on domain terminology, sarcasm, code-switching, low-resource languages, or very long documents.

8. Comparison of Representations

The representations are not mutually exclusive. A system may combine TF–IDF with metadata, character n-grams with word features, or contextual embeddings with a task-specific classifier.

9. Selecting a Representation

Representation selection should begin with the task rather than with a fashionable model. The following criteria are useful.

Semantic requirements

Ask what information must be preserved:

• If exact keywords determine the outcome, TF–IDF may be sufficient.
• If short phrases and negation matter, word or character n-grams are valuable.
• If synonymy and paraphrase matter, embeddings are preferable.
• If the same word changes meaning by context, contextual embeddings are more suitable.
• If long-range relationships are essential, local n-grams alone are inadequate.

Computational requirements

Consider feature dimension, sparsity, training time, inference latency, memory, hardware, and expected traffic. Sparse TF–IDF can be highly efficient for large document collections. Contextual models may require GPUs or model compression for production deployment. Distillation, quantization, caching, and shorter input sequences can reduce cost, but may reduce accuracy.

Data and domain requirements

A small labelled dataset often benefits from pretrained embeddings or a strong linear TF–IDF baseline. A specialized domain may require domain-specific pretraining or continued training. Vocabulary coverage should be tested on real operational data, including misspellings and technical terms.

Interpretability and governance

In regulated or high-impact settings, feature-level explanations may favour TF–IDF and linear models. Embeddings can still be used, but the organization should document training data, model versions, evaluation groups, retention policies, and human-review procedures.

A practical evaluation procedure is:

1. establish a simple baseline, such as majority class and TF–IDF with logistic regression;
2. split data by time or entity when random splitting would cause leakage;
3. compare representations using task-appropriate metrics;
4. measure latency, memory, and failure cases, not only accuracy;
5. test robustness to spelling variation, negation, new terminology, and distribution shift; and
6. choose the simplest representation that meets the required quality and operational constraints.

10. Applied Case: Classifying Customer Support Messages

The following is an illustrative case.

Background

A telecommunications provider receives messages that must be routed to billing, technical support, cancellation, or account security teams. The company has 40,000 labelled historical messages.

Problem

The existing keyword rules perform poorly when customers use paraphrases. For example, “I was charged twice” and “the same payment appears two times” express a similar billing problem. However, an advanced model may be too expensive for the provider’s limited inference infrastructure.

Analysis

A BoW model is transparent and fast but may fail on paraphrases. Word bigrams can identify phrases such as “charged twice” and “cancel service”. Character n-grams can handle spelling errors. TF–IDF can reduce the impact of common terms such as “please” and “account”. Static embeddings may improve semantic similarity but treat “charge” identically in billing and battery-related contexts. Contextual embeddings can distinguish these meanings, but their cost and governance requirements are higher.

Approach and solution

The team first builds a TF–IDF model using word unigrams, bigrams, and character n-grams, followed by a regularized linear classifier. It evaluates performance using a time-based validation set because future messages should be predicted from past data. It reports macro-F1, because a smaller security class is operationally important, as well as latency and memory.

A contextual encoder is then tested as a second-stage model only for messages for which the baseline confidence is low. This hybrid architecture preserves speed for routine cases and applies deeper semantic processing to ambiguous cases. Human agents review security-related predictions and can correct labels for future retraining.

Outcome

The sparse model provides a transparent and inexpensive production baseline. The hybrid design improves handling of paraphrases without applying the expensive model to every message. The final system is selected not solely because it has the highest test score, but because it satisfies response-time, auditability, and security requirements.

Learning points

1. A strong baseline is essential for meaningful comparison.
2. Representation choice should reflect the error costs of the application.
3. Hybrid systems can balance semantic quality and computation.
4. Evaluation must include operational and governance criteria.
5. Confidence thresholds and human review are important for high-impact categories.

11. Challenges, Ethics, and Best Practices

Text representations inherit properties from data. If a corpus contains stereotypes, under-representation, or offensive associations, embeddings may encode them. A representation can also expose sensitive information through memorization or logging. Text may contain names, health information, financial records, or confidential business material.

Best practices include:

• minimize collection and retain only necessary text;
• remove or protect personal identifiers where feasible;
• document corpus sources, preprocessing, vocabulary, and model versions;
• assess performance across languages, dialects, demographic groups, and writing styles;
• avoid treating embedding similarity as proof of factual or social equivalence;
• use access controls and encryption for sensitive data;
• monitor vocabulary drift and changes in user behaviour; and
• provide human escalation for consequential decisions.

Subword tokenization and multilingual models improve coverage but do not guarantee equal performance across languages. Contextual models also have finite input limits; long documents may require chunking, which can separate evidence and distort meaning. Chunking strategy must therefore be evaluated as part of the representation pipeline.

12. Contemporary Developments

Current NLP systems increasingly use pretrained Transformer models, instruction-tuned language models, retrieval-augmented generation, and domain-adapted encoders. In retrieval systems, dense embeddings can find semantically related passages, while sparse lexical methods preserve exact-match performance. Hybrid retrieval often combines both signals.

Smaller distilled models, quantized models, and efficient attention mechanisms are making contextual representations more practical at the edge and in high-volume services. Nevertheless, model size is not a substitute for careful data design, evaluation, privacy protection, and monitoring.

Chapter Summary

Text representation converts language into numerical form for machine learning. One-hot encoding preserves token identity but provides no semantic similarity. Bag-of-words and n-grams are sparse, interpretable methods; n-grams add local order but can produce many features. TF–IDF weights terms according to their importance within a document collection and is a strong baseline for search and classification. Static embeddings provide compact distributional semantics but assign one vector to each word type. Contextual embeddings condition representations on surrounding text and therefore handle ambiguity and long-range meaning more effectively, at greater computational and governance cost.

Representation selection should consider semantic requirements, data size, vocabulary coverage, computation, latency, interpretability, privacy, fairness, and error consequences. In professional systems, a carefully evaluated simple baseline, sometimes combined with a selective contextual model, may be more appropriate than an unnecessarily large model.

Key Terms

One-hot encoding: A vector representation with one active position identifying a token.

Bag-of-words: A document representation based on token presence or counts without word order.

N-gram: A consecutive sequence of n tokens or characters.

Term frequency: The frequency of a term within a document.

Document frequency: The number of documents containing a term.

Inverse document frequency: A weight that decreases as a term appears in more documents.

TF–IDF: The product of term frequency and inverse document frequency.

Sparse vector: A vector containing mostly zero values.

Dense vector: A vector in which most dimensions contain non-zero values.

Embedding: A learned numerical vector representing an item such as a word or sentence.

Static embedding: An embedding that assigns one vector to a word type regardless of context.

Contextual embedding: An embedding whose value depends on surrounding text.

Vocabulary: The set of tokens recognized by a representation system.

Out-of-vocabulary token: A token not present in the system’s vocabulary.

Cosine similarity: A measure of the angle-based similarity between two non-zero vectors.

Attention: A mechanism that weights information from other positions in a sequence.

Semantic similarity: Similarity based on meaning rather than exact word identity.

Feature sparsity: The condition in which most feature values are zero.

Distribution shift: A change between the data used for development and the data encountered in operation.

Review Questions

Short Answer Questions

1. What information is lost by a unigram bag-of-words representation?
2. Why does TF–IDF reduce the influence of terms appearing in nearly every document?
3. State one difference between static and contextual embeddings.
4. What is the purpose of character n-grams?
5. Name two computational costs associated with contextual embeddings.

Long Answer Questions

1. Compare one-hot encoding, bag-of-words, and TF–IDF in terms of representation, interpretability, sparsity, and applications.
2. Explain how n-grams improve upon unigram models and discuss their computational limitations.
3. Describe how static embeddings learn semantic relationships and explain why they cannot fully resolve polysemy.
4. Explain the attention formula conceptually and identify the roles of queries, keys, values, and scaling.
5. Develop a framework for selecting a text representation for a regulated classification application.

Analytical / Application Questions

1. A fraud-detection dataset contains short, noisy messages with frequent spelling errors. Which combination of representations would you test first, and why?
2. A search engine must match both exact product codes and paraphrased user questions. Design a hybrid representation and explain how it would be evaluated.
3. A contextual model improves accuracy by 2 percentage points but triples latency and requires sending customer text to a third-party API. Recommend whether to adopt it, identifying the technical, business, privacy, and governance factors that should influence the decision.

Multiple Choice Questions with Answers

Question 1

Which representation assigns a single active position to identify a token?

A. TF–IDF

B. One-hot encoding

C. Contextual embedding

D. Character trigram

Correct Answer: B. One-hot encoding

Question 2

What information is ignored by a unigram bag-of-words model?

A. The presence of terms

B. The number of documents

C. Word order

D. Vocabulary membership

Correct Answer: C. Word order

Question 3

If a term occurs in every document, what is its basic inverse document frequency using idf(t) = log(D/df(t))?

A. 0

B. 1

C. D

D. Undefined because df(t) equals D

Correct Answer: A. 0

Question 4

Which representation is especially useful for handling spelling variation through subword patterns?

A. Character n-grams

B. One-hot document vectors

C. Binary class labels

D. Document IDs

Correct Answer: A. Character n-grams

Question 5

What is a key limitation of static word embeddings?

A. They cannot be stored as numerical vectors

B. They assign one vector to a word type across contexts

C. They always have one dimension

D. They contain only zero values

Correct Answer: B. They assign one vector to a word type across contexts

Question 6

For non-zero vectors, cosine similarity primarily measures what?

A. The difference in document length only

B. The angle or orientation between vectors

C. The number of vocabulary entries

D. The training time of a model

Correct Answer: B. The angle or orientation between vectors

Question 7

A document contains a term twice. In a collection of 100 documents, the term appears in 10 documents. Under raw TF and idf(t) = ln(D/df(t)), what is its approximate TF–IDF weight?

A. 0.230

B. 2.303

C. 4.606

D. 20.000

Correct Answer: C. 4.606

Question 8

Why can contextual embeddings represent the two meanings of “bank” more effectively than static embeddings?

A. They remove all punctuation

B. They use surrounding sequence information

C. They always use fewer dimensions

D. They count only document frequency

Correct Answer: B. They use surrounding sequence information

Question 9

Which is a strong reason to begin an NLP project with a TF–IDF baseline?

A. It guarantees perfect semantic understanding

B. It is interpretable and computationally efficient

C. It eliminates the need for evaluation

D. It cannot suffer from vocabulary limitations

Correct Answer: B. It is interpretable and computationally efficient

Question 10

A system must process millions of short documents with strict latency limits and exact keywords are important. Which initial choice is most appropriate?

A. A large contextual model for every document

B. TF–IDF with a linear classifier

C. Only a random vector for each document

D. A one-hot vector for the entire corpus without a vocabulary

Correct Answer: B. TF–IDF with a linear classifier