Classical NLP and Baseline Models

Introduction

Natural language processing (NLP) enables computers to analyse, retrieve, classify and generate human language. Although modern systems often use neural networks and large language models, classical NLP remains essential. It provides interpretable methods, computationally efficient solutions and reliable baselines against which complex models can be evaluated.

Classical NLP combines linguistic knowledge with statistical and information-retrieval techniques. Typical tasks include assigning grammatical categories, analysing sentence structure, identifying entities, discovering topics, classifying documents and retrieving relevant information. These tasks are closely related: tokenisation supplies units for tagging; tagging supports parsing; parsing and entity recognition provide structured information; and representations derived from text support classification, topic analysis and retrieval.

A baseline is a simple, reproducible method that establishes a minimum level of performance. Without a baseline, it is difficult to determine whether a sophisticated model has learned useful patterns or merely added complexity.

Learning Objectives

By the end of this chapter, students should be able to:

• explain the linguistic foundations of classical NLP;
• preprocess text appropriately for different tasks;
• apply part-of-speech tagging, parsing and named-entity recognition concepts;
• represent documents using n-grams, term frequency and TF–IDF;
• construct simple topic-analysis, classification and information-retrieval systems;
• select suitable evaluation metrics and interpret errors;
• establish meaningful baselines and recognise their limitations;
• discuss reproducibility, fairness, privacy and other governance concerns.

1. Linguistic Foundations and Text Preparation

1.1 Levels of linguistic analysis

Language can be analysed at several levels:

These levels interact. For example, the word “record” may be a noun in “a record” and a verb in “to record.” Its syntactic context helps determine its meaning and grammatical category.

1.2 Tokenisation and normalisation

Tokenisation divides text into units such as words, punctuation marks, subwords or sentences. A whitespace tokenizer is simple but fails with contractions, URLs, emojis and languages that do not separate words by spaces. Rule-based tokenisers use punctuation and language-specific conventions; subword tokenisers divide unknown words into smaller pieces, although subword methods are more commonly associated with modern neural NLP.

Normalisation may include lowercasing, Unicode standardisation, removal of unwanted markup and conversion of repeated whitespace. It must be task-dependent. Lowercasing can improve matching for information retrieval, but it may destroy useful distinctions in named entities such as “Apple” and “apple.” Removing punctuation may harm sentiment analysis, where exclamation marks carry meaning.

1.3 Stemming and lemmatisation

Stemming applies crude rules to reduce related words to a common form. For example, “connected,” “connecting” and “connection” may be reduced to similar stems, though the result may not be a valid word. Lemmatisation uses vocabulary and grammatical information to produce a dictionary form, such as “studies” → “study.” Stemming is faster; lemmatisation is usually linguistically cleaner.

Stop-word removal eliminates frequent function words such as “the” or “of.” It can reduce dimensionality, but it is not universally appropriate. In sentiment, authorship and phrase-based retrieval, function words may be informative.

1.4 Ambiguity and context

NLP systems must address ambiguity. Lexical ambiguity occurs when one form has several meanings; syntactic ambiguity occurs when a sentence has multiple possible structures. In “The student saw the teacher with a telescope,” the telescope may belong to the student or the teacher. Statistical models resolve ambiguity by assigning probabilities to alternative analyses, while rule-based systems use explicit linguistic constraints.

2. Part-of-Speech Tagging

Part-of-speech (POS) tagging assigns a grammatical category to each token, such as noun, verb, adjective, determiner or preposition. Tagsets vary. The Penn Treebank tagset, for example, distinguishes singular nouns (NN), plural nouns (NNS), base-form verbs (VB) and past-tense verbs (VBD).

2.1 Tagging approaches

A rule-based tagger applies lexical and contextual rules, such as assigning “-ed” words to a past-tense category unless an exception applies. A unigram tagger assigns each word its most frequent tag in training data. It is simple but ignores context.

An n-gram tagger estimates a tag from neighbouring tags and words. A bigram model may estimate:

where t_i is the tag at position i, t_i-1 is the preceding tag, and w_i is the current word. A hidden Markov model (HMM) treats tags as hidden states and words as observations. Its sequence probability is commonly expressed as:

The model assumes that the current tag depends only on the previous tag and that the current word depends only on its tag. These are simplifying assumptions, not facts about language. The most probable tag sequence can be found using the Viterbi algorithm.

A conditional random field (CRF) directly models the conditional probability of a tag sequence given the observation sequence and can incorporate overlapping features. It avoids the HMM’s generative independence assumptions but requires feature design and efficient training.

2.2 Evaluation

POS tagging is commonly evaluated using token accuracy:

Accuracy can hide poor performance on rare tags. Per-class precision, recall and F1 score are useful when tag distributions are uneven. Errors often involve unknown words, proper nouns, abbreviations and context-dependent forms.

3. Syntactic Parsing

Parsing assigns a structural representation to a sentence. Constituency parsing represents phrases such as noun phrases and verb phrases. Dependency parsing represents directed relations between words, such as subject, object and modifier.

For example, in “The analyst reviewed the report,” a dependency structure identifies “analyst” as the subject of “reviewed” and “report” as its object. Dependency representations are useful for information extraction because they expose grammatical relations directly.

3.1 Constituency and dependency models

A context-free grammar (CFG) consists of non-terminals, terminals, production rules and a start symbol. A simplified grammar might include:

• S → NP VP
• NP → Det N
• VP → V NP

The grammar generates structures, but natural language usually requires probabilities because grammars may permit multiple parses. A probabilistic context-free grammar (PCFG) assigns a probability to each production. Under the usual PCFG assumptions, the probability of a parse tree is the product of the probabilities of the rules used to construct it.

Chart parsers such as the Cocke–Younger–Kasami algorithm use dynamic programming to avoid repeatedly solving identical subproblems. Dependency parsers may use transition-based decisions or graph-based optimisation.

3.2 Parsing evaluation and limitations

Constituency parsing is often assessed with labelled precision and recall over predicted constituents. Dependency parsing commonly uses unlabeled attachment score (UAS) and labeled attachment score (LAS). UAS measures whether a word has the correct head; LAS additionally requires the correct dependency relation.

Parsing is sensitive to domain, genre and annotation scheme. A parser trained on news articles may perform poorly on social media, technical documents or learner writing. Ambiguity, coordination and long-distance dependencies remain difficult for classical systems.

4. Named-Entity Recognition

Named-entity recognition (NER) identifies spans referring to entities and assigns types such as PERSON, ORGANISATION, LOCATION, DATE or MONEY. In “Dr. Amina Khan joined Orion Labs in Nairobi,” a system may identify “Amina Khan” as PERSON, “Orion Labs” as ORGANISATION and “Nairobi” as LOCATION.

NER is commonly represented with BIO labels:

B indicates the beginning of an entity, I indicates continuation and O indicates a non-entity token. BIO encoding distinguishes adjacent entities of the same type.

Rule-based NER uses gazetteers, regular expressions and contextual patterns. Statistical NER uses features such as word identity, prefixes, suffixes, capitalisation, neighbouring words and POS tags. HMMs, maximum-entropy models and CRFs have historically been important. A CRF is particularly suitable because it can model dependencies between neighbouring output labels and discourage invalid sequences.

NER evaluation should use entity-level precision, recall and F1 rather than only token accuracy. A predicted entity is correct only when its span and type satisfy the evaluation convention. Partial matches may be counted as incorrect in strict evaluation.

5. Text Representation for Classical Models

Machine-learning algorithms require numerical representations. The simplest representation is a document-term matrix, where rows represent documents and columns represent vocabulary terms. An entry may be a binary indicator, a raw count or a weighted value.

5.1 N-grams

An n-gram is a sequence of n consecutive tokens. Unigrams represent individual words, bigrams represent two-word sequences and trigrams represent three-word sequences. Bigrams can preserve limited word order, distinguishing “not good” from “good,” but increasing n causes vocabulary size and sparsity to grow rapidly.

5.2 TF–IDF weighting

Term frequency–inverse document frequency (TF–IDF) gives high weight to terms that are frequent in one document but uncommon in the collection. A common formulation is:

Here, tf(t,d) is the frequency of term t in document d, N is the number of documents and df(t) is the number of documents containing the term. The logarithm reduces the influence of very common terms. If a term occurs in every document, its unsmoothed inverse-document-frequency component is zero. In practice, smoothed variants such as log((1+N)/(1+df(t))) + 1 may be used to avoid undesirable edge cases.

TF–IDF does not understand synonymy, polysemy or word meaning. Nevertheless, it is often a strong and transparent baseline for classification and retrieval. Vectors are frequently normalised, for example using Euclidean norm:

where x_j is the value of feature j and m is the number of features.

6. Text Classification

Text classification assigns one or more labels to a document. In single-label classification, exactly one class is selected; in multilabel classification, several labels may apply. Applications include spam detection, sentiment analysis, intent detection and document routing.

6.1 Baseline classifiers

A majority-class classifier always predicts the most frequent training class. It is a necessary baseline but can be misleading for imbalanced data. A keyword or regular-expression baseline is useful when domain rules are transparent, such as detecting invoice numbers.

Naive Bayes is a probabilistic classifier based on the assumption that features are conditionally independent given the class. For a document represented by features x₁,...,x_m, it estimates:

where c is a class, P(c) is its prior probability and P(x_j|c) is the class-conditional probability of feature j. The proportionality sign means that a normalising constant is omitted when comparing classes. Laplace smoothing adds a positive count to prevent zero probabilities. Multinomial Naive Bayes is well suited to word counts; Bernoulli Naive Bayes uses binary feature presence.

Logistic regression estimates class probabilities using a linear decision function. For binary classification:

Here, x is the feature vector, w is a learned weight vector, b is an intercept and y is the binary label. The model is linear in the input features, although the sigmoid converts the score to a probability. Regularisation is commonly added to reduce overfitting. An L2 penalty discourages large weights; an L1 penalty can produce sparse feature selection.

Support vector machines (SVMs) seek a separating boundary with a large margin. Linear SVMs are effective for high-dimensional sparse text vectors. Decision trees and random forests can also be used, but they may be less competitive than linear models for very large sparse vocabularies.

6.2 Evaluation

For binary classification:

TP, FP and FN denote true positives, false positives and false negatives. Precision answers “Of predicted positives, how many were correct?” Recall answers “Of actual positives, how many were found?” F1 is the harmonic mean and is appropriate when both matter. Accuracy is suitable when class costs and frequencies are reasonably balanced. For multiclass problems, macro-F1 averages class-level F1 scores equally, while micro-F1 aggregates decisions and gives greater influence to common classes.

7. Topic Analysis

Topic analysis explores recurring semantic themes in a document collection. It is exploratory rather than necessarily predictive. A topic may be interpreted as a probability distribution over words.

Latent Dirichlet allocation (LDA) assumes that each document is a mixture of topics and each topic is a distribution over words. At a high level:

1. choose a topic mixture for each document;
2. choose a topic for each token from that mixture;
3. choose a word from the selected topic’s word distribution.

The Dirichlet parameters control how concentrated document-topic and topic-word distributions tend to be. LDA requires a chosen number of topics and often produces topics that require human interpretation. Topic coherence, representative documents and stability across runs are more informative than inspecting a few high-probability words alone.

Non-negative matrix factorisation (NMF) decomposes a non-negative document-term matrix into lower-dimensional non-negative matrices. Its additive representations are often easy to inspect, especially with TF–IDF input. Neither LDA nor NMF guarantees that discovered topics correspond to objectively real categories.

8. Information Retrieval

Information retrieval (IR) finds documents relevant to a user query. A retrieval system usually includes document collection, indexing, query processing, scoring and ranking.

The Boolean model uses exact logical operators such as AND, OR and NOT. It is interpretable but does not naturally rank partially matching documents. The vector-space model represents documents and queries as vectors. Cosine similarity is:

where q is the query vector and d is a document vector. The score is defined when neither vector is zero. With non-negative TF–IDF vectors, higher cosine similarity generally indicates greater overlap in weighted terms, not semantic equivalence.

The probabilistic BM25 ranking function improves on simple TF–IDF by incorporating term frequency saturation and document-length normalisation. One common form is:

Here, f(t,d) is the frequency of query term t in document d, |d| is document length, avgdl is average document length, and k₁ and b are tunable parameters, commonly selected near 1.2–2.0 and 0.75 respectively. IDF(t) is an inverse-document-frequency weight. Exact implementations differ in IDF smoothing, so the chosen definition must be documented.

IR evaluation requires relevance judgments. Precision at k measures the proportion of the first k retrieved documents that are relevant:

Recall measures the proportion of all relevant documents that were retrieved. Mean average precision rewards systems that place relevant documents early, while normalized discounted cumulative gain (nDCG) supports graded relevance and discounts lower-ranked results.

9. Establishing Meaningful Baselines

A credible baseline is not merely the easiest model. It should reflect the task, available information and operational objective. A recommended progression is:

1. Data baseline: report label frequencies, missing values, document lengths and duplicates.
2. Heuristic baseline: use majority class, keyword rules or exact matching where appropriate.
3. Sparse lexical baseline: use unigram or unigram–bigram TF–IDF with logistic regression, linear SVM or Naive Bayes.
4. Task-specific classical baseline: use HMM/CRF for sequence labelling, BM25 for retrieval or LDA/NMF for topic analysis.
5. Ablation baseline: remove preprocessing steps or feature groups to determine their contribution.

Data must be divided into training, validation and test sets without leakage. Near-duplicate documents should not appear across splits. For time-dependent applications, a chronological split may be more realistic than a random split. Hyperparameters must be selected using training and validation data; the test set should be used only for final assessment.

A baseline report should state the dataset, preprocessing, feature extraction, model, hyperparameters, random seeds, evaluation metrics and error categories. Confidence intervals or repeated runs are useful when datasets are small. The strongest baseline is often a simple model with careful preprocessing and honest evaluation.

10. Illustrative Case: Classifying Support Requests

Background

A fictional telecommunications provider receives customer messages and wants to route them to billing, technical support, account access or cancellation teams.

Problem

Messages are short, contain spelling variation and frequently mention multiple issues. Misrouting delays service, while a majority-class system may appear accurate because billing messages are most common.

Analysis

The data team first removes personally identifying information, assigns one primary routing label and examines class frequencies. It finds that account-access requests are less common but operationally urgent. Therefore, macro-F1 and recall for account access are reported alongside overall accuracy.

Approach

The team creates three baselines:

• a majority-class classifier;
• keyword rules, such as “password,” “login” and “locked” for account access;
• word and character n-gram TF–IDF features with regularised logistic regression.

Character n-grams help with misspellings and variations such as “passwrd.” The classifier is trained on a stratified training split, tuned on validation data and evaluated on a held-out test set. Personally identifying information is masked before feature extraction.

Outcome

The majority baseline establishes the minimum expected accuracy. The rules improve recall for obvious access problems but produce false positives when “login” appears in a historical explanation. The linear TF–IDF model provides the best balanced performance and is fast enough for deployment. Error analysis shows that cancellation and billing messages are confused when customers request refunds after cancellation.

Learning points

The case demonstrates that baseline selection must reflect consequences, not only accuracy. Error analysis suggests a useful next step: add structured metadata and a rule for urgent account lockouts, then evaluate whether the hybrid system improves recall without unacceptable false alarms. It also illustrates why anonymisation, reproducible splits and label-quality checks are part of NLP engineering rather than optional documentation.

11. Limitations, Ethics and Professional Practice

Classical NLP systems are often interpretable but still encode bias from data and annotation. A classifier trained on historical support decisions may reproduce unequal routing. NER systems can perform differently across names, dialects and languages. Topic models may impose misleading labels on communities or sensitive documents.

Privacy is important because text may contain names, health information, credentials or location data. Data minimisation, access control, retention limits and de-identification should be applied before modelling. De-identification itself is imperfect and should be audited.

Practitioners should document intended use, excluded uses, data provenance, annotation instructions and known failure modes. Human review is appropriate for high-impact decisions. Systems should not infer sensitive attributes merely because linguistic signals correlate with them. Evaluation should be stratified across relevant language varieties and user groups where lawful and ethically justified.

Classical methods also have technical limitations. Bag-of-words representations ignore much word order and meaning; linear models may fail on compositional language; HMM and PCFG independence assumptions are restrictive; topic models are unstable under preprocessing and topic-number choices; and retrieval systems can amplify popular or already visible information. These limitations motivate neural representations, but a neural model should improve upon a well-designed baseline under a fair comparison.

12. Contemporary Relevance

Pretrained language models have changed NLP practice, yet classical methods remain valuable in hybrid systems. BM25 is commonly combined with neural retrieval, rules support safety filters, and linear classifiers remain attractive for small datasets, strict latency requirements and transparent decisions. Classical tokenisation, annotation schemes and evaluation metrics continue to underpin modern datasets.

A practical development workflow is therefore iterative: define the task, establish a transparent baseline, inspect errors, improve data and features, compare more advanced models, and monitor performance after deployment. Complexity should be justified by measurable benefit, maintainability and risk reduction.

Chapter Summary

Classical NLP applies linguistic structure and statistical modelling to language tasks. Tokenisation, normalisation, stemming and lemmatisation prepare text, but each choice must match the task. POS taggers assign grammatical categories, while constituency and dependency parsers represent sentence structure. NER identifies typed entity spans, commonly using BIO labels and entity-level evaluation.

TF–IDF, n-grams and document-term matrices provide transparent sparse representations. Naive Bayes, logistic regression and linear SVMs are strong classification baselines. LDA and NMF support exploratory topic analysis, whereas Boolean retrieval, cosine similarity and BM25 support ranked search. Precision, recall, F1, accuracy, MAP and nDCG measure different aspects of performance.

Meaningful baselines require leakage-free data splitting, suitable metrics, reproducible preprocessing and error analysis. Ethical practice requires privacy protection, bias assessment, documentation and human oversight for consequential uses. Classical methods remain professionally relevant because they are efficient, interpretable and valuable for comparison with neural systems.

Key Terms

Tokenisation: Dividing text into processing units.

Normalisation: Standardising text forms for analysis.

Lemmatisation: Reducing a word to its dictionary form.

Part-of-speech tagging: Assigning grammatical categories to tokens.

Hidden Markov model: A probabilistic sequence model with hidden states and observed outputs.

Viterbi algorithm: Dynamic-programming method for finding the most probable HMM state sequence.

Parsing: Assigning syntactic structure to a sentence.

Constituency parsing: Representing nested phrases in a sentence.

Dependency parsing: Representing head–dependent grammatical relations.

Named-entity recognition: Detecting and typing entity mentions.

BIO encoding: Beginning–inside–outside representation for entity spans.

N-gram: A sequence of n consecutive tokens.

TF–IDF: A weighting scheme combining term frequency and inverse document frequency.

Naive Bayes: A probabilistic classifier using conditional-independence assumptions.

Logistic regression: A linear probabilistic classifier for categorical outcomes.

Topic model: A model that represents documents through latent thematic components.

BM25: A ranked retrieval function with term-frequency saturation and length normalisation.

Precision: The proportion of predicted positives that are correct.

Recall: The proportion of actual positives that are retrieved or identified.

Baseline: A reference method used to contextualise the performance of improved systems.

Review Questions

Short Answer Questions

1. What is the purpose of a baseline in an NLP experiment?
2. Distinguish stemming from lemmatisation.
3. What do B, I and O represent in BIO encoding?
4. Why can accuracy be misleading on an imbalanced classification task?
5. State one difference between constituency and dependency parsing.

Long Answer Questions

1. Explain the stages of a classical NLP pipeline and discuss why preprocessing should be task-dependent.
2. Compare HMM, CRF and rule-based approaches to sequence labelling.
3. Explain TF–IDF and discuss its strengths and limitations for text classification.
4. Compare cosine similarity and BM25 as information-retrieval methods.
5. Describe a rigorous experimental procedure for comparing a baseline classifier with a more advanced model.

Analytical / Application Questions

1. A support dataset contains 70% billing messages, 20% technical messages and 10% account-access messages. Explain why a majority classifier may have acceptable accuracy but unacceptable operational value. Recommend metrics and justify them.
2. Design a classical NER system for extracting product names and defect types from customer reviews. Specify preprocessing, features, label encoding, evaluation and two likely error sources.
3. A search engine returns long documents containing many repetitions of a query term above shorter, more focused documents. Explain how BM25 can address this problem and identify the parameters that influence the correction.

Multiple Choice Questions with Answers

Question 1

Which representation assigns greater weight to a term that is frequent in one document but rare across the collection?

A. One-hot encoding only

B. TF–IDF

C. A majority-class label

D. A stop-word list

Correct Answer: B. TF–IDF

Question 2

In BIO named-entity encoding, what does I-ORGANISATION indicate?

A. The first token of an organisation

B. A token outside every entity

C. A continuation token inside an organisation entity

D. An organisation that has been deleted

Correct Answer: C. A continuation token inside an organisation entity

Question 3

Which metric is the harmonic mean of precision and recall?

A. Accuracy

B. F1 score

C. Specificity

D. Mean squared error

Correct Answer: B. F1 score

Question 4

What is a central simplifying assumption of the basic Naive Bayes classifier?

A. All documents have equal length

B. Features are conditionally independent given the class

C. Every class has the same prior probability

D. Words occur in alphabetical order

Correct Answer: B. Features are conditionally independent given the class

Question 5

Which method is specifically designed to find the most probable hidden-state sequence in an HMM?

A. Viterbi algorithm

B. TF–IDF

C. Cosine normalisation

D. Lemmatization

Correct Answer: A. Viterbi algorithm

Question 6

A classifier predicts 30 positive cases, of which 24 are correct. What is its precision?

A. 0.20

B. 0.60

C. 0.80

D. 1.25

Correct Answer: C. 0.80

Question 7

Which retrieval method includes document-length normalisation and term-frequency saturation?

A. BM25

B. A unigram tagger

C. A CFG

D. BIO encoding

Correct Answer: A. BM25

Question 8

What is the main purpose of macro-F1 in a multiclass problem?

A. To give equal importance to each class’s F1 score

B. To remove all minority classes

C. To measure only training speed

D. To calculate document length

Correct Answer: A. To give equal importance to each class’s F1 score

Question 9

Which baseline always predicts the most frequent class?

A. Random forest baseline

B. Majority-class baseline

C. Dependency baseline

D. Topic-mixture baseline

Correct Answer: B. Majority-class baseline

Question 10

Why should the test set normally be used only at the end of model selection?

A. To prevent test information from influencing model choices

B. Because test data cannot contain labels

C. Because test documents must be longer

D. To guarantee perfect accuracy

Correct Answer: A. To prevent test information from influencing model choices