Supervised Text Classification

Introduction and Context

Supervised text classification is the task of assigning predefined categories to documents, messages, posts, reviews, conversations, or other units of language using examples whose correct labels are known. It is central to business analytics because many organisational decisions depend on converting unstructured language into structured signals. Examples include identifying spam, detecting customer intent, routing complaints, assessing risk disclosures, predicting churn-related messages, and categorising social-media content.

A supervised classifier learns a relationship between an input text and an output label from a labelled training set. After learning, it produces predictions for previously unseen texts. Although the computational objective is predictive, successful classification is not merely a modelling exercise. It also requires a defensible labelling scheme, representative data, leakage controls, suitable evaluation metrics, explainability, monitoring, and governance.

In business settings, the consequences of errors are asymmetric. Misclassifying a harmless email as spam may inconvenience a customer, whereas failing to identify a fraudulent message may create financial or regulatory exposure. Similarly, incorrectly routing a vulnerable customer’s complaint can cause reputational damage. Consequently, model selection must be guided by decision costs rather than accuracy alone.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the supervised text-classification problem and distinguish binary, multiclass, and multilabel classification.
2. Design reliable labelling schemes and data-partitioning strategies.
3. Construct classical text classifiers using sparse representations and understand their assumptions.
4. Explain the role of word embeddings, pretrained language models, and neural classifiers.
5. Select and interpret appropriate performance metrics under class imbalance.
6. Diagnose leakage, overfitting, concept drift, and annotation disagreement.
7. Apply explainability, fairness, privacy, and governance principles.
8. Evaluate a classification system in a realistic business context.

1. The Classification Problem

Let a dataset contain n labelled observations:

Here, xᵢ is a text document and yᵢ is its target label. For a K-class problem, yᵢ belongs to the set {1, 2, …, K}. A classifier is a function f that maps text to a predicted label:

where ŷ is the prediction and θ represents learned model parameters. In probabilistic classification, the model estimates class probabilities:

and commonly predicts the class with the largest estimated probability:

This rule is appropriate when all errors have equal cost. If the cost of a false negative is greater than that of a false positive, a different decision threshold may be preferable.

1.1 Types of classification

Binary classification assigns one of two labels, such as spam or legitimate.

Multiclass classification assigns exactly one of several mutually exclusive labels, such as billing, delivery, technical support, or cancellation.

Multilabel classification permits several labels for one text. A social-media post may simultaneously concern pricing, service quality, and privacy. Multilabel problems require separate decisions for each label, usually through label-specific thresholds.

Hierarchical classification uses categories organised into levels. A complaint may first be classified as “product” or “service” and then into a more specific subcategory.

Ordinal classification involves ordered categories, such as low, medium, and high severity. Treating ordinal labels as unrelated nominal classes can discard useful information.

1.2 The unit of analysis

A classifier requires a clearly defined prediction unit. It may be an email, a complete customer conversation, one sentence, a post, or a user over a time window. Ambiguous units create inconsistent labels and invalid evaluation. For example, if a complaint is split into sentences, one sentence may appear neutral even though the conversation is clearly dissatisfied. The unit should therefore correspond to the business decision being automated.

2. From Business Objective to Label Definition

The most consequential decisions often occur before model training. A label is not simply a column in a spreadsheet; it operationalises a concept. “Complaint,” “risk,” “intent,” and “churn signal” require explicit inclusion and exclusion criteria.

A useful label specification contains:

• the business purpose and decision supported;
• the unit of analysis;
• label definitions and examples;
• rules for ambiguous or mixed cases;
• the reference time available for prediction;
• the identity and training of annotators;
• a policy for missing, uncertain, or unclassifiable texts.

2.1 Annotation quality

Human annotation should be treated as a measurement process. Two annotators may disagree because a category is vague, because context is missing, or because the text genuinely supports multiple interpretations. Agreement can be assessed using Cohen’s kappa for two annotators and categorical labels:

where pₒ is observed agreement and pₑ is the agreement expected by chance under the selected marginal distributions. Kappa is most useful when the category design is clear; it can be misleading under extreme prevalence imbalance. For more than two annotators, Fleiss’ kappa or agreement statistics designed for multiple raters may be used. Agreement should not be treated as an absolute measure of truth: a group of annotators can consistently apply a flawed definition.

A robust workflow uses a written codebook, pilot annotation, adjudication of disagreements, periodic quality checks, and documentation of uncertainty. If labels are probabilistic or inherently subjective, retaining annotator disagreement may be more informative than forcing a single label.

2.2 Weak and noisy supervision

When manual labelling is expensive, organisations may use heuristics, keywords, customer outcomes, or existing business rules as imperfect labels. This is weak supervision. For example, a message followed by a refund may be treated as a complaint proxy. Such labels can be useful for initial modelling but may encode systematic bias and should be validated against a manually reviewed sample.

3. Data Preparation and Leakage Control

Text preparation should remove irrelevant variation without destroying predictive meaning. Common steps include Unicode normalisation, handling malformed characters, preserving or standardising case, tokenisation, and treatment of URLs, mentions, emojis, hashtags, punctuation, and negation.

Aggressive preprocessing can damage performance. Removing “not” from “not satisfied” reverses the meaning. Stemming may combine words that should remain distinct, while removing all punctuation may eliminate useful signals in code, addresses, or social-media expressions. Preprocessing should therefore be evaluated empirically and documented as part of the model pipeline.

3.1 Training, validation, and test sets

The training set estimates model parameters. The validation set supports hyperparameter selection and threshold tuning. The test set provides a final, untouched estimate of generalisation. In a simple random split, the data are divided into these sets, but random splitting is not always appropriate.

Messages from the same customer, campaign, document, or conversation should not be distributed across training and test sets when that would allow identity or near-duplicate information to cross the boundary. For temporal applications, training should use earlier observations and testing should use later observations. This better represents deployment and exposes concept drift.

Cross-validation estimates performance across several training-validation splits. In stratified K-fold cross-validation, each fold approximately preserves class proportions. Grouped or time-series cross-validation is preferable when observations are clustered or ordered in time.

3.2 Leakage

Leakage occurs when information unavailable at prediction time enters the features, labels, preprocessing, or evaluation procedure. Examples include:

• using a post-resolution field to predict the initial complaint category;
• fitting a vocabulary or feature selector on the complete dataset before splitting;
• allowing duplicate messages into both training and test sets;
• using a future customer outcome as a present-time feature;
• selecting the best model after repeatedly examining the test set.

Leakage often produces impressive but non-reproducible results. Every transformation, including vocabulary construction, scaling, dimensionality reduction, and feature selection, should be fitted only on the training data within a reproducible pipeline.

4. Representing Text as Features

Machine-learning algorithms require numerical representations. The choice of representation determines what kinds of linguistic patterns the model can detect.

4.1 Bag-of-words and n-grams

The bag-of-words representation records token counts while ignoring word order beyond the selected n-gram length. A unigram model uses individual words; a bigram model also includes adjacent pairs such as “credit card.” Word and character n-grams are both useful. Character n-grams can handle misspellings, morphological variation, usernames, and obfuscated spam.

A count vector records frequency. A binary vector records presence or absence. Term frequency–inverse document frequency (TF–IDF) reduces the weight of terms that occur in many documents. For term t in document d, a common formulation is:

where tf(t,d) is the term frequency, df(t) is the number of documents containing t, and N is the number of documents. The +1 adjustments avoid division by zero and are a smoothing convention. Other definitions of term frequency and inverse document frequency are also valid. TF–IDF is dimensionless and is especially effective with linear classifiers for many business-text tasks.

4.2 Classical supervised models

Multinomial Naïve Bayes models word counts conditional on the class and assumes conditional independence of features given the class. The assumption is unrealistic linguistically but often effective because the model is simple and data-efficient. Smoothing prevents unseen words from producing zero probabilities.

Logistic regression estimates class probabilities using a linear combination of features. For binary classification:

Here x is the feature vector, β is the coefficient vector, β₀ is the intercept, and e is the base of the natural logarithm. The model is linear in the features but nonlinear in the probability. Coefficients indicate the direction and strength of association on the log-odds scale, subject to feature correlation and preprocessing choices.

A regularised objective commonly minimises average log loss plus a penalty:

where pᵢ is the predicted probability for observation i, λ ≥ 0 controls regularisation strength, and R(β) is a penalty such as ∑ⱼ|βⱼ| for L1 regularisation or ∑ⱼβⱼ² for L2 regularisation. The intercept is commonly excluded from the penalty. Regularisation helps control overfitting, particularly in high-dimensional sparse text spaces.

Linear support vector machines seek a separating hyperplane with a margin and can perform strongly on TF–IDF features. Their raw scores are not automatically calibrated probabilities.

Tree ensembles and gradient-boosting methods can capture nonlinear interactions but may be less natural than linear models for very high-dimensional sparse word features. They can be useful when text features are combined with structured variables.

4.3 Neural representations

Neural models learn dense representations. Word embeddings map tokens to vectors in which semantic and syntactic relationships may be encoded. Convolutional neural networks can identify local patterns, while recurrent models process sequences. Transformer models use attention mechanisms to represent contextual relationships, so the meaning of a word can vary with surrounding text.

A pretrained language model may be used in two main ways:

1. Feature extraction: obtain a document representation and train a separate classifier.
2. Fine-tuning: update some or all pretrained parameters using the task-specific labelled data.

Fine-tuning can improve performance where sufficient, representative labels exist, but it requires careful validation, computational resources, monitoring, and attention to domain shift. A large language model is not automatically a reliable classifier; its outputs must be evaluated against a defined benchmark and decision policy.

Classical linear models remain valuable because they are fast, comparatively transparent, strong on modest datasets, and easy to retrain. A sound practice is to establish a TF–IDF plus linear baseline before adopting a more complex neural architecture.

5. Training, Tuning, and Decision Thresholds

Training minimises a loss function on labelled examples. Classification loss should be aligned with the operational goal. Log loss rewards calibrated probabilities and penalises confident wrong predictions. Hinge loss is associated with margin-based classifiers. Weighted losses can give greater importance to minority classes or costly errors.

Hyperparameters include n-gram range, vocabulary limits, regularisation strength, class weights, neural learning rate, batch size, number of layers, and decision threshold. They should be selected using training data and validation procedures, not by repeatedly optimising on the final test set.

The default threshold of 0.5 is not universally optimal. Suppose a fraud-screening model outputs a probability p of fraud. If a false negative costs C_FN and a false positive costs C_FP, a simplified expected-cost decision chooses the positive class when:

provided the probabilities are calibrated and the two-class cost structure is adequate. Rearranging gives the threshold:

This formula assumes only two outcomes, known costs, calibrated probabilities, and no additional operational constraints. In practice, capacity limits, review queues, customer harm, and regulatory requirements may require a more detailed policy.

6. Evaluation: Beyond Accuracy

For binary classification, define:

• True positive (TP): a positive case correctly identified.
• True negative (TN): a negative case correctly identified.
• False positive (FP): a negative case incorrectly labelled positive.
• False negative (FN): a positive case missed by the classifier.

Accuracy is:

It is appropriate only when class proportions and error costs make overall correctness meaningful.

Precision and recall are:

Precision answers, “Of the cases predicted positive, how many were positive?” Recall answers, “Of the actual positive cases, how many were found?” The F1 score is their harmonic mean:

The harmonic mean is used because a low value in either component limits the result. If both precision and recall are zero, F1 is conventionally reported as zero, with the convention documented.

Specificity measures the true-negative rate:

Balanced accuracy is the mean of sensitivity (recall) and specificity:

For varying thresholds, the receiver operating characteristic curve plots true-positive rate against false-positive rate. The area under this curve (ROC-AUC) measures ranking ability across thresholds, but can appear favourable under severe imbalance. The precision–recall curve is often more informative when positive cases are rare.

Macro-averaging calculates a metric separately for each class and then gives every class equal weight. Micro-averaging aggregates decisions before calculating the metric and therefore gives more influence to frequent classes. Weighted averaging weights each class by its support. Reports should state which averaging method is used.

6.1 Calibration and operational evaluation

A calibrated probability estimate means that among cases assigned probability approximately 0.7, about 70% are positive, under comparable conditions. Calibration can be assessed with reliability diagrams, calibration error measures, or a held-out calibration set. Ranking performance and probability calibration are different properties.

Evaluation should include confidence intervals or variation across folds where feasible, subgroup performance, error examples, latency, throughput, review workload, and cost per decision. A model with a slightly lower F1 score may be preferable if it is more stable, interpretable, and inexpensive to operate.

7. Imbalanced Data and Rare Events

Business text datasets often contain many ordinary messages and few high-risk, fraudulent, or severe cases. Imbalance does not automatically make a dataset defective, but it makes naive evaluation and training unreliable.

Useful responses include:

• stratified or prevalence-aware sampling;
• class-weighted loss;
• threshold adjustment;
• targeted collection of minority examples;
• cautious oversampling within training folds only;
• hard-negative mining;
• precision–recall analysis;
• separate human review for uncertain cases.

Synthetic oversampling can create unrealistic language and should not be applied before data splitting. Undersampling may discard informative negative examples. Class weights change the training objective but do not necessarily produce calibrated probabilities, so calibration should be reassessed.

8. Explainability, Fairness, and Governance

Explainability serves several audiences: analysts debugging a model, managers validating decisions, customers requesting reasons, and regulators assessing accountability. For linear text models, feature coefficients and document-level contributions can be comparatively direct. Perturbation methods such as LIME and additive attribution methods such as SHAP can provide local explanations, but explanations are approximations and should not be confused with causal evidence.

A useful explanation should identify relevant text spans, state the predicted class and confidence, disclose important limitations, and distinguish model association from factual causation. Explanations should be tested for stability: if tiny irrelevant changes produce completely different rationales, trust is weakened.

Fairness concerns arise when language reflects protected characteristics, dialects, identity terms, or unequal exposure to moderation and service systems. A model may have different false-positive or false-negative rates across groups. There is no single fairness metric that is universally correct; metrics may conflict, especially when group base rates differ. Organisations should define the harm they seek to prevent, measure subgroup performance where legally and ethically appropriate, and include affected stakeholders in governance.

Privacy controls include data minimisation, access restrictions, retention limits, removal of unnecessary personal identifiers, secure annotation environments, and careful handling of sensitive text. Social-media data may be publicly visible but still ethically sensitive. Terms of service, consent expectations, applicable privacy law, and the risk of re-identification must be considered.

High-impact classification should include human oversight, appeal or correction mechanisms, audit logs, version control, model cards or equivalent documentation, and a clear owner for incidents. Automation should not be used to evade responsibility.

9. A Practical End-to-End Workflow

A defensible project commonly follows this sequence:

1. Define the business decision, prediction time, unit of analysis, and error costs.
2. Develop and pilot the label codebook.
3. Collect, de-identify, quality-check, and document data.
4. Partition data using stratified, grouped, or temporal methods as appropriate.
5. Establish a simple baseline, such as a majority classifier and TF–IDF logistic regression.
6. Build preprocessing and modelling steps in one reproducible pipeline.
7. Tune hyperparameters using validation or nested cross-validation.
8. Compare models using metrics aligned with the use case.
9. Inspect errors, subgroup results, calibration, and explanations.
10. Select a threshold and operating policy, including human review rules.
11. Test on an untouched holdout set.
12. Deploy with monitoring for drift, data quality, latency, calibration, and fairness.
13. Retrain or revise labels when the business environment changes.

10. Illustrative Case: Customer Complaint Routing

Background

A telecommunications provider receives 80,000 monthly customer messages through email, web chat, and social media. The service operation wants to route messages into five categories: billing, network reliability, device support, cancellation, and general information. Faster routing is expected to reduce resolution time.

Problem

Historical labels were created by different teams. Some messages were labelled after the issue had been resolved, and a few fields included the final resolution code. The cancellation class represented only 3% of messages, but missing a cancellation signal could increase customer churn.

Analysis

The analytics team first defines the prediction point as the time the message is received. Resolution codes and later agent actions are therefore excluded. Duplicate messages from the same conversation are grouped into a single partition. A manually reviewed sample reveals that “I want to know how to cancel” and “I have cancelled” were inconsistently labelled, so the codebook distinguishes information-seeking intent from completed cancellation.

A temporal split is used: the first nine months for training and validation, and the following three months for testing. A TF–IDF logistic-regression model provides a transparent baseline. Character n-grams are added because customers use spelling variations and abbreviated device names. A pretrained transformer is evaluated as a challenger, but its additional performance is compared with its inference cost and explanation requirements.

Approach and solution

The provider uses class-weighted training for the rare cancellation category and tunes a separate review threshold. Messages with high cancellation probability are routed to a specialist queue, while uncertain messages are sent to human triage. Performance is reported per class, not only as overall accuracy. Calibration is checked because probabilities determine queue priority.

Error analysis shows that messages containing “charge” are sometimes routed to billing even when the real issue is unauthorised payment fraud. The label taxonomy is expanded to separate fraud-related billing cases, and a small targeted annotation campaign collects additional examples. This demonstrates that model improvement may require better labels and category design, not merely algorithmic tuning.

Outcome

In the illustrative scenario, the final system improves macro-F1 and reduces average routing time, while the specialist queue accepts a controlled increase in manual review. The provider retains a human override, records model versions, monitors monthly drift, and reviews subgroup error rates. The result is not a fully autonomous decision-maker but a governed decision-support system.

Learning points

The case illustrates five principles: define labels before modelling; avoid post-outcome leakage; evaluate rare classes explicitly; combine automated prediction with human review; and treat deployment as an ongoing measurement and governance process.

11. Contemporary Developments and Critical Issues

Large language models have expanded the range of possible text-classification strategies, including zero-shot and few-shot prompting, instruction tuning, retrieval-supported classification, and automated explanation generation. These methods can be useful when labelled data are scarce, but prompt outputs may be unstable, costly, difficult to calibrate, or sensitive to wording. They can also reproduce training-data biases and disclose sensitive content.

For high-stakes business use, prompted classification should be benchmarked against supervised baselines, tested across time and subgroups, and constrained by structured output validation. Human review remains essential where errors have material consequences. Smaller fine-tuned models may be preferable when latency, privacy, cost, and reproducibility matter.

Domain adaptation is important because language changes across industries, regions, platforms, and time. A model trained on formal email may fail on short social-media posts. Monitoring should detect changes in vocabulary, class prevalence, confidence distributions, and error rates. Drift detection is a signal for investigation, not proof that the model has failed.

Best-Practice Checklist

• Start with an explicit decision and label codebook.
• Establish a strong, interpretable baseline.
• Split data according to time, groups, and duplication structure.
• Fit preprocessing only on training data.
• Report class-wise metrics and the confusion matrix.
• Use precision–recall analysis for rare positive classes.
• Tune thresholds according to costs and operational capacity.
• Inspect errors and disagreement, not only aggregate scores.
• Assess calibration, fairness, privacy, and explainability.
• Document data, model, assumptions, limitations, and version history.
• Monitor the deployed system and provide human escalation.

Chapter Summary

Supervised text classification converts labelled language data into predictions that support business decisions. Its success depends on more than selecting an algorithm. Reliable systems require valid labels, an appropriate prediction unit, leakage-resistant data partitioning, and representations suited to the domain. TF–IDF with linear models remains an important baseline, while embeddings and transformer models can capture richer contextual information at greater computational and governance cost.

Accuracy is insufficient for imbalanced or high-stakes applications. Precision, recall, F1, specificity, balanced accuracy, ranking metrics, calibration, subgroup analysis, and operational cost should be selected according to the decision context. Thresholds should reflect error costs and capacity. Explainability, privacy, fairness, human oversight, documentation, and monitoring are integral to responsible deployment. The central professional principle is to evaluate the entire decision system—labels, data, model, threshold, people, and consequences—rather than treating the classifier as an isolated technical object.

Key Terms

Supervised learning: Learning from examples paired with known target labels.

Text classification: Assigning one or more predefined categories to text.

Binary classification: Classification involving two possible classes.

Multiclass classification: Classification in which exactly one of several classes is selected.

Multilabel classification: Classification in which several labels may apply to one text.

Annotation: Human or rule-based assignment of labels to data.

Label leakage: Use of information unavailable at prediction time in the target or features.

TF–IDF: A text representation weighting terms by local frequency and inverse document frequency.

N-gram: A contiguous sequence of n tokens or characters.

Regularisation: A method for controlling model complexity by penalising parameters.

Class imbalance: Unequal prevalence of target classes.

Precision: The proportion of predicted positives that are truly positive.

Recall: The proportion of actual positives that are identified.

F1 score: The harmonic mean of precision and recall.

Calibration: Agreement between predicted probabilities and observed frequencies.

Threshold: A decision boundary used to convert scores or probabilities into classes.

Concept drift: Change over time in language, relationships, prevalence, or task conditions.

Explainability: Methods for communicating how a model produces or supports a prediction.

Weak supervision: Labelling through imperfect rules, proxies, or indirect signals.

Review Questions

Short Answer Questions

1. Distinguish binary, multiclass, and multilabel text classification.
2. Why can accuracy be misleading for a rare-event classification problem?
3. Define data leakage and provide two examples in text analytics.
4. What is the difference between precision and recall?
5. Why should a TF–IDF linear model usually be considered before a complex neural model?

Long Answer Questions

1. Explain how a business analyst should design and validate a labelling scheme for customer-intent classification.
2. Compare multinomial Naïve Bayes, logistic regression, linear support vector machines, and transformer-based classifiers.
3. Discuss how temporal splitting, grouped splitting, and ordinary random splitting differ in their validity.
4. Explain the role of threshold selection, class weighting, and calibration in an imbalanced classification system.
5. Evaluate the ethical, privacy, fairness, and governance requirements for automated complaint classification.

Analytical / Application Questions

1. A fraud classifier identifies 90 of 100 fraudulent messages and incorrectly flags 180 legitimate messages. Calculate recall and precision, and explain which operational concern each metric represents.
2. Design an evaluation plan for a churn-signal classifier trained on social-media posts when language and customer behaviour change over time. Specify the split strategy, metrics, monitoring signals, and retraining triggers.
3. A complaint model achieves high test accuracy but performs poorly after deployment. Diagnose at least four possible causes and propose an investigation for each.

Multiple Choice Questions with Answers

Question 1

Which problem is an example of multilabel text classification?

A. Assigning an email to either spam or legitimate

B. Assigning a message to exactly one of five departments

C. Assigning a post the labels pricing, privacy, and service quality

D. Ranking messages by predicted probability

Correct Answer: C. Assigning a post the labels pricing, privacy, and service quality

Question 2

In a leakage-resistant pipeline, when should the vocabulary for TF–IDF features be fitted?

A. On the complete dataset before splitting

B. Only on the training data within the modelling pipeline

C. Only on the final test set

D. Separately on every production message

Correct Answer: B. Only on the training data within the modelling pipeline

Question 3

Which metric answers the question, “Of the cases predicted positive, how many were actually positive?”

A. Recall

B. Specificity

C. Precision

D. Accuracy

Correct Answer: C. Precision

Question 4

A classifier has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 5

What is a principal limitation of a default probability threshold of 0.5?

A. It always produces calibrated probabilities

B. It ignores differences in error costs and operational constraints

C. It can only be used with neural networks

D. It prevents the calculation of recall

Correct Answer: B. It ignores differences in error costs and operational constraints

Question 6

Which representation is generally most suitable for capturing misspellings and character-level variation in social-media text?

A. Character n-grams

B. A majority-class label

C. A confusion matrix

D. A one-row aggregate table

Correct Answer: A. Character n-grams

Question 7

Why is a temporal test split often appropriate for deployed business text classification?

A. It guarantees perfect accuracy

B. It makes labels unnecessary

C. It simulates predicting future observations from past data

D. It eliminates class imbalance

Correct Answer: C. It simulates predicting future observations from past data

Question 8

What does calibration assess?

A. Whether model probabilities correspond to observed event frequencies

B. Whether every class has equal prevalence

C. Whether the vocabulary contains only nouns

D. Whether the test set is larger than the training set

Correct Answer: A. Whether model probabilities correspond to observed event frequencies

Question 9

Which action is most appropriate when a rare positive class is important and missed cases are costly?

A. Report accuracy only

B. Ignore the confusion matrix

C. Consider recall, precision–recall analysis, class weighting, and threshold adjustment

D. Remove all negative examples

Correct Answer: C. Consider recall, precision–recall analysis, class weighting, and threshold adjustment

Question 10

Why should a complex transformer classifier be compared with a TF–IDF linear baseline?

A. The baseline provides a transparent performance and cost reference

B. Linear models always outperform transformers

C. Transformers cannot classify text

D. The baseline removes the need for test data

Correct Answer: A. The baseline provides a transparent performance and cost reference