Applied NLP Systems

Introduction

Natural language processing (NLP) systems convert human language into representations that computers can analyse, search, generate, or act upon. In practical environments, success depends not only on choosing a powerful neural model but also on designing a reliable pipeline: defining the task, collecting representative data, preprocessing responsibly, selecting a model, evaluating it against suitable baselines, deploying it, and monitoring its behaviour.

Applied NLP includes sentiment analysis, document classification, information retrieval, question answering, summarisation, machine translation, conversational agents, and information extraction. These applications share components such as tokenisation, embeddings, attention, classification heads, retrieval indexes, sequence generation, and evaluation procedures, but each has different error costs and operational requirements.

This chapter presents a practical framework for developing NLP systems at diploma level. It emphasises model selection, data quality, evaluation, deployment, and responsible use.

Learning Objectives

By the end of this chapter, a learner should be able to:

• describe the architecture of a modern NLP application pipeline;
• select suitable models for classification, retrieval, generation, and extraction;
• build and evaluate systems for sentiment analysis and document classification;
• distinguish extractive from generative question answering and summarisation;
• explain the main stages of machine translation and conversational-agent design;
• identify information-extraction tasks and suitable output formats;
• calculate and interpret common NLP evaluation metrics;
• recognise risks involving bias, privacy, hallucination, security, and accountability.

1. From NLP Task to Production System

An NLP project begins with a precise task definition. “Understand customer feedback” is too vague. A usable specification might be: “Assign each English support message to one of six service categories within two seconds, with macro-F1 of at least 0.80 and a human-review route for low-confidence cases.” The specification identifies the input, output, users, performance target, latency, and escalation policy.

A typical applied pipeline is:

Data collection → annotation → splitting → preprocessing → representation → model training → evaluation → deployment → monitoring → improvement

Data must be divided into training, validation, and test sets. The training set fits model parameters; the validation set supports model and threshold selection; the test set is used once, or sparingly, for final estimation. Near-duplicate documents, messages from the same user, or documents from the same event should not be distributed across sets if this would create leakage.

Preprocessing may include Unicode normalisation, language identification, removal of corrupt records, sentence segmentation, and tokenisation. Aggressive removal of punctuation, stop words, or casing is not universally appropriate. For example, negation in “not useful” and punctuation in “Really?” may carry sentiment information. Modern subword tokenisers usually reduce the need for manually designed vocabulary rules.

2. Representing Text with Neural Models

A tokeniser maps text into token IDs. A neural model then maps these IDs into vectors. Contextual encoders, such as Transformer encoders, produce a representation whose meaning depends on surrounding words. In “bank approved the loan” and “bank of the river,” the representation of bank should differ because context differs.

The Transformer relies on self-attention. For a sequence representation matrix X, the usual attention operation is:

Here, Q, K, and V are query, key, and value matrices, dₖ is the dimensionality of each key vector, and softmax converts scores into non-negative weights that sum to one for each query. Scaling by √dₖ helps prevent excessively large dot products, which can make the softmax poorly behaved. This equation describes one attention head; practical Transformers use multiple heads and additional components such as residual connections, layer normalisation, and feed-forward networks.

Encoder models are commonly used for classification, ranking, and extraction. Decoder or encoder–decoder models are commonly used for text generation, translation, and summarisation. A pretrained model can be adapted through fine-tuning, in which task-specific data updates model parameters, or through prompting and retrieval-augmented generation, in which external information is supplied at inference time.

3. Sentiment Analysis and Document Classification

Sentiment analysis assigns a polarity or emotion label to a text. A binary system may distinguish positive and negative sentiment; a multiclass system may use positive, neutral, and negative labels; a multilabel system may identify emotions such as anger, fear, and satisfaction simultaneously.

Document classification assigns documents to categories such as topic, urgency, risk level, or department. The pipeline usually includes labelled examples, a text encoder, a classification layer, a loss function, and a decision threshold.

For a K-class classifier, the softmax probability for class k is:

where zₖ is the model logit for class k and K is the number of classes. For a labelled example with true class y, cross-entropy loss is:

The loss is small when the model assigns high probability to the correct class and large when it assigns low probability. These equations assume mutually exclusive classes. For multilabel classification, independent sigmoid outputs are generally used instead of one softmax distribution.

Accuracy is:

For a particular class, precision and recall are:

TP denotes true positives, FP false positives, and FN false negatives. The F1 score is the harmonic mean:

F1 is undefined when both precision and recall are zero unless a convention is specified. Macro-F1 averages class-level F1 scores equally, whereas weighted-F1 weights each class by its support. Macro-F1 is often more informative for imbalanced categories.

Worked example: classification metrics

Suppose a fraud-message detector produces TP = 36, FP = 9, and FN = 15. Then precision is 36/(36+9) = 0.80, while recall is 36/(36+15) ≈ 0.706. Therefore:

F1 = 2 × 0.80 × 0.706/(0.80+0.706) ≈ 0.75.

The detector identifies many fraudulent messages but misses approximately 29.4% of the fraud cases in this sample. If missing fraud is costly, the threshold may need adjustment, accepting more false positives to improve recall.

Practical design decisions

• Use stratified splitting when labels are imbalanced, while preventing user or time leakage.
• Inspect performance by language, product, demographic group, document length, and writing style.
• Calibrate probabilities if users will interpret them as confidence.
• Select thresholds according to business costs rather than automatically using 0.5.
• Use class weights, resampling, or focal objectives only after understanding the data distribution.
• Preserve an abstention or human-review option for uncertain cases.

4. Search and Information Retrieval

Search systems normally have two stages. The first stage retrieves a manageable candidate set quickly. The second stage reranks candidates using a more expensive model.

A classical lexical method is term frequency–inverse document frequency (TF–IDF). A common inverse document frequency definition is:

where N is the number of documents and df(t) is the number of documents containing term t. A term’s TF–IDF weight is TF(t,d) × IDF(t), where TF(t,d) measures the term’s frequency in document d. Exact variants differ in logarithm smoothing and term-frequency scaling.

Dense retrieval represents queries and documents as vectors. Similarity may be measured with cosine similarity:

where q and d are non-zero query and document vectors and ||·||₂ is the Euclidean norm. Approximate nearest-neighbour indexes make large-scale vector search practical. Hybrid retrieval combines lexical matching, which is strong for exact names and codes, with dense retrieval, which can capture semantic similarity.

Search evaluation should reflect ranked results. Precision at k measures the fraction of the first k results that are relevant. Recall at k measures the fraction of all relevant items retrieved in the first k, when the relevant set is known. Mean reciprocal rank focuses on the position of the first relevant result. Offline measures should be complemented by click behaviour, successful task completion, latency, and user feedback.

5. Question Answering

Question-answering (QA) systems may be extractive, generative, or retrieval-augmented.

An extractive QA system selects a span from a supplied passage. It is relatively easy to verify because the answer appears in the source, but it cannot normally answer questions whose wording or answer is absent from the passage. A generative system constructs an answer token by token and can synthesise information, but it may introduce unsupported claims.

Retrieval-augmented QA first retrieves documents, then provides selected passages to a reader or generator. A robust implementation should retain document identifiers, passage boundaries, retrieval scores, and citations. The generator should be instructed to state when evidence is insufficient rather than inventing an answer.

QA evaluation depends on the task. Exact match gives credit only when the normalised prediction exactly matches a reference. Token-level F1 measures overlap between predicted and reference answer tokens, but overlap does not guarantee factual correctness. For open-domain systems, retrieval recall and citation validity should also be evaluated.

A practical QA workflow is:

1. identify the user’s language and intent;
2. retrieve candidate passages;
3. rerank or filter passages;
4. generate or extract an answer;
5. check evidence, confidence, and permissions;
6. return an answer with sources or escalate.

6. Text Summarisation

Extractive summarisation selects sentences or phrases from a document. Abstractive summarisation generates new wording. Extractive methods tend to preserve source fidelity but may be repetitive or poorly structured. Abstractive methods are more fluent but can hallucinate names, numbers, dates, or causal relationships.

A summarisation system should be evaluated for at least three properties: content coverage, readability, and faithfulness. N-gram overlap measures such as ROUGE can compare generated and reference summaries, but they are imperfect proxies for meaning. Human or model-assisted factuality checks should test whether each important claim is supported by the source.

Long documents require chunking, hierarchical encoding, long-context models, or retrieval of relevant sections. Chunking must preserve headings and references where possible. A summary intended for executives should differ in length, terminology, and emphasis from one intended for technical investigators.

7. Machine Translation

Machine translation maps a source-language sequence to a target-language sequence. Modern neural systems commonly use an encoder–decoder Transformer. The encoder represents the source, while the decoder predicts target tokens conditioned on previous target tokens and the encoded source.

During training, teacher forcing supplies the correct previous target token. During inference, the model generates tokens sequentially until an end-of-sequence symbol or length limit is reached. Greedy decoding selects the most probable next token; beam search retains several partial hypotheses and may improve output, although it can also produce generic or repetitive text.

Translation quality depends on language pair, domain, terminology, sentence length, and available training data. Evaluation may use BLEU or chrF, but automatic scores do not fully measure adequacy, fluency, cultural appropriateness, or safety. Professional systems should use terminology glossaries, human review for high-risk content, and tests for names, numbers, negation, and formatting.

8. Conversational Agents

A conversational agent may contain intent classification, entity extraction, dialogue state tracking, policy selection, response generation, retrieval, and safety controls. Traditional task-oriented agents use explicit flows and are predictable. Generative agents are flexible but require stronger grounding and monitoring. Many production systems combine both approaches.

Dialogue state is a structured representation of what the system believes about the conversation, such as the requested service, account identifier, and missing fields. State should be updated only from supported evidence. The system should confirm high-impact actions, authenticate users before exposing private information, and provide a clear transfer route to a human.

Conversation quality is not measured solely by linguistic fluency. Useful measures include task completion, successful resolution, containment rate, fallback frequency, average turns, latency, user satisfaction, and unsafe-response rate. A fluent response that performs the wrong transaction is a system failure.

9. Information Extraction

Information extraction converts unstructured text into structured records. Named-entity recognition identifies entities such as people, organisations, locations, products, dates, and monetary values. Relation extraction identifies links such as company acquired company. Event extraction identifies an event, its type, and roles such as agent, object, time, and location.

A typical extraction pipeline is:

document → sentence and token boundaries → entity detection → entity linking or normalisation → relation/event extraction → schema validation → database or knowledge graph

The output schema should be defined before model training. For example, an invoice extractor may require supplier, invoice number, date, currency, tax amount, and total. Validation rules can detect impossible dates, totals inconsistent with line items, or currencies missing from monetary values.

Span-level extraction requires careful annotation guidelines. Annotators must decide whether to include titles, punctuation, nested entities, abbreviations, and discontinuous phrases. Evaluation should report entity-level precision, recall, and F1, using an explicitly defined matching rule such as exact span match or overlap match.

10. Illustrative Case: University Support Assistant

Background

A fictional university receives thousands of support messages each month about registration, fees, examinations, accommodation, and technical access. Students complain that answers are slow and inconsistent.

Problem

The university wants an assistant that classifies incoming messages, retrieves approved policy passages, answers routine questions with citations, and routes sensitive cases to staff. The system must not expose student records or make decisions about disciplinary matters.

Analysis

Historical messages contain personally identifiable information, inconsistent labels, duplicate cases, and outdated policy references. A random split would leak repeated conversations from the same student into testing. The project therefore uses student-level and time-aware splitting. Staff define a taxonomy and mark messages requiring mandatory human review.

Approach and solution

A lightweight classifier assigns one of eight intents and estimates confidence. A hybrid search engine retrieves policy documents using both lexical and dense representations. A reranker selects the best passages. A grounded generator produces an answer only from retrieved, currently approved documents and includes links to those documents. Requests involving grades, appeals, identity changes, financial hardship, or threats are routed to trained staff. Logs record the model version, retrieved sources, response, confidence, and escalation decision while excluding unnecessary personal data.

The test plan includes macro-F1 for intent classification, recall at 5 for retrieval, citation-supported answer rate, average latency, escalation recall, and fairness checks across language varieties and accessibility-related writing styles. Staff review a sample of conversations weekly, and outdated documents are removed from the retrieval index.

Outcome

In this illustrative case, the system is successful only if it reduces routine workload without creating unsupported or unauthorised answers. A high answer rate alone is not a valid success criterion. The university retains human control over consequential cases and treats monitoring as part of the system rather than a one-time final step.

Learning points

• A narrow, well-defined scope is safer than unrestricted automation.
• Retrieval quality and document governance directly affect generated answers.
• Privacy protection must be designed into data, prompts, logs, and access control.
• Escalation is a feature, not merely a failure.
• Evaluation must measure operational and social outcomes as well as model scores.

11. Challenges, Governance, and Responsible Practice

NLP data may contain personal information, copyrighted material, confidential business content, or socially biased language. Organisations should establish a lawful purpose, minimise collected data, control access, define retention periods, and provide procedures for correction or deletion where applicable. Sensitive information should not be sent to an external model service without an approved data-processing arrangement.

Bias can arise from unrepresentative data, historical labels, annotation disagreement, and unequal error costs. Report subgroup performance where legally and ethically appropriate, but do not assume that one aggregate fairness number solves the problem. Engage affected users and domain experts in defining harms and acceptable trade-offs.

Generative systems face prompt injection, data leakage, hallucination, toxic output, and overreliance. Useful controls include input filtering, retrieval access control, output validation, citation checking, rate limits, red-team testing, audit logs, and human approval for high-impact actions. System cards, data documentation, model versioning, and incident-response procedures improve accountability.

12. Best-Practice Development Checklist

1. Define the user, task, output, error costs, and escalation path.
2. Establish annotation rules and measure agreement on a sample.
3. Remove duplicates and prevent temporal, user, and document leakage.
4. Build a simple baseline before adopting a larger model.
5. Evaluate with task-appropriate metrics and subgroup analyses.
6. Test adversarial, ambiguous, multilingual, and out-of-domain inputs.
7. Ground generation in approved evidence when factual accuracy matters.
8. Record model, data, prompt, retrieval, and configuration versions.
9. Monitor drift, latency, cost, confidence, and user outcomes.
10. Maintain human oversight for high-impact decisions.

Chapter Summary

Applied NLP systems are engineered pipelines rather than isolated models. Classification systems require suitable labels, leakage-resistant splits, probability outputs, and metrics such as precision, recall, and F1. Search systems combine lexical and dense retrieval, while question-answering systems may extract, generate, or retrieve evidence before generating. Summarisation must be assessed for coverage, readability, and factuality. Translation requires attention to language pair, terminology, and human review. Conversational agents need dialogue state, policy controls, authentication, and escalation. Information extraction depends on a clear schema, consistent annotation, and validation.

Across all applications, data quality, evaluation design, governance, monitoring, and human oversight are as important as model architecture. A professionally deployable NLP system should be accurate enough for its purpose, transparent about uncertainty, secure with sensitive data, and designed to fail safely.

Key Terms

Tokenisation: Splitting text into units processed by a language model.

Embedding: A numerical vector representing linguistic or semantic information.

Transformer: A neural architecture based primarily on attention mechanisms.

Self-attention: A mechanism that weights relationships among tokens in a sequence.

Fine-tuning: Updating a pretrained model using task-specific data.

Sentiment analysis: Predicting polarity or emotion expressed in text.

Document classification: Assigning documents to predefined categories.

Precision: The proportion of predicted positives that are true positives.

Recall: The proportion of actual positives that are detected.

F1 score: The harmonic mean of precision and recall.

Information retrieval: Finding relevant items in a collection in response to a query.

Reranking: Reordering retrieved candidates with a more accurate model.

Question answering: Producing an answer to a natural-language question.

Retrieval-augmented generation: Generating responses using retrieved external evidence.

Extractive summarisation: Creating a summary by selecting source text.

Abstractive summarisation: Creating a summary with newly generated wording.

Dialogue state: Structured information representing the current conversation.

Named-entity recognition: Identifying and classifying entity mentions in text.

Hallucination: An unsupported or fabricated model output.

Data leakage: Unintended information transfer from evaluation data into training or development.

Review Questions

Short Answer Questions

1. Why should an NLP project define an escalation policy before deployment?
2. Distinguish precision from recall.
3. What is the purpose of a reranker in a search system?
4. State one difference between extractive and abstractive summarisation.
5. Why are citations useful in retrieval-augmented question answering?

Long Answer Questions

1. Explain the stages of an applied NLP pipeline from data collection to monitoring.
2. Compare encoder, decoder, and encoder–decoder models and give an application for each.
3. Describe how a university support assistant could protect privacy and prevent unsupported answers.
4. Explain the evaluation requirements for a document-classification system with imbalanced classes.
5. Discuss the major risks of deploying generative conversational agents and practical controls for each.

Analytical / Application Questions

1. A classifier has high accuracy but poor macro-F1. Analyse likely causes and propose an evaluation and improvement plan.
2. Design a hybrid search and question-answering system for an organisation whose policies change frequently. Specify indexing, retrieval, grounding, and monitoring procedures.
3. An extraction model performs well on general news but poorly on invoices. Analyse why domain transfer fails and propose a data, schema, validation, and deployment strategy.

Multiple Choice Questions with Answers

Question 1

Which metric is generally most informative when each class should receive equal importance despite class imbalance?

A. Macro-F1

B. Raw training loss only

C. Vocabulary size

D. Average document length

Correct Answer: A. Macro-F1

Question 2

In the attention equation, why is the dot product divided by √dₖ?

A. To remove all tokenisation errors

B. To control the scale of attention scores

C. To guarantee perfect translation

D. To convert logits directly into labels

Correct Answer: B. To control the scale of attention scores

Question 3

Which search approach is strongest for matching an exact product code?

A. Lexical retrieval

B. Random retrieval

C. Abstractive summarisation

D. Dialogue-state tracking

Correct Answer: A. Lexical retrieval

Question 4

What is a principal advantage of extractive question answering?

A. It always paraphrases the source creatively

B. Its answer can be directly located in the supplied passage

C. It requires no evidence

D. It cannot be evaluated

Correct Answer: B. Its answer can be directly located in the supplied passage

Question 5

A detector has TP = 40, FP = 10, and FN = 10. What is its precision?

A. 0.40

B. 0.50

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 6

Which component identifies what the system believes has already been established in a conversation?

A. Dialogue state tracking

B. TF–IDF weighting

C. Beam search

D. Sentence compression

Correct Answer: A. Dialogue state tracking

Question 7

What is the main purpose of schema validation in information extraction?

A. To make every document longer

B. To check whether extracted records satisfy structural or business rules

C. To replace annotation entirely

D. To eliminate the need for entity detection

Correct Answer: B. To check whether extracted records satisfy structural or business rules

Question 8

Which is an appropriate control for high-impact conversational actions?

A. Removing authentication

B. Automatic execution without confirmation

C. Human approval or confirmation before execution

D. Ignoring model uncertainty

Correct Answer: C. Human approval or confirmation before execution

Question 9

What does data leakage mean in NLP evaluation?

A. A model has too few parameters

B. Evaluation information unintentionally influences training or development

C. A document contains punctuation

D. A query contains synonyms

Correct Answer: B. Evaluation information unintentionally influences training or development

Question 10

Which property is especially important when evaluating a factual abstractive summary?

A. Citation or source faithfulness

B. Number of tokeniser vocabulary entries

C. Random seed alone

D. The colour of the user interface

Correct Answer: A. Citation or source faithfulness