Foundations of NLP and Generative AI

Introduction

Natural Language Processing (NLP) is the area of artificial intelligence concerned with enabling computers to process, interpret, generate, and interact through human language. Language is not merely a sequence of words. It carries syntax, meaning, context, social intention, ambiguity, emotion, and cultural assumptions. Consequently, NLP combines ideas from computer science, linguistics, statistics, machine learning, information retrieval, and cognitive science.

Generative AI refers to systems that create new content—such as text, code, images, audio, or video—by learning patterns from data. Generative language systems are an important contemporary development in NLP. They can translate, summarise, answer questions, write software, extract information, and conduct dialogue. However, they do not possess human understanding in the philosophical sense, and their outputs may be inaccurate, biased, or unsafe.

This chapter establishes the conceptual foundation required for later study of language models, embeddings, sequence modelling, attention, transformers, prompt engineering, retrieval-augmented generation, and evaluation.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Define NLP, artificial intelligence, machine learning, deep learning, and generative AI.
2. Classify major NLP tasks and explain their practical uses.
3. Describe the historical development of NLP from rule-based systems to foundation models.
4. Explain the main components of a modern NLP system.
5. Distinguish rule-based, statistical, neural, and foundation-model approaches.
6. Explain the capabilities, limitations, risks, and governance requirements of generative AI.
7. Select an appropriate NLP approach for a practical organisational problem.

1. What NLP Attempts to Solve

Human language is expressed in many forms: speech, writing, sign, and multimodal communication. An NLP system may receive text as input, produce text as output, or connect language with other data types. Typical questions include:

• What does a sentence mean?
• Which entities, such as people or organisations, are mentioned?
• What information is being requested?
• Is a customer satisfied or dissatisfied?
• What is the most appropriate response?
• How can a document be translated or summarised?

The difficulty arises because language is variable and ambiguous. The sentence “The bank is near the river” contains a word, “bank”, whose meaning depends on context. The sentence “I saw the man with the telescope” has more than one possible syntactic interpretation. A system must also handle spelling variation, dialects, code-switching, slang, incomplete input, domain-specific terminology, and implied meaning.

NLP is therefore not one single task. It is a collection of related tasks operating at different linguistic levels.

Major NLP Tasks

These tasks can be combined. A customer-support assistant might detect the language, identify the user’s intent, retrieve a relevant policy, generate a response, and check the response for safety before displaying it.

2. Important Terminology

Corpus: A structured collection of language data used for analysis or model development.

Token: A unit processed by an NLP system. It may be a word, punctuation mark, character, or subword.

Vocabulary: The set of tokens known to a system.

Embedding: A numerical vector representing an item, such as a word, sentence, or document, so that relationships can be processed mathematically.

Language model: A model that assigns probabilities to sequences of tokens or predicts likely continuation tokens.

Parameter: A learned numerical value inside a model.

Training: The process of adjusting model parameters using data and an objective function.

Inference: Using a trained model to produce predictions or generated output.

Prompt: Instructions or context supplied to a generative model.

Context window: The amount of input information a model can consider in one processing instance.

Fine-tuning: Additional training on task-specific or domain-specific data.

Pre-training: Initial training on a broad dataset before adaptation to particular tasks.

Foundation model: A large, broadly trained model that can support many downstream tasks.

Hallucination: An output that is fluent but factually unsupported, incorrect, or invented.

Grounding: Linking an output to reliable evidence, data, tools, or retrieved documents.

Zero-shot learning: Performing a task without examples of that task in the prompt.

Few-shot learning: Guiding a model by providing a small number of examples in the prompt.

Multimodal model: A model that processes or generates more than one modality, such as text and images.

3. Historical Development of NLP

3.1 Rule-Based NLP

Early NLP systems relied heavily on manually written rules, dictionaries, grammars, and logical representations. A rule might state that a sentence beginning with “please” is likely to express a request, or that a word following “the” may be a noun phrase.

Rule-based systems are transparent and can be effective in narrow, stable environments. A compliance system, for example, may use carefully defined patterns to identify mandatory legal phrases. Their weaknesses include high development cost, poor adaptability, difficulty handling exceptions, and dependence on expert-maintained rules.

3.2 Statistical NLP

From the late twentieth century, researchers increasingly used probability and data rather than relying only on hand-written rules. A statistical language model estimates the likelihood of a sequence. For a token sequence w₁,w₂,…,w_T, the chain rule gives:

Here, P denotes probability, w_t is the token at position t, and T is the sequence length. The identity is general; practical models approximate the conditional probabilities because storing every possible history is infeasible. An n-gram model makes a limited-history approximation, such as P(w_t | w_t-2,w_t-1).

Statistical methods introduced measurable uncertainty and could learn from examples. Hidden Markov models were used for tagging and speech recognition, while probabilistic context-free grammars supported parsing. Conditional random fields became important for sequence labelling. Nevertheless, statistical systems often required substantial feature engineering and struggled with long-range dependencies and sparse data.

3.3 Neural NLP

Neural networks learn internal representations from data. Distributed representations, commonly called embeddings, represent words or other units as vectors. Similar linguistic or semantic items may occupy related regions of a vector space.

Recurrent neural networks process sequences step by step. Long short-term memory networks and gated recurrent units were designed to preserve information over longer sequences. Neural machine translation replaced many manually engineered translation pipelines with end-to-end learned systems. Convolutional networks were also applied to text classification and character-level processing.

Neural systems reduced the need for manual feature design but introduced new requirements: large datasets, specialised hardware, careful optimisation, and improved evaluation. Their internal decisions are generally less transparent than explicit rules.

3.4 Attention and Transformers

Attention allows a model to weigh different parts of an input when processing a particular token. In simplified scaled dot-product attention:

Q represents queries, K keys, and V values. d_k is the dimensionality of the key vectors. The transpose Kᵀ enables similarity calculations; division by √d_k helps control the scale of dot products; softmax converts scores into normalised weights. This formulation assumes compatible matrix dimensions and is used within transformer architectures.

Transformers process relationships between tokens efficiently using attention rather than relying exclusively on sequential recurrence. They became the dominant architecture for many language applications. Encoder-oriented transformers are often effective for understanding and classification, decoder-oriented transformers are commonly used for autoregressive generation, and encoder–decoder models are suited to transformations such as translation and summarisation.

3.5 Foundation Models and Generative AI

Large-scale pre-training produced models capable of supporting multiple tasks. A foundation model may learn general language patterns from diverse data and then be adapted through prompting, fine-tuning, instruction tuning, or connection to external tools.

A generative language model commonly predicts the next token repeatedly. At each step it produces a probability distribution over possible tokens. Decoding may select the highest-probability token, sample from a distribution, or use a constrained strategy. The generated sequence is therefore shaped by the model, the prompt, decoding settings, and any external context.

Foundation models have changed NLP from building one separate model for every task toward configuring a general model for many tasks. This improves flexibility, but it does not eliminate the need for data quality, evaluation, security, domain expertise, and human oversight.

4. Components of an NLP System

A practical NLP system usually contains several layers rather than only a model.

4.1 Input and Pre-processing

Input may arrive as typed text, scanned documents, speech transcripts, web pages, or application records. Pre-processing can include encoding conversion, removal of corrupted content, sentence segmentation, tokenisation, language identification, and document deduplication. Aggressive cleaning can be harmful: removing punctuation may destroy meaning, while lowercasing can erase distinctions in names or programming languages.

4.2 Representation

The system converts language into a machine-processable representation. Traditional systems use counts, binary indicators, or manually designed features. Modern systems often use dense embeddings. Tokenisation is a design choice: word-level tokens are intuitive but produce large vocabularies, whereas subword tokens handle rare words and novel compounds more effectively.

4.3 Model

The model may be a rules engine, classifier, probabilistic sequence model, neural network, transformer, or foundation model. Its suitability depends on the task, available data, latency requirements, explainability needs, and risk level.

4.4 Knowledge and Tools

A model may be connected to search, databases, calculators, workflow software, or retrieval systems. Retrieval-augmented generation first finds relevant external material and then supplies it to the generator. This can improve grounding and allow information to be updated without retraining the entire model.

4.5 Output Control and Evaluation

Production systems need validation, filtering, citation or evidence display where appropriate, access control, logging, monitoring, and escalation to a human. Evaluation should measure task performance and operational risk, not merely fluency.

5. Comparing Four Major Approaches

The approaches are not mutually exclusive. A modern system may use a foundation model for language understanding, a rule-based policy layer for safety, a statistical retrieval ranker, and a deterministic calculator for numerical answers.

Choosing an Approach

A rule-based approach is appropriate when requirements are narrow, rules are stable, and auditability is essential. Statistical or smaller neural models may be preferable for high-volume classification with labelled data and strict latency limits. A foundation model is attractive when tasks are diverse, labelled data are scarce, and natural-language interaction is important. The choice should be based on total system performance rather than model popularity.

6. Capabilities of Generative AI

Generative AI can:

• draft and transform text;
• summarise documents at different levels of detail;
• translate and localise content;
• generate software code and documentation;
• extract structured fields from unstructured text;
• simulate dialogue for training or service delivery;
• propose ideas, alternatives, and explanations;
• combine language with images, audio, or structured data in multimodal systems.

These capabilities are probabilistic. A fluent answer is not automatically a true answer. The model may follow an instruction incorrectly, overlook a qualification, repeat a bias in its data, or invent a source. High-stakes uses therefore require verification against authoritative evidence.

7. Practical Evaluation

Evaluation must match the task. Classification may use accuracy, precision, recall, and F1 score. For a binary class, precision is the proportion of predicted positive cases that are actually positive, while recall is the proportion of actual positive cases that are detected. F1 is the harmonic mean:

Here, P is precision and R is recall. The formula applies when both values are defined and is useful when a balance between false positives and false negatives matters.

Generation requires broader assessment: factuality, relevance, coherence, completeness, toxicity, citation correctness, latency, cost, and user satisfaction. Automated metrics can assist but cannot fully replace expert review. Evaluation data should represent real users, languages, dialects, edge cases, and adversarial inputs.

8. Illustrative Case: A University Student-Support Assistant

Background

A university receives thousands of questions about admissions, fees, examinations, accommodation, and academic regulations. Students need quick answers, but policies change and incorrect advice may have serious consequences.

Problem

A general-purpose chatbot produces fluent responses but occasionally invents deadlines or confuses undergraduate and postgraduate rules. Staff cannot manually review every interaction.

Analysis

The problem is not simply text generation. It requires intent identification, retrieval of current policy documents, access control, answer generation, evidence display, and escalation. A fully autonomous foundation model would create unacceptable governance risk.

Approach / Solution

The university designs a hybrid system:

1. A language detector and classifier identify the question category.
2. A retrieval component searches an approved, version-controlled policy collection.
3. A generative model drafts an answer using only retrieved passages.
4. The response displays document titles, dates, and links.
5. Rules block unsupported claims and prevent disclosure of private student data.
6. Low-confidence or high-risk questions are transferred to staff.
7. Logs are sampled for factuality, fairness, privacy, and policy compliance.

The system is tested on historical questions and newly written edge cases. Accuracy alone is insufficient; the university also measures citation correctness, escalation quality, response time, and the rate of unsupported answers.

Outcome

In this illustrative case, the hybrid design reduces routine workload while preserving human responsibility for exceptional and high-impact decisions. It is more maintainable than retraining a model whenever a policy changes because the authoritative knowledge is updated in the retrieval collection.

Learning Points

The case demonstrates that successful generative AI is a system-design problem. Retrieval, rules, identity management, monitoring, and human escalation may be as important as the language model itself. The most capable model is not necessarily the safest or most suitable model.

9. Challenges, Ethics, and Governance

Bias and Fairness

Training data may reflect historical discrimination or unequal representation. Outputs can differ in quality across languages, dialects, demographic groups, or cultural contexts. Organisations should test subgroup performance, document limitations, and avoid using systems for decisions whose consequences cannot be justified or appealed.

Privacy and Confidentiality

Sensitive prompts may contain health, financial, educational, or personal information. Data minimisation, encryption, retention limits, access control, contractual review, and careful configuration are essential. Users should know how their data are processed.

Intellectual Property and Attribution

Training data and generated outputs raise questions about copyright, licensing, ownership, and attribution. Organisations must examine applicable law and licence conditions rather than assuming that generated material is unrestricted.

Security

Prompt injection, data exfiltration, malicious documents, model abuse, and insecure tool use are significant threats. External content should not automatically override system instructions. Tools should use least-privilege access, validate inputs, and require confirmation for irreversible actions.

Reliability and Accountability

A model should not be treated as an authoritative person or independent decision-maker. Responsibility remains with the organisation and designated professionals. High-impact use requires documented purpose, risk assessment, testing, incident response, audit trails, and routes for correction or appeal.

Environmental and Economic Costs

Large-scale training and inference consume computational resources. Model selection should consider energy, cost, latency, and whether a smaller model or deterministic method is sufficient. Human work may be augmented, redesigned, or displaced; responsible deployment includes workforce consultation and reskilling.

10. Best Practices for NLP and Generative AI Projects

1. Define the task and acceptable error types before selecting a model.
2. Use representative, legally obtained, high-quality data.
3. Establish a baseline, including a simple rule-based or non-neural method.
4. Separate development, validation, and test data to avoid misleading results.
5. Keep authoritative knowledge outside the model when information changes frequently.
6. Ground important answers in retrieved evidence or verified tools.
7. Use structured outputs and validation where downstream software is involved.
8. Test multilingual, adversarial, ambiguous, and out-of-distribution inputs.
9. Protect confidential information and minimise data retention.
10. Monitor deployed systems continuously and provide human escalation.
11. Document model purpose, data sources, limitations, evaluation results, and ownership.
12. Prefer the least complex system that meets the requirements.

Contemporary Direction of the Field

Current NLP development is moving toward multimodal foundation models, smaller specialised models, retrieval and tool use, agentic workflows, synthetic data, parameter-efficient adaptation, and more rigorous safety evaluation. Research also focuses on factuality, interpretability, efficient inference, multilingual inclusion, and methods for measuring uncertainty.

An important conceptual shift is from viewing a model as a complete application to viewing it as one component in a governed sociotechnical system. A reliable application combines models with data pipelines, interfaces, policies, security controls, evaluation procedures, and accountable human roles.

Chapter Summary

NLP enables computers to analyse and generate human language across tasks including classification, extraction, translation, retrieval, summarisation, question answering, and dialogue. Its history has moved from manually written rules to statistical models, neural networks, transformers, and foundation models.

Rule-based systems offer transparency and precision in narrow domains. Statistical systems model uncertainty from data. Neural systems learn distributed representations and complex patterns. Foundation models provide broad, reusable capabilities through large-scale pre-training, prompting, fine-tuning, retrieval, and tool use. These approaches can be combined in a single production architecture.

Generative AI can create useful text, code, and multimodal content, but fluent output is not proof of truth. Hallucination, bias, privacy loss, security threats, intellectual-property concerns, cost, and accountability must be addressed through grounding, evaluation, monitoring, access controls, documentation, and human oversight. Effective NLP engineering begins with the problem and its risks, not with a fashionable model.

Key Terms

NLP: Artificial intelligence concerned with processing and generating human language.

Generative AI: AI that produces new content from learned patterns.

Corpus: A structured collection of language data.

Token: A unit processed by a language model.

Embedding: A numerical representation of language or another object.

Language model: A model that estimates or predicts token sequences.

Training: Adjusting model parameters using data and an objective.

Inference: Using a trained model to produce an output.

Prompt: Instructions or context given to a generative model.

Context window: The input capacity available in one model invocation.

Pre-training: Broad initial training before task adaptation.

Fine-tuning: Additional training for a specific task or domain.

Foundation model: A broadly trained model supporting many downstream uses.

Hallucination: Fluent but unsupported or incorrect generated content.

Grounding: Connecting output to reliable evidence or external data.

Transformer: A neural architecture based substantially on attention mechanisms.

Precision: The proportion of predicted positive cases that are truly positive.

Recall: The proportion of actual positive cases that are detected.

Retrieval-augmented generation: Generation supported by retrieved external information.

Multimodal model: A model that handles multiple data modalities.

Review Questions

Short Answer Questions

1. What is NLP, and why is human language difficult for computers to process?
2. Distinguish pre-training from fine-tuning.
3. What is a foundation model?
4. Define hallucination and grounding.
5. Name four major NLP tasks and give one application for each.

Long Answer Questions

1. Explain the historical development from rule-based NLP to foundation models.
2. Compare rule-based, statistical, neural, and foundation-model approaches.
3. Describe the components of a production NLP system.
4. Explain why a fluent generative-AI answer may still be unreliable.
5. Discuss the ethical, privacy, and security requirements for deploying an NLP application.

Analytical / Application Questions

1. A hospital wants to classify incoming messages, retrieve current clinical policies, and draft responses for staff review. Which components and safeguards would you recommend, and why?
2. An organisation has 50,000 labelled examples for a stable two-class document classification task and strict low-latency requirements. Compare a small specialised model with a foundation model for this use case.
3. Design an evaluation plan for a university policy assistant. Identify at least four performance or risk measures and explain why accuracy alone is insufficient.

Multiple Choice Questions with Answers

Question 1

Which NLP task identifies names of people, organisations, and locations in text?

A. Named-entity recognition

B. Text summarisation

C. Speech synthesis

D. Machine translation

Correct Answer: A. Named-entity recognition

Question 2

In an n-gram language model, the probability of the next token is approximated using:

A. All possible future documents

B. A limited preceding context

C. Only the document title

D. A manually written grammar exclusively

Correct Answer: B. A limited preceding context

Question 3

What is the main purpose of an embedding?

A. To encrypt a document

B. To represent an item numerically for computational processing

C. To guarantee factual output

D. To remove all ambiguity from language

Correct Answer: B. To represent an item numerically for computational processing

Question 4

Which approach is generally most transparent when its decisions are based on explicit human-written rules?

A. Foundation-model approach

B. Rule-based approach

C. Large-scale pre-training

D. Autoregressive sampling

Correct Answer: B. Rule-based approach

Question 5

In the attention expression Attention(Q,K,V) = softmax(QKᵀ / √d_k)V, what does d_k denote?

A. The number of documents

B. The key-vector dimensionality

C. The number of output classes

D. The sequence error rate

Correct Answer: B. The key-vector dimensionality

Question 6

Which statement best describes retrieval-augmented generation?

A. It removes the need for evaluation

B. It supplies retrieved external information to support generation

C. It replaces tokenisation with encryption

D. It guarantees that every generated claim is true

Correct Answer: B. It supplies retrieved external information to support generation

Question 7

If precision is 0.80 and recall is 0.50, what is the F1 score using F1 = 2PR/(P+R)?

A. 0.40

B. 0.615

C. 0.65

D. 1.30

Correct Answer: B. 0.615

Question 8

Which is a major limitation of generative language models?

A. They cannot process any sequence of tokens

B. They may generate fluent but unsupported information

C. They require no training data

D. They always reveal their reasoning accurately

Correct Answer: B. They may generate fluent but unsupported information

Question 9

Which use most clearly requires human escalation and strong governance?

A. Generating alternative email subject lines

B. Suggesting informal brainstorming ideas

C. Making an unreviewed decision about a person’s medical treatment

D. Correcting spelling in a private draft

Correct Answer: C. Making an unreviewed decision about a person’s medical treatment

Question 10

Why might a hybrid NLP system combine a foundation model with deterministic rules?

A. To combine broad language capability with controlled policy enforcement

B. To ensure that the model never needs input data

C. To eliminate the need for security testing

D. To make every output creative rather than factual

Correct Answer: A. To combine broad language capability with controlled policy enforcement