Natural Language, Vision, and Generative AI

Introduction

Artificial intelligence increasingly works with information that humans communicate through language, images, sound, and combinations of these forms. Natural-language systems process text and speech; computer-vision systems interpret images and video; generative-AI systems produce new text, images, audio, code, and synthetic combinations of media. These capabilities support search, translation, accessibility, customer service, medical analysis, education, design, and software development.

However, modern AI is not a human mind. A model identifies statistical patterns in data and produces outputs according to its training and design. It may generate fluent but false statements, misinterpret an image, reproduce bias, or expose sensitive information. Professional use therefore requires a complete workflow: define the task, collect and prepare data, select an appropriate model, evaluate it, deploy it responsibly, and monitor its behaviour.

This chapter introduces practical foundations for text, image, speech, and multimodal AI. It explains embeddings, transformers, large language models, prompting, retrieval-augmented generation (RAG), hallucinations, and introductory generative-AI applications.

Learning Objectives

By the end of this chapter, you should be able to:

1. Describe practical workflows for text, image, speech, and multimodal data.
2. Explain tokens, embeddings, attention, transformers, and large language models.
3. Distinguish classification, extraction, retrieval, generation, and synthesis tasks.
4. Design effective prompts and a basic retrieval-augmented generation workflow.
5. Identify hallucinations, bias, privacy risks, copyright concerns, and security threats.
6. Select suitable evaluation methods and apply human oversight to AI systems.

1. Data Modalities and AI Tasks

A modality is a type of information, such as text, an image, audio, or video. A practical AI solution normally begins by identifying the input modality, desired output, users, risks, and success criteria.

Text

Text tasks include sentiment classification, topic classification, named-entity recognition, translation, summarisation, question answering, information extraction, and generation. A text workflow commonly includes:

1. Collect documents and define permitted uses.
2. Remove duplicates, corrupted records, and unnecessary personal information.
3. Detect language and encode text consistently.
4. Split data into training, validation, and test sets when training a model.
5. Tokenise text and represent it numerically.
6. Train or select a model.
7. Evaluate both accuracy and practical usefulness.
8. Deploy with logging, access control, and monitoring.

For example, a support-ticket classifier may predict categories such as billing, account access, or technical fault. A generative assistant may instead draft a response, but a human may still need to approve it.

Images and Video

Computer vision uses pixels as input. Typical tasks are image classification, object detection, semantic segmentation, instance segmentation, optical character recognition (OCR), image captioning, and image generation.

A vision workflow includes image acquisition, consent and provenance checks, resizing or cropping, colour and format standardisation, annotation, augmentation where appropriate, model training or selection, and evaluation. Object detection differs from classification: classification assigns a label to an image, whereas detection identifies object locations, often using bounding boxes. Segmentation assigns labels to individual pixels or regions.

Important risks include poor lighting, camera changes, demographic imbalance, adversarial manipulation, and the possibility that a visually plausible model prediction is unsafe in a high-stakes setting.

Speech and Audio

Speech systems commonly perform automatic speech recognition (ASR), speaker identification, speech synthesis, keyword detection, translation, and emotion or event classification. An ASR workflow may include recording, noise reduction, voice-activity detection, feature or representation extraction, transcription, punctuation restoration, and human review.

The word error rate is a common ASR metric:

Here, S is the number of substitutions, D the number of deletions, I the number of insertions, and N the number of words in the reference transcription. The metric assumes a reference transcript and is most meaningful when the evaluation set represents real operating conditions. Lower WER is generally better, although errors in names or safety-critical terms may matter more than errors in common words.

Multimodal Data

Multimodal systems accept or produce more than one modality, such as an image and a question, a spoken request and a map, or a document containing text, tables, and diagrams. A document assistant may use OCR, layout analysis, text embeddings, image representations, and a language model.

Multimodal pipelines are harder to evaluate because errors can occur at several stages. A system may read the text correctly but misunderstand a chart, or retrieve the correct image but generate an inaccurate explanation. Evaluation should therefore test each component and the complete user task.

2. Representing Meaning: Tokens and Embeddings

Computers do not directly process meaning in the human sense. They process numerical representations. Tokenisation divides text into units such as words, subwords, punctuation marks, or characters. Subword tokenisation is useful because it can represent unfamiliar words through smaller units, although token counts vary across languages and tokenisers.

An embedding is a dense numerical vector representing an item such as a word, sentence, document, image, or audio segment. Items with similar use or content often have nearby vectors. Embeddings are used for semantic search, clustering, recommendation, duplicate detection, classification, and retrieval-augmented generation.

Similarity is often measured with cosine similarity:

Here, u and v are vectors, u · v is their dot product, and ||u|| and ||v|| are their Euclidean lengths. The formula is applicable when both vectors are non-zero. It measures orientation rather than magnitude and ranges from −1 to 1 in the usual real-valued setting. In many embedding systems, larger cosine similarity indicates greater semantic similarity, but the appropriate interpretation depends on how the model was trained.

For example, suppose u = (1, 0) and v = (0.8, 0.6). Their lengths are 1 and 1, and their dot product is 0.8. Thus, cosine similarity is 0.8. This suggests a strong directional similarity, but it does not prove that the two texts are factually equivalent.

Embeddings do not guarantee truth, fairness, or causal understanding. They can encode stereotypes present in training data, and similarity may reflect writing style, topic, or superficial associations rather than the relationship a user intends.

3. Transformers and Large Language Models

Earlier language systems often processed text sequentially with recurrent architectures. Transformers introduced a more parallel and flexible approach based on attention. In self-attention, each token can assign different importance to other tokens in the context. This helps a model relate words that are far apart and represent context-dependent meaning.

A simplified scaled dot-product attention operation is:

Q contains queries, K contains keys, V contains values, and dₖ is the dimensionality of the key vectors. The softmax operation converts scores into non-negative weights that sum to one for each query. Scaling by √dₖ helps control the size of dot products and supports stable learning. This equation describes an attention mechanism; a transformer also includes positional information, multiple attention heads, feed-forward layers, residual connections, and normalisation.

A large language model (LLM) is a model with many parameters trained on large quantities of text, usually with a language-modelling objective. In autoregressive generation, the model estimates a probability distribution for the next token given previous tokens:

Here, xₜ is the next token and x₁ through xₜ₋₁ are preceding tokens. During generation, the system repeatedly selects or samples a token and adds it to the context. The model is not necessarily retrieving a verified sentence from a database; it is producing a likely continuation according to learned patterns and the current context.

Training commonly has several stages. Pre-training develops broad language patterns. Instruction tuning teaches the model to respond to requests. Preference optimisation or human feedback may improve helpfulness and safety. Deployment may add system instructions, tools, filters, retrieval, and monitoring. Each stage can introduce strengths and failure modes.

An LLM may have a large context window, but more context does not automatically mean better reasoning. Irrelevant, contradictory, or poorly structured information can reduce performance. Models may also show sensitivity to wording and position of information.

4. Prompting as Task Specification

A prompt is an instruction and associated context supplied to a model. Effective prompting is closer to specifying a task than to asking a vague question. A useful prompt identifies:

• the role or perspective required;
• the task and intended audience;
• relevant context and source boundaries;
• constraints such as length, format, tone, or prohibited claims;
• examples when a format is difficult to describe; and
• a verification or uncertainty requirement.

For instance, “Summarise this policy” is underspecified. A stronger instruction is: “Summarise the supplied policy for new employees in five bullet points. Use only the policy text, distinguish mandatory rules from recommendations, and quote section numbers for each important claim.”

Prompting techniques include zero-shot prompting, few-shot prompting, decomposition into subtasks, structured output, asking for assumptions, and requesting citations to supplied material. A prompt should not be treated as a security boundary. User-provided text can contain prompt injection, such as instructions designed to override system rules or disclose confidential content.

Prompt quality should be evaluated empirically. Test representative inputs, difficult cases, ambiguous requests, multilingual examples, and adversarial content. A prompt that works in a demonstration may fail when documents, users, or model versions change.

5. Retrieval-Augmented Generation

Retrieval-augmented generation combines information retrieval with language generation. It is useful when answers should reflect changing, private, specialised, or traceable information.

A basic RAG pipeline is:

1. Collect authorised documents.
2. Clean and divide them into meaningful chunks.
3. Create an embedding for each chunk.
4. Store embeddings and metadata in an index.
5. Embed the user query.
6. Retrieve relevant chunks using vector, keyword, or hybrid search.
7. Place selected evidence in the model context.
8. Instruct the model to answer using the evidence.
9. Display citations or document references.
10. Evaluate retrieval and answer quality separately.

Chunk size involves a trade-off. Very small chunks may lose context; very large chunks may dilute relevance and consume context capacity. Metadata such as date, authorisation level, department, and document version supports filtering and governance.

RAG does not eliminate hallucination. Retrieval may return irrelevant or outdated passages, and the model may misread or contradict them. A reliable system should permit the response “insufficient evidence,” show sources, control access before retrieval, and log the evidence used.

RAG evaluation has at least two layers. Retrieval evaluation asks whether relevant evidence was returned. Generation evaluation asks whether the answer is supported, complete, relevant, and well written. Human assessment is especially important for complex or high-risk questions.

6. Generative AI Across Modalities

Generative AI models learn patterns from examples and produce new outputs. Text generation may create explanations, summaries, code, or dialogue. Image-generation systems may create images from text or transform existing images. Speech systems may synthesise voices or alter audio. Multimodal models may answer questions about images or generate text from video.

A practical generative workflow includes:

1. Define the purpose, users, acceptable use, and risk level.
2. Select a model whose licence, data practices, quality, and operating cost are suitable.
3. Prepare prompts, reference material, and output constraints.
4. Generate several test outputs.
5. Evaluate factuality, relevance, bias, style, safety, and intellectual-property concerns.
6. Add human approval when errors could cause harm.
7. Monitor performance and update the system as data and models change.

Common applications include drafting marketing variations, generating synthetic training examples, creating accessible descriptions of images, assisting software development, producing educational feedback, and prototyping designs. Generated content should be labelled where users could mistake it for human-produced or verified material.

7. Hallucinations, Bias, and Reliability

A hallucination is an output that is presented as plausible but is unsupported, false, or inconsistent with the available evidence. Causes include incomplete training data, ambiguous prompts, pressure to produce an answer, retrieval errors, flawed reasoning, and statistical generation without a built-in truth guarantee.

Risk-reduction practices include grounding answers in authoritative sources, requiring citations, using constrained output formats, checking calculations with external tools, asking the system to state uncertainty, and conducting human review. Confidence expressed in natural language is not a reliable probability estimate unless calibrated and validated.

Bias can enter through data collection, labels, model architecture, prompts, deployment context, or human interpretation. Evaluation should examine performance across relevant languages, groups, accents, image conditions, and use cases. Fairness cannot be reduced to one universal metric; a suitable measure depends on the task, harms, and legal context.

Reliability also includes robustness, privacy, security, and availability. A system may be accurate on average but unsafe for rare cases. Testing should include distribution shifts, missing information, unusual inputs, attempts to manipulate the model, and failures of connected tools.

8. Ethical, Legal, and Governance Considerations

AI projects should establish responsibility before deployment. Important questions include: What data may be used? Who owns or controls it? Is consent required? Can a person challenge an automated decision? Who approves high-impact outputs? How are incidents reported?

Privacy requires data minimisation, purpose limitation, access control, retention rules, and protection of personal and confidential information. Sensitive data should not be pasted into a public model service without an authorised legal and technical basis.

Copyright and licensing issues apply to training data, retrieved documents, prompts, generated outputs, and images or voices resembling identifiable people. Organisations should review licences and applicable laws rather than assuming that generated content is automatically unrestricted.

Security threats include prompt injection, data poisoning, model extraction, sensitive-data leakage, malicious files, and unsafe tool use. Tool-enabled agents should follow least privilege: they should receive only the access needed for the task and should require confirmation for consequential actions.

Governance should include documentation, model and data versioning, audit logs, access management, incident response, impact assessment, and periodic review. Human oversight must be meaningful: reviewers need sufficient expertise, time, authority, and visibility into evidence and uncertainty.

9. Evaluation and Professional Practice

Evaluation must match the task. Classification may use accuracy, precision, recall, and F1 score. For binary classification, precision is the proportion of predicted positives that are correct, while recall is the proportion of actual positives that are found:

Here, TP is true positives, FP false positives, and FN false negatives. These formulas assume a defined positive class and a fixed decision threshold. Accuracy can be misleading when classes are imbalanced.

Text generation may be evaluated for factual consistency, relevance, completeness, fluency, toxicity, and task success. Image systems may be evaluated for detection accuracy, segmentation overlap, visual quality, and safety. Speech systems may use WER but should also assess accent, noise, names, and domain vocabulary.

A professional evaluation set should be kept separate from development examples. It should contain ordinary cases, edge cases, counterexamples, and cases representing affected groups. Automated metrics are useful signals, not complete substitutes for expert judgement.

Illustrative Case: A College Policy Assistant

Background

A college wants an assistant that answers questions about attendance, assessment deadlines, academic misconduct, and disability support. The policy documents are updated each semester and contain confidential procedures.

Problem

A general-purpose chatbot gives fluent answers but sometimes uses obsolete rules and invents deadlines. Students need source-linked answers, while staff require access controls and an audit trail.

Analysis

The task is primarily retrieval and grounded question answering, not unrestricted creative writing. The risk is moderate to high because incorrect advice may affect assessment or student rights. Documents have different audiences, versions, and access permissions.

Approach and Solution

The college creates an authorised document repository, removes obsolete versions from normal retrieval, and attaches metadata for semester, policy owner, audience, and access level. Documents are chunked by headings and embedded. A hybrid retrieval system combines keyword matching for exact policy terms with vector search for semantic similarity.

The assistant is instructed to answer only from retrieved evidence, cite the policy title and section, identify the applicable semester, and say that it cannot determine an answer when evidence is insufficient. Questions involving formal appeals or personal circumstances are routed to a human adviser. Logs record the question, retrieved passages, model version, and response, while excluding unnecessary personal data.

Outcome

In testing, the system answers routine policy questions more consistently and makes unsupported answers less often. Human reviewers still find occasional problems when a question combines several policies or refers to an unusual personal situation. The assistant is therefore deployed as an information aid rather than an autonomous decision-maker.

Learning Points

The case demonstrates that model selection alone does not solve reliability. Document governance, retrieval quality, access control, refusal behaviour, citations, evaluation, and escalation are equally important. It also shows why a system’s role must be defined narrowly enough to support meaningful testing and accountability.

10. Contemporary Developments and Limitations

Current systems increasingly combine language, vision, speech, code, and external tools. Smaller specialised models may be cheaper and easier to run privately, while larger general models may provide broader capabilities. Quantisation, distillation, caching, and efficient retrieval can reduce cost and latency, but may affect quality.

Agents are systems that use models to plan steps, call tools, inspect results, and continue toward a goal. They can automate useful workflows, but every additional tool increases the attack surface and the number of possible failure paths. Human confirmation is appropriate before sending messages, changing records, making purchases, or taking irreversible actions.

Important limitations remain. Models can be expensive to train and operate, consume substantial energy and computing resources, and reproduce errors at scale. They may lack grounded understanding, struggle with long chains of reasoning, and perform unevenly across languages and populations. Generative systems can also affect employment, education, authorship, information integrity, and public trust.

Best practice is therefore not “use AI whenever possible.” It is to use AI where the task, data, risk, accountability, and expected benefit justify it. A simpler search, rules-based system, database query, or human process may be more reliable for some problems.

Chapter Summary

Natural-language, vision, speech, and multimodal AI convert diverse human information into computational representations and useful predictions or generated outputs. Text is tokenised and represented with embeddings; embeddings support semantic comparison and retrieval. Transformers use attention to model relationships among tokens, and LLMs generate sequences by estimating probable next tokens.

Prompting specifies tasks, context, constraints, and output requirements. RAG supplements generation with retrieved evidence, but it requires authorised documents, careful chunking, retrieval evaluation, citations, and refusal behaviour. Generative AI supports drafting, synthesis, accessibility, design, and assistance, but it does not guarantee truth.

Reliable professional systems require representative data, appropriate metrics, human review, privacy protection, security controls, documentation, monitoring, and clear accountability. Hallucinations, bias, distribution shift, copyright concerns, prompt injection, and tool misuse must be treated as engineering and governance problems, not merely as prompt-writing problems.

Key Terms

Embedding: A numerical vector representing an item such as text, an image, or audio.

Attention: A mechanism that assigns different weights to parts of an input when producing a representation.

Transformer: A neural-network architecture built around attention and related processing layers.

Token: A unit produced by a text-tokenisation process.

Large language model: A large model trained to represent and generate language.

Prompt: Instructions and context supplied to an AI model.

Few-shot prompting: Prompting that includes examples of the desired task or format.

Hallucination: A plausible but unsupported, false, or inconsistent model output.

Retrieval-augmented generation: Generation grounded by information retrieved from an external collection.

Chunk: A segment of a document used for indexing or retrieval.

Multimodal AI: AI that processes or generates more than one data modality.

Computer vision: AI methods for interpreting images and video.

Automatic speech recognition: Conversion of spoken audio into text.

Optical character recognition: Conversion of text in an image into machine-readable characters.

Precision: The proportion of predicted positive cases that are truly positive.

Recall: The proportion of actual positive cases that are correctly found.

Prompt injection: Input designed to manipulate a model into violating intended instructions or controls.

Model drift: A change in real-world data or behaviour that reduces system performance over time.

Human-in-the-loop: A design in which people review, approve, correct, or supervise AI outputs.

Review Questions

Short Answer Questions

1. What is an embedding, and name two uses for embeddings.
2. Distinguish image classification from object detection.
3. What is the purpose of retrieval in a RAG system?
4. Define hallucination in generative AI.
5. Name two controls that can reduce privacy risk in an AI project.

Long Answer Questions

1. Explain the main stages of a practical text-AI workflow, from data collection to monitoring.
2. Describe self-attention and explain why transformers are effective for language processing.
3. Explain how a RAG system works and discuss two reasons why RAG can still produce incorrect answers.
4. Compare the evaluation requirements for an ASR system and a document-question-answering system.
5. Discuss ethical, legal, and security issues that should be addressed before deploying a generative-AI assistant.

Analytical / Application Questions

1. A company wants an AI system to answer questions from frequently changing internal procedures. Design a suitable workflow, including document handling, retrieval, prompting, evaluation, and access control.
2. An image classifier performs well overall but poorly on images from one lighting condition and one demographic group. Analyse possible causes and propose an evaluation and improvement plan.
3. A language model produces confident answers to questions outside its supplied evidence. Recommend changes to the system design that would reduce unsupported answers while preserving useful responses.

Multiple Choice Questions with Answers

Question 1

Which representation is most directly used for semantic similarity search?

A. Embedding vector

B. Raw monitor resolution

C. Audio sample rate only

D. Database password

Correct Answer: A. Embedding vector

Question 2

In the attention equation, what do Q, K, and V represent?

A. Queries, keys, and values

B. Quantisation, kernels, and variance

C. Questions, knowledge, and verification

D. Quality, knowledge, and vocabulary

Correct Answer: A. Queries, keys, and values

Question 3

What is the main purpose of retrieval-augmented generation?

A. To ground generated answers in retrieved evidence

B. To remove the need for document access controls

C. To guarantee that every generated statement is true

D. To replace evaluation with a larger prompt

Correct Answer: A. To ground generated answers in retrieved evidence

Question 4

Which metric is commonly used for automatic speech-recognition errors?

A. WER

B. IoU only

C. GDP

D. Latency-free accuracy

Correct Answer: A. WER

Question 5

A model identifies whether an entire photograph contains a bicycle but does not locate it. Which task is this?

A. Image classification

B. Object detection

C. Instance segmentation

D. Speech synthesis

Correct Answer: A. Image classification

Question 6

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0.50

C. 0.80

D. 1.25

Correct Answer: C. 0.80

Question 7

Which practice best reduces the risk of unsupported answers in a policy assistant?

A. Require answers to use authorised evidence and cite it

B. Increase creativity without changing the evidence

C. Remove all refusal behaviour

D. Allow every user to retrieve every document

Correct Answer: A. Require answers to use authorised evidence and cite it

Question 8

What is prompt injection?

A. Input intended to manipulate a model into violating instructions or controls

B. A method for compressing image files

C. A standard speech-recognition metric

D. A technique for labelling bounding boxes

Correct Answer: A. Input intended to manipulate a model into violating instructions or controls

Question 9

Why should tool-enabled AI agents use least privilege?

A. To limit damage if the model or input is manipulated

B. To make the model generate longer responses

C. To eliminate the need for authentication

D. To ensure all tools are always available

Correct Answer: A. To limit damage if the model or input is manipulated

Question 10

Which statement about hallucinations is most accurate?

A. They are plausible outputs that may be unsupported or false

B. They occur only in image models

C. They are eliminated by using a longer prompt

D. They are always obvious to users

Correct Answer: A. They are plausible outputs that may be unsupported or false