11. Applied AI: Language, Vision, Recommendations and Generative AI

Introduction and Context

Artificial intelligence becomes economically and socially significant when it moves beyond prediction in controlled datasets and interacts with language, images, speech, preferences, and human work. Applied AI systems interpret customer messages, inspect industrial products, recommend media, transcribe meetings, generate software, and support decisions. These applications draw on several related fields: natural language processing (NLP), computer vision, speech technologies, recommender systems, and multimodal AI.

Recent progress has been accelerated by representation learning, embeddings, transformer architectures, large-scale pre-training, and foundation models. These techniques allow one model to support many tasks, although successful deployment still requires carefully designed data, evaluation, security, governance, and human oversight. A fluent model is not necessarily a truthful model; a highly accurate classifier may still be unfair; and a useful recommendation engine may create excessive personalization or filter bubbles.

This chapter examines the principal technologies, their relationships, enterprise uses, and limitations. It also considers prompting, retrieval-augmented generation (RAG), fine-tuning, and agentic workflows as practical methods for building applications with foundation models.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the roles of NLP, computer vision, speech AI, recommender systems, and multimodal AI.

2. Describe embeddings, attention, transformers, foundation models, and large language models.

3. Distinguish prompting, RAG, fine-tuning, and tool-using agentic workflows.

4. Select suitable architectures and evaluation measures for applied AI problems.

5. Analyse enterprise use cases, deployment risks, governance requirements, and ethical trade-offs.

6. Design a responsible applied-AI solution for a realistic organizational scenario.

From Raw Data to Useful Representations

Applied AI begins with a representation of the world. Text can be represented as tokens, images as pixels or visual features, audio as waveforms or spectrograms, and user activity as events such as clicks, purchases, or viewing duration. Machine-learning models transform these inputs into representations that preserve information relevant to a task.

An embedding is a dense numerical vector representing an item, such as a word, document, image, product, or user. Items with related meanings tend to occupy nearby positions in an embedding space. Similarity can be estimated using cosine similarity or another distance measure. Embeddings support semantic search, clustering, duplicate detection, recommendation, classification, and retrieval. They are not complete meanings: their quality depends on training data, context, language, domain, and the objective used to learn them.

A text embedding, for example, may place two documents about loan eligibility close together even when they use different words. A product embedding may capture relationships between products purchased or viewed by similar users. Multimodal models attempt to place text and images in a shared space, enabling a query such as “red formal shoes” to retrieve both product descriptions and photographs.

Attention and Transformers

Earlier sequence models processed language largely one step at a time. Transformer architectures introduced self-attention, allowing each token to weigh the relevance of other tokens in a sequence. In simplified form, attention combines queries, keys, and values to calculate context-sensitive representations. Positional information is added because attention alone does not inherently specify word order.

Transformers can be organized as encoders, decoders, or encoder–decoder systems. Encoder models are often effective for classification and representation learning; decoder models generate sequences; encoder–decoder models transform one sequence into another, as in translation or summarization. Their capacity for parallel training and long-range contextual modelling made them central to modern foundation models.

A foundation model is trained on broad data at substantial scale and adapted to many downstream tasks. A large language model (LLM) is a foundation model specialized in generating or understanding language, usually after pre-training to predict tokens and subsequent instruction or preference alignment. Scale improves capability in many settings, but it does not guarantee factuality, reasoning reliability, domain expertise, or safety.

Language, Vision and Speech as Applied Systems

Natural Language Processing

NLP involves computational analysis and generation of human language. Typical tasks include sentiment analysis, named-entity recognition, document classification, question answering, translation, summarization, information extraction, and conversational interaction. Traditional systems often relied on manually designed features, while contemporary systems use learned representations and transformers.

An enterprise NLP pipeline may include document ingestion, language identification, cleaning, tokenization, embedding, retrieval or classification, generation, and post-processing. The choice depends on the task. A regulatory reporting system may require extraction with traceable evidence rather than unconstrained generation. A customer-support assistant may combine intent classification, retrieval, dialogue management, and escalation to a human operator.

Evaluation must reflect the task. Precision and recall are useful for entity extraction and search; F1 balances them. Translation and summarization may use overlap-based measures, but human evaluation remains important because grammatical output can still be inaccurate or misleading. For question answering, factuality, citation correctness, refusal behaviour, latency, and cost may be more important than surface similarity.

Computer Vision

Computer vision enables machines to interpret images and video. Classification assigns a label to an image; object detection identifies and localizes objects; semantic segmentation assigns a class to each pixel; instance segmentation distinguishes individual objects. Optical character recognition converts visual text into machine-readable form. Vision-language models connect visual content with language, enabling captioning, visual question answering, and image-text retrieval.

A vision system is shaped by its data collection process. Lighting, camera position, device quality, seasonal change, and demographic representation can produce distribution shifts. A model trained on ideal product photographs may perform poorly on damaged goods photographed in a warehouse. Thus, validation should represent the deployment environment rather than only the training distribution.

Speech and Audio AI

Speech recognition converts audio into text, while text-to-speech generates spoken output. Related tasks include speaker identification, diarization, keyword detection, emotion or paralinguistic analysis, and speech translation. A practical meeting assistant may separate speakers, transcribe a conversation, summarize decisions, and extract action items.

Speech applications face accents, background noise, overlapping speakers, code-switching, privacy concerns, and variable microphones. Word error rate is a common metric for transcription, but an organization may also measure the accuracy of names, figures, medical terms, or legally significant phrases. Voice systems require clear disclosure when users are interacting with synthetic speech.

Recommender Systems: Personalization as Prediction and Decision

A recommender system estimates which items a user may value and presents a ranked list. Collaborative filtering learns from interactions among users and items. In user–item matrices, explicit feedback includes ratings, whereas implicit feedback includes clicks, purchases, skips, and dwell time. Matrix factorization represents users and items in a shared latent space. Content-based methods use item attributes and user profiles. Modern systems often combine collaborative, content, sequential, and contextual signals.

Large-scale recommendation commonly uses a two-stage architecture. Candidate generation retrieves a manageable set of potentially relevant items. Ranking then orders candidates using richer features such as recency, price, availability, history, and context. A final policy may apply business rules, diversity constraints, safety filters, and exploration.

Offline measures include precision@k, recall@k, mean reciprocal rank, and normalized discounted cumulative gain. Online measures may include click-through rate, conversion, retention, revenue, satisfaction, or complaint rate. Optimizing one metric can cause undesirable behaviour: click maximization may promote sensational content, while short-term sales optimization may reduce long-term trust. Recommenders should therefore consider diversity, novelty, calibration, fairness, and user welfare.

The cold-start problem occurs when a new user or item has little interaction history. Metadata, onboarding questions, popularity baselines, contextual information, and controlled exploration can help. Feedback loops are another concern: recommendations influence what users see, which creates the data used for future recommendations. A system can therefore reinforce narrow preferences or existing inequalities.

Multimodal and Generative AI

Multimodal AI processes or produces more than one modality, such as text, image, audio, video, or structured data. A retail assistant might accept a photograph, identify a product defect, consult a manual, and explain the repair in speech. Multimodality can improve accessibility and enable richer workflows, but it also introduces cross-modal errors. A system may correctly read text in an image yet misunderstand its context, or generate a plausible caption that invents an object.

Generative AI produces new content based on learned distributions. Language models generate text, diffusion models commonly generate images, and related systems generate audio, video, code, or structured outputs. Generation is controlled by inputs such as prompts, system instructions, examples, decoding settings, retrieved context, and tools.

A prompt is an instruction or input supplied to a model. Effective prompting defines the task, supplies relevant context, specifies constraints and output format, and identifies uncertainty or refusal conditions. Few-shot prompting gives examples; structured prompting requests a schema such as JSON. Prompting is usually the fastest adaptation method, but it cannot reliably provide information absent from the model or guarantee compliance.

Choosing among Prompting, RAG and Fine-Tuning

These methods solve different problems.

Prompting changes the instruction and context without changing model parameters. It is appropriate for prototyping, task specification, role definition, formatting, and simple workflow control.

Retrieval-augmented generation retrieves relevant documents or records and places them in the model’s context before generation. A typical RAG pipeline includes document ingestion, chunking, metadata creation, embedding, vector or hybrid search, reranking, prompt construction, generation, and citation or validation. RAG is valuable when information is private, changing, or too large to fit reliably into training. It can reduce unsupported answers, but retrieval errors, poor chunking, stale indexes, access-control failures, and irrelevant context remain possible.

Fine-tuning updates model parameters using task- or domain-specific examples. It may improve style, classification, instruction following, or specialized terminology. Fine-tuning is less suitable as a substitute for frequently changing factual knowledge. It requires representative, high-quality data and may cause overfitting, capability regression, privacy leakage, or unexpected behaviour. Parameter-efficient methods can reduce computational cost, but they do not remove the need for careful evaluation.

A useful decision rule is to begin with prompting and a small evaluation set, add RAG when authoritative external knowledge is required, and consider fine-tuning when repeated behaviour or domain adaptation cannot be achieved adequately through context and instructions.

Agentic Workflows and Enterprise Architecture

An agentic workflow uses a model to plan or select actions, call tools, inspect results, and continue toward a goal. Tools may include databases, search systems, calculators, workflow software, code interpreters, or enterprise APIs. A reliable agent is not merely a chatbot; it is a governed software system with defined permissions, state, error handling, observability, and escalation.

A basic workflow is:

1. Interpret the request and determine whether it is within scope.

2. Retrieve relevant information or choose an approved tool.

3. Execute a bounded action.

4. Validate the result and record evidence.

5. Ask for confirmation before consequential actions.

6. Escalate uncertainty or failure to a human.

Planning loops can be useful for complex tasks, but open-ended autonomy increases risks such as unauthorized transactions, prompt injection, data exfiltration, tool misuse, and cascading errors. Deterministic workflows are often preferable when the task has a stable sequence and high compliance requirements.

A production architecture generally separates the user interface, orchestration layer, model gateway, retrieval and data services, tool layer, monitoring, identity management, and governance controls. Model gateways can manage versioning, routing, rate limits, logging, and cost. Sensitive information should be minimized, encrypted, access-controlled, and retained according to policy.

Applied Enterprise Scenarios

In financial services, NLP can classify customer correspondence, extract entities from applications, and support analysts with policy retrieval. Such systems should not silently make high-impact decisions without explainability, review, and controls against discriminatory proxies.

In manufacturing, computer vision can identify surface defects and predict maintenance needs from images and sensor data. The system should be evaluated across equipment types and operating conditions, with a fallback inspection process when confidence is low.

In healthcare, speech transcription and summarization can reduce documentation burden, but clinical facts, negations, medication names, and patient identity require stringent validation. Generative output should support—not replace—professional judgment.

In education, recommendation systems can suggest learning resources and generative systems can provide feedback. Institutions must distinguish tutoring from assessment, protect student data, and prevent automated feedback from reproducing linguistic or cultural bias.

Illustrative Case Study: An Internal Policy Assistant

Background. A multinational organization maintains thousands of policies, procedures, and technical manuals across several repositories. Employees spend considerable time searching for current guidance.

Situation. A proposed LLM assistant answers questions in natural language. A pilot reveals that it sometimes cites obsolete documents, combines rules from unrelated regions, and presents uncertain answers confidently.

Analysis. The core problem is not simply model intelligence. Documents have inconsistent ownership and metadata; access rights differ by department; and evaluation has focused on fluency rather than evidence. The assistant also lacks a clear escalation path for legal or safety questions.

Approach. The organization creates an authoritative document registry with owner, region, effective date, and review status. A hybrid retrieval system combines keyword and embedding search, then reranks results using metadata and relevance. The assistant is instructed to answer only from retrieved sources, cite document identifiers and dates, identify missing information, and refuse questions outside its scope. Access controls are applied before retrieval, not merely after generation. A test set covers common questions, ambiguous cases, obsolete policies, regional conflicts, and adversarial prompts. High-risk topics require human approval.

Outcome. The resulting system gives fewer unsupported answers and makes its evidence visible to users. Its value is measured through answer correctness, citation validity, time saved, escalation quality, and user satisfaction rather than conversational smoothness alone.

Learning points. Enterprise generative AI depends on information governance, retrieval quality, evaluation design, and workflow integration. RAG improves access to current information but does not automatically make an organization’s information authoritative. Human accountability remains essential for consequential decisions.

Limitations, Risks and Governance

Applied AI systems can fail through biased data, distribution shift, adversarial input, hallucination, privacy leakage, copyright conflict, insecure tool use, and overreliance by users. Generative models may fabricate sources, misinterpret ambiguous instructions, reproduce training-data stereotypes, or expose confidential content through prompts and logs. Vision systems may perform unequally across demographic groups or environments. Recommenders may manipulate attention or disadvantage less-visible creators.

Governance should cover the full lifecycle: problem definition, data provenance, model selection, testing, deployment, monitoring, incident response, retirement, and audit. Risk classification should consider the impact of an error, not only the technical novelty of a system. Important controls include:

• documented purpose, ownership, data sources, and known limitations;
• representative validation and subgroup performance analysis;
• access control, privacy protection, retention limits, and secure logging;
• human review for high-impact or irreversible actions;
• content provenance, citation, and disclosure of synthetic media where appropriate;
• monitoring for drift, abuse, prompt injection, latency, cost, and quality degradation;
• user mechanisms for correction, appeal, and reporting harmful outputs.

Professional practice also requires distinguishing assistance from delegation. A model can draft a report, but the accountable professional must verify claims. An agent can prepare a transaction, but authorization and execution controls should remain explicit.

Evaluation and Best Practices

A strong evaluation framework combines technical, human, operational, and business measures. Build a task-specific benchmark before deployment, including normal, difficult, ambiguous, and adversarial cases. Compare against a simple baseline; complexity is justified only when it creates measurable value. Test retrieval separately from generation, and inspect failures rather than relying only on averages.

Use confidence or uncertainty signals cautiously. Language-model probabilities do not guarantee truth, and a high-confidence answer may still be wrong. Grounding, citations, constrained outputs, validation rules, and human review are often more reliable controls.

Best practice is to begin with a narrowly defined use case, use the least autonomous design that meets the objective, and establish an evaluation-and-monitoring loop. Design for accessibility and multiple languages where relevant. Involve domain experts, security specialists, legal and compliance teams, and affected users. Track total cost, including data preparation, inference, integration, review, and failure remediation.

Current Developments and Emerging Directions

Current research and industry development are moving toward smaller domain-specific models, efficient inference, multimodal foundation models, long-context processing, synthetic data, model routing, and structured tool use. Retrieval is becoming more hybrid and context-aware, while evaluation is shifting from generic benchmarks toward task-specific, evidence-based and safety-sensitive testing.

Agentic systems are gaining attention, but reliable autonomy remains constrained by planning errors, uncertain world models, security threats, and accountability requirements. On-device and private models may improve latency and data control. Provenance technologies and watermarking may assist in identifying generated media, although no single mechanism is universally reliable. The central direction is not simply larger models; it is better integration of models with trustworthy data, constrained workflows, and responsible institutions.

Chapter Summary

Applied AI combines learned representations, domain data, software systems, and human workflows. NLP interprets and generates language; computer vision analyses visual information; speech systems process audio; recommender systems rank items under feedback and business constraints; and multimodal systems connect several forms of data. Embeddings support semantic comparison, while transformers and attention underpin many foundation models and LLMs.

Prompting provides task instructions, RAG connects generation to external knowledge, and fine-tuning adapts model behaviour through examples. Agentic workflows add tools and planning but require permissions, validation, monitoring, and escalation. Enterprise success depends on more than model capability: it requires suitable evaluation, secure architecture, data governance, fairness, privacy, transparency, and accountable human judgment.

Key Terms

Embedding: A dense vector representation that captures useful relationships among data items.

Attention: A mechanism that weights the relevance of elements in a sequence when computing a representation.

Transformer: A neural architecture based primarily on attention for processing sequential or multimodal data.

Foundation model: A broadly pretrained model adapted to multiple downstream tasks.

Large language model: A large pretrained model designed to process and generate language.

Natural language processing: Computational analysis and generation of human language.

Computer vision: AI concerned with interpreting images and video.

Speech recognition: Conversion of spoken audio into text.

Recommender system: A system that predicts and ranks items likely to interest a user.

Collaborative filtering: Recommendation based on patterns across users and items.

Cold start: Difficulty making recommendations when users or items have little historical data.

Multimodal AI: AI that processes or connects multiple data modalities.

Prompting: Controlling model behaviour through instructions and contextual input.

Retrieval-augmented generation: Generation grounded in information retrieved from external sources.

Fine-tuning: Updating model parameters using task- or domain-specific examples.

Agentic workflow: A model-supported process that plans, uses tools, and performs bounded actions.

Hallucination: A plausible but unsupported or inaccurate model output.

Distribution shift: A change between training conditions and deployment conditions.

Grounding: Connecting an output to verified external data or evidence.

Model drift: Degradation caused by changes in data, behaviour, or operating conditions.

Review Questions

Short Answer Questions

1. What is an embedding, and why is it useful in semantic search?

2. Distinguish object detection from image classification.

3. What is the cold-start problem in recommender systems?

4. State one important difference between RAG and fine-tuning.

5. Why should an agentic workflow include human escalation?

Long Answer Questions

1. Explain how attention and transformers support modern NLP applications.

2. Compare collaborative filtering, content-based recommendation, and hybrid recommendation.

3. Describe an enterprise RAG architecture, including retrieval, generation, and governance.

4. Discuss the principal risks of multimodal and generative AI in professional settings.

5. Explain how an organization should evaluate an AI system beyond model accuracy.

Analytical / Application Questions

1. A company wants an assistant to answer questions about frequently changing internal policies. Should it use prompting, RAG, fine-tuning, or a combination? Justify your decision.

2. A recommender increases click-through rate but also increases complaints and reduces content diversity. Analyse the likely cause and propose revised measures and controls.

3. Design a responsible AI workflow for a meeting assistant that transcribes speech, summarizes decisions, and assigns action items. Identify data, evaluation, privacy, and human-review requirements.

22. Multiple Choice Questions with Answers

Question 1

A company needs an assistant to answer questions using private policies that change every month. Which approach is most directly appropriate?

A. Train a model once on public web data

B. Use retrieval-augmented generation over governed policy documents

C. Use only image classification

D. Remove all source documents from the workflow

Correct Answer: B. Use retrieval-augmented generation over governed policy documents

Question 2

What is the principal function of self-attention in a transformer?

A. It permanently stores every training document

B. It assigns relevance among elements in a sequence when forming representations

C. It converts images into database tables without training

D. It guarantees that generated statements are factual

Correct Answer: B. It assigns relevance among elements in a sequence when forming representations

Question 3

Which situation best illustrates the cold-start problem?

A. A popular item has millions of interactions

B. A recommender ranks items using several features

C. A new user has no interaction history

D. A model is evaluated using recall@k

Correct Answer: C. A new user has no interaction history

Question 4

Which statement best distinguishes fine-tuning from prompting?

A. Fine-tuning changes model parameters using examples, whereas prompting changes the input instructions or context

B. Fine-tuning is only used for image pixels, whereas prompting is only used for databases

C. Prompting always requires more training data than fine-tuning

D. Fine-tuning guarantees current factual knowledge

Correct Answer: A. Fine-tuning changes model parameters using examples, whereas prompting changes the input instructions or context

Question 5

A vision model performs well in laboratory photographs but poorly in a factory with low light and dust. What is the most likely explanation?

A. Data distribution shift

B. Excessive semantic search

C. The cold-start problem in recommendations

D. A valid increase in model generalization

Correct Answer: A. Data distribution shift

Question 6

Why can optimizing a recommender only for click-through rate be harmful?

A. Clicks are never measurable

B. It may promote attention-grabbing content while reducing trust, diversity, or long-term welfare

C. Click-through rate is identical to every fairness measure

D. It prevents candidate generation entirely

Correct Answer: B. It may promote attention-grabbing content while reducing trust, diversity, or long-term welfare

Question 7

Which control most directly reduces the risk of an AI agent making an unauthorized high-impact transaction?

A. Allow unrestricted tool access

B. Require bounded permissions and human confirmation before execution

C. Increase the temperature of generation

D. Remove all monitoring logs

Correct Answer: B. Require bounded permissions and human confirmation before execution

Question 8

In the policy-assistant case, why was document metadata important?

A. It helped distinguish current, region-specific and authoritative documents

B. It replaced the need for retrieval

C. It guaranteed that the language model could not hallucinate

D. It converted every document into an image

Correct Answer: A. It helped distinguish current, region-specific and authoritative documents

Question 9

Which evaluation is most appropriate for a speech-transcription system used to record medication names?

A. Only image classification accuracy

B. Word error rate together with targeted accuracy for medication terms

C. Only recommendation diversity

D. The number of prompt examples

Correct Answer: B. Word error rate together with targeted accuracy for medication terms

Question 10

What is a central limitation of an LLM with fluent output?

A. Fluency guarantees evidence

B. Fluency can coexist with fabricated or unsupported claims

C. Fluency prevents multilingual use

D. Fluency makes retrieval impossible

Correct Answer: B. Fluency can coexist with fabricated or unsupported claims