AI, Text Analytics and Generative Analytics

Introduction and Context

Data analytics has expanded from structured tables and statistical models to include language, images, audio, software code and multimodal records. Text is especially valuable because organisations express plans, decisions, complaints, risks and expertise in natural language. However, text is also difficult to analyse: meaning depends on context, words may be ambiguous, documents vary in quality, and important information may be implicit rather than explicitly encoded.

Artificial intelligence (AI) provides methods for learning patterns from data and using those patterns for prediction, classification, retrieval and generation. Natural language processing (NLP) focuses on computational treatment of human language. Deep learning supplies powerful representation-learning methods, while generative AI produces new text, code, images or structured outputs. In analytics, these technologies can support document discovery, customer intelligence, compliance, forecasting workflows and decision support.

Their usefulness must not be confused with reliability. A fluent model can be factually wrong; an embedding can encode social bias; a language model can disclose sensitive information; and a generative system can create substantial computational cost. Consequently, postgraduate analytics practice requires both technical competence and critical evaluation.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Explain the evolution from traditional text processing to deep language models and generative analytics.

2. Distinguish tokens, embeddings, language models, retrieval systems and generative models.

3. Design an NLP workflow for classification, extraction, clustering and semantic search.

4. Explain transformer-based modelling, attention and retrieval-augmented generation (RAG).

5. Evaluate accuracy, calibration, hallucination, fairness, security, privacy and cost.

6. Select appropriate models and human controls for professional analytical applications.

1. From Text Processing to Foundation Models

Early text analytics represented documents using manually designed features. A document-term matrix records the frequency of terms in documents. A common weighting method is term frequency–inverse document frequency (TF–IDF):

TF–IDF(t,d) = TF(t,d) × log(N / DF(t)).

Here, TF(t,d) is the frequency of term t in document d, N is the number of documents, and DF(t) is the number of documents containing t. The formula assumes that a term occurring often in one document but in relatively few documents is informative. Different software packages use normalised frequencies and smoothing variants, so the precise implementation should be documented. TF–IDF is useful for interpretable retrieval and linear classification, but it does not reliably represent synonyms, word order or long-range meaning.

Distributional semantics improved representation by learning that words occurring in similar contexts tend to have related meanings. Word embeddings such as Word2Vec and GloVe represent a word as a dense vector. Similarity is often measured using cosine similarity:

cos(x,y) = (x · y) / (||x|| ||y||),

where x and y are non-zero vectors, x · y is their dot product, and ||x|| and ||y|| are their Euclidean norms. Cosine similarity measures orientation rather than magnitude and is meaningful only when the embedding space and preprocessing are compatible. It should not be interpreted as a universal measure of semantic truth.

Word-level embeddings assign one vector to a word type, so the word “bank” may have difficulty representing financial and geographic meanings simultaneously. Contextual models generate representations depending on surrounding words. Transformer models advanced this approach by processing relationships among tokens using attention and by pre-training on large corpora.

A foundation model is a broadly trained model that can be adapted to many downstream tasks. A large language model (LLM) is a foundation model primarily trained to model sequences of language. It may be used through prompting, fine-tuning, retrieval augmentation or tool integration. These methods are not interchangeable: prompting changes instructions, fine-tuning changes model parameters, and retrieval supplies external evidence at inference time.

2. Core NLP Tasks

Classification

Classification assigns one or more labels to a text unit. Examples include spam detection, sentiment analysis, incident categorisation and regulatory-risk screening. Binary classification has two labels; multiclass classification assigns one of several mutually exclusive labels; multilabel classification permits several labels simultaneously.

Evaluation must reflect the decision context. For a positive class, precision is:

Precision = TP / (TP + FP),

and recall is:

Recall = TP / (TP + FN).

TP, FP and FN denote true positives, false positives and false negatives. Precision answers, “Among predicted positives, how many were correct?” Recall answers, “Among actual positives, how many were found?” The F1 score is the harmonic mean of precision and recall:

F1 = 2PR / (P + R),

provided precision P and recall R are defined and their sum is non-zero. F1 gives equal importance to precision and recall; it is inappropriate when business costs are strongly asymmetric. In medical triage, for example, a false negative may be much more harmful than a false positive, so threshold selection should reflect expected cost rather than F1 alone.

Information Extraction

Information extraction converts unstructured language into structured fields. Named-entity recognition identifies people, organisations, locations, products or dates. Relation extraction identifies links such as “supplier delivers component.” Event extraction identifies an event, its participants and its time. Extraction systems require clear annotation rules because labels such as “organisation,” “brand” and “department” may overlap.

Topic Discovery and Clustering

Topic models and clustering help analysts explore collections without predefined labels. Latent Dirichlet allocation (LDA) treats documents as mixtures of latent topics and topics as distributions over words. Its assumptions—such as a bag-of-words representation and a chosen number of topics—are simplifications. Topic coherence, stability across samples and human interpretability should be examined rather than treating topics as objectively discovered categories.

Sentiment and Emotion Analysis

Sentiment analysis estimates polarity or affect. It can support customer-service triage and product monitoring, but sarcasm, negation, domain vocabulary and cultural variation create errors. A sentence such as “The battery died after two hours—excellent design” demonstrates why word-level sentiment counts are insufficient. Sentiment should therefore be treated as an estimate with uncertainty, not as a direct measurement of a person’s psychological state.

Semantic Search and Question Answering

Lexical search matches terms, whereas semantic search compares representations of queries and documents. A production system commonly combines lexical retrieval, vector retrieval and a reranker. Hybrid retrieval is often stronger because exact identifiers and rare technical terms are important to lexical search, while paraphrases benefit from embeddings.

3. Deep Learning and Transformer Architecture

A neural network transforms an input through parameterised layers. During training, parameters are adjusted to reduce a loss function on examples. Deep models learn intermediate representations rather than relying entirely on manually specified features. Their effectiveness depends on data quality, objective design, optimisation, evaluation and deployment controls.

Transformers use attention to allow each token to weigh other tokens in a sequence. In simplified scaled dot-product attention:

Attention(Q,K,V) = softmax(QKᵀ / √d_k)V.

Q, K and V are query, key and value matrices; d_k is the dimensionality of the keys. The scaling term reduces the tendency of large dot products to produce excessively concentrated softmax values. The softmax operation converts each row of scores into non-negative weights summing to one. Multi-head attention uses several learned projections so that different heads can represent different relationships. Positional information is added because attention alone does not inherently encode sequence order.

Transformer families differ in architecture and training objective. Encoder models are effective for representation and classification. Decoder-only models generate continuations autoregressively. Encoder–decoder models transform one sequence into another and are suitable for tasks such as translation and summarisation. Pre-training learns general linguistic patterns, while supervised fine-tuning and preference optimisation adapt behaviour to particular tasks or instructions.

A language model estimates a probability distribution over the next token given preceding context. Generation is controlled by decoding parameters. Temperature changes the sharpness of the probability distribution; top-k restricts selection to the k most probable tokens; top-p selects from the smallest set whose cumulative probability exceeds p. These parameters affect variability, not factual correctness. Lower variability can still produce a confidently incorrect answer.

4. Embeddings as an Analytical Infrastructure

An embedding is a numerical representation intended to preserve useful relationships among items. Text embeddings can represent sentences, paragraphs, documents, queries, products or support tickets. An embedding pipeline normally includes segmentation, model encoding, vector storage, similarity retrieval and evaluation.

Important design choices include chunk size, overlap, metadata, language coverage and update policy. Very small chunks may lose context; very large chunks may reduce retrieval precision and exceed the model context window. Metadata filters can restrict retrieval by department, date, access level or document type. Access control must be enforced before or during retrieval, not merely described in a prompt.

Embeddings are not neutral. They reflect training data, model objectives and cultural or linguistic coverage. Similarity does not prove equivalence, causation or authorisation. Analysts should test retrieval using a representative query set containing synonyms, misspellings, languages, adversarial queries and rare identifiers.

A useful retrieval evaluation distinguishes retrieval quality from answer quality. Recall@k measures whether a relevant item appears among the first k results. Precision@k measures the proportion of the first k results that are relevant. Ranking metrics such as mean reciprocal rank may be useful when the first relevant result is especially important. These metrics require relevance judgments and should be reported with the dataset and query construction method.

5. Generative Analytics

Generative analytics uses models that create analytical artefacts, including summaries, explanations, hypotheses, synthetic records, code, queries and scenario descriptions. It can reduce the time required to inspect large collections, but generation is not the same as discovery of truth. A generated explanation may be plausible without being causally valid.

Retrieval-Augmented Generation

RAG connects a generative model to an external knowledge source. A typical workflow is:

1. Ingest and clean documents.

2. Split documents into meaningful chunks.

3. Generate and index embeddings.

4. Retrieve candidates for a user query.

5. Rerank and filter candidates using relevance and permissions.

6. Place selected evidence in the model context.

7. Generate an answer with citations or evidence links.

8. Validate the response and record the interaction.

RAG can reduce errors caused by stale internal knowledge, but it does not guarantee correctness. Failure may arise from missing documents, poor chunking, incorrect retrieval, prompt injection, conflicting sources or an unsupported synthesis. A high-quality system should distinguish evidence directly stated in sources from model interpretation and should allow the user to inspect the evidence.

Fine-Tuning, Prompting and Tools

Prompting is appropriate when the task can be specified through instructions and examples. Fine-tuning is useful when consistent style, classification behaviour or domain adaptation is needed and sufficient labelled data exist. Tool use allows a model to call a database, calculator, code interpreter or workflow service. Tool outputs should be treated as external data requiring validation; a model’s decision to call a tool is not itself proof that the tool result is appropriate.

For quantitative analytics, language models should generally orchestrate validated statistical or database tools rather than perform critical calculations solely through token prediction. SQL generated by an LLM should be checked for access restrictions, joins, aggregation logic and leakage of confidential fields.

6. Evaluation: Beyond Fluency

Evaluation should be task-specific, risk-sensitive and performed on data not used for model development. A useful evaluation framework has five layers.

First, measure task performance. Classification may use precision, recall, F1, area under the precision–recall curve or cost-weighted error. Generation may use factuality, completeness, citation correctness, answer relevance and human ratings. Automatic metrics such as BLEU or ROUGE can compare surface overlap, but overlap is not equivalent to truth or usefulness.

Second, test calibration. A probability is calibrated when predictions assigned a stated confidence level are correct at approximately that frequency over comparable cases. Reliability diagrams and expected calibration error can help diagnose miscalibration, although calibration estimates depend on binning and sample size.

Third, evaluate robustness. Test changes in terminology, document format, language, input length, spelling, temporal period and adversarial instructions. Conduct temporal validation where future deployment data differ from historical training data.

Fourth, assess fairness. Compare error rates and access to service across relevant groups, while recognising that group definitions and labels may themselves be sensitive or incomplete. Equal performance is not always simultaneously achievable under differing base rates, so fairness criteria require explicit normative and legal justification.

Fifth, evaluate operational outcomes. Measure latency, availability, review time, escalation rates, user satisfaction, energy use and total cost. A model with higher benchmark accuracy may be inferior if it is too slow, expensive or difficult to govern.

Hallucination and Groundedness

Hallucination is the production of content unsupported by available evidence or inconsistent with reality. It may involve invented citations, altered numbers, fabricated entities or unjustified causal claims. Groundedness evaluation asks whether each material claim is supported by authoritative evidence. Mitigations include constrained generation, retrieval, citation checking, abstention, structured outputs and human review. None eliminates hallucination completely.

Abstention is an important capability. A system should be able to state that evidence is insufficient, the question is outside scope or a human decision is required. A refusal or escalation policy should be designed according to risk, not simply maximised because excessive refusal can also harm users.

7. Responsible, Secure and Economically Sustainable Deployment

Privacy and Data Governance

Text frequently contains personal, confidential and regulated information. Governance should define lawful purpose, data minimisation, retention, access control, encryption, auditability and deletion procedures. Sensitive text should not be sent to an external model provider without an approved legal and security basis. De-identification can reduce risk but is not automatically irreversible; names, dates, locations and writing style may enable re-identification.

Security Threats

Prompt injection occurs when untrusted content attempts to override system instructions. Indirect prompt injection is especially relevant to RAG: a retrieved document may contain instructions aimed at the model rather than information for the user. Other risks include data poisoning, model extraction, insecure tool calls, excessive permissions, secret disclosure and denial-of-service through oversized inputs.

Defences include separating instructions from data, treating retrieved text as untrusted, allow-listing tools, limiting permissions, validating tool arguments, filtering outputs, monitoring unusual behaviour and conducting red-team tests. Security controls must be implemented outside the model as well as in prompts.

Copyright, Provenance and Accountability

Training and retrieval data may be subject to copyright, licence and contractual restrictions. Organisations should document data provenance, model versions, prompts, retrieved sources and material human interventions. Generated text should not be represented as independently verified authorship. Accountability remains with the deploying organisation and responsible professionals, not with the model.

Cost and Capacity Planning

Cost includes model usage, embedding generation, vector storage, data preparation, evaluation, monitoring, human review, security and failure remediation. A simple estimate for variable inference cost is:

Total variable cost = number of requests × average billable units per request × unit price.

The billable unit may be a token, character, image or compute interval, depending on the provider. This estimate excludes fixed engineering and governance costs and must be checked against the provider’s pricing rules. Caching, batching, smaller models, prompt compression and selective routing can reduce cost, but not when they undermine required accuracy or privacy.

8. A Professional Design Framework

A responsible project begins with the decision, not the model. Define the user, decision, acceptable error, evidence requirement, escalation route and harm from failure. Establish a data inventory and legal basis. Create a representative evaluation set before tuning the system. Select the simplest method that satisfies the requirement: a keyword rule or linear classifier may be preferable to an LLM for a stable, auditable task.

Next, design the data and model pipeline. Specify preprocessing, chunking, embedding model, retrieval strategy, prompt template, output schema, tools and permissions. Keep model-generated content separate from authoritative records. Version all components so results can be reproduced.

Pilot with shadow deployment where the system produces recommendations without affecting decisions. Compare it with existing practice, inspect subgroup performance and collect expert feedback. Introduce human review proportional to risk. After release, monitor drift, error reports, retrieval failures, cost, latency, privacy incidents and changes in source documents. Establish a retirement or rollback plan.

9. Illustrative Case: Internal Policy Intelligence System

Background

A fictional multinational engineering firm maintains thousands of safety, procurement and quality documents. Employees spend considerable time locating the current policy applicable to a project. Older documents remain in shared folders, and terminology differs among regions.

Problem

Management proposes an LLM chatbot that answers policy questions. The main risks are citing obsolete procedures, exposing documents across regional permissions and presenting an interpretation as mandatory policy.

Analysis

The team defines success as retrieving the current authorised source, answering routine questions with traceable citations and escalating ambiguous or high-risk questions. It creates a benchmark of 300 questions, including ordinary queries, conflicting versions, restricted documents, rare equipment names and deliberately misleading instructions. Each question receives expert relevance and answer-groundedness judgments.

Approach

The system uses document metadata containing owner, effective date, jurisdiction and access group. Documents are versioned and obsolete items are excluded by default. A hybrid lexical and vector retriever produces candidates, followed by a reranker. Permission checks occur before context construction. The model is instructed to answer only from supplied sources, show document identifiers and dates, distinguish quotation from interpretation and abstain when evidence conflicts. A separate rules engine handles effective-date filtering. High-risk safety questions require a link to the policy owner and human confirmation.

The team measures retrieval recall@5, citation correctness, answer groundedness, latency and cost per resolved question. It tests prompt injection in documents and attempts to access restricted policies. Logs record the query, permitted sources, model version and escalation outcome while excluding unnecessary personal information.

Outcome

In the pilot, routine retrieval questions are resolved more quickly, but the system performs poorly on ambiguous questions involving multiple jurisdictions. Rather than expanding the model’s authority, the organisation adds a jurisdiction-selection step and routes unresolved cases to specialists. The result is a narrower but more defensible service.

Learning Points

The case demonstrates that successful generative analytics depends on information governance, retrieval quality, access control and workflow design. A larger model alone would not solve obsolete documents or unclear policy ownership. Evaluation must include failure cases, and abstention can be a sign of quality rather than weakness.

Challenges and Contemporary Developments

Current systems increasingly support multimodal input, long contexts, structured tool calls, smaller specialised models and local deployment. Synthetic data can expand training sets, but synthetic records may reproduce bias or fail to represent rare events. Agentic systems can plan and execute multistep actions, increasing productivity and also increasing the consequences of permission errors and uncontrolled tool use.

Long context does not guarantee that a model will use every relevant passage. Model compression and edge deployment may improve privacy and latency, but can reduce capability. Open-weight models provide inspectability and deployment flexibility, while shifting responsibility for hosting security, licensing, updates and evaluation to the organisation. Regulation is developing unevenly across jurisdictions; therefore, compliance teams should map use cases to applicable privacy, sectoral, employment, safety and AI governance requirements.

A mature analytics function treats generative AI as a socio-technical system. It combines model evaluation with data management, process redesign, user training, security engineering and accountability. The central professional question is not “Can a model generate an answer?” but “What evidence, controls and responsibility make this answer appropriate for this decision?”

Chapter Summary

Text analytics progresses from lexical features such as TF–IDF to contextual embeddings and transformer-based foundation models. NLP supports classification, extraction, clustering, sentiment analysis, semantic search and question answering. Embeddings enable similarity-based retrieval, but their meaning depends on the model, data and evaluation design.

Generative analytics produces summaries, explanations, code, queries and other artefacts. RAG can connect an LLM to current organisational evidence, but retrieval errors, conflicting sources, prompt injection and unsupported synthesis remain important risks. Prompting, fine-tuning and tool use serve different purposes and should be selected deliberately.

Reliable deployment requires task-specific evaluation, calibration, robustness testing, fairness analysis, groundedness checks and operational measurement. Privacy, provenance, security, cost and human accountability are integral design requirements. The most defensible systems use appropriate evidence, enforce permissions outside the model, support abstention and maintain human oversight for consequential decisions.

Key Terms

Artificial intelligence: Computational systems that perform tasks associated with human intelligence.

Natural language processing: Methods for analysing and generating human language computationally.

Token: A unit processed by a language model, such as a word, subword or character sequence.

TF–IDF: A term-weighting method combining within-document frequency with inverse corpus frequency.

Embedding: A numerical vector representation intended to encode useful relationships.

Cosine similarity: A measure of angular similarity between non-zero vectors.

Contextual representation: A representation whose value depends on surrounding tokens.

Transformer: A neural architecture based primarily on attention mechanisms.

Attention: A mechanism that assigns learned weights to relationships among sequence elements.

Foundation model: A broadly trained model adaptable to multiple downstream tasks.

Large language model: A large neural model trained to represent and generate language.

Generative AI: AI that produces new content, predictions or structured artefacts.

Retrieval-augmented generation: Generation conditioned on information retrieved from an external source.

Hallucination: Unsupported, fabricated or factually incorrect generated content.

Groundedness: The extent to which an output is supported by authoritative evidence.

Prompt injection: Untrusted input attempting to manipulate model instructions or behaviour.

Calibration: Agreement between predicted confidence and observed correctness.

Abstention: Declining to answer or escalating when evidence or confidence is insufficient.

Model drift: Degradation caused by changes in data, language, behaviour or operating conditions.

Human-in-the-loop: A design in which human review or intervention is integrated into the workflow.

Review Questions

Short Answer Questions

1. What is the main difference between TF–IDF and contextual embeddings?

2. Define precision and recall and explain when they may conflict.

3. What problem does retrieval-augmented generation attempt to address?

4. Why is prompt injection a security concern in a RAG system?

5. What is meant by model abstention?

Long Answer Questions

1. Explain the transformer attention mechanism and discuss why positional information is required.

2. Compare prompting, fine-tuning, RAG and tool use for enterprise analytics.

3. Develop an evaluation framework for an LLM-based document classification system.

4. Discuss privacy, provenance, fairness and accountability issues in generative analytics.

5. Explain how an organisation should manage cost, quality and risk when deploying language models.

Analytical / Application Questions

1. Design a hybrid search and RAG architecture for a regulated organisation whose documents have different access permissions and effective dates. Identify controls and evaluation metrics.

2. A classifier has high accuracy on a balanced test set but misses most rare safety incidents in production. Analyse possible causes and propose a revised evaluation and threshold strategy.

3. An LLM generates SQL queries over a confidential data warehouse. Specify a secure workflow that preserves analytical usefulness while preventing unauthorised access and incorrect aggregation.

Multiple Choice Questions with Answers

Question 1

Which measure answers the question: among predicted positive cases, how many were correct?

A. Recall

B. Precision

C. Accuracy

D. Specificity

Correct Answer: B. Precision

Question 2

What is the primary purpose of embeddings in semantic search?

A. To encrypt documents

B. To represent items in a space where useful relationships can be compared

C. To guarantee factual correctness

D. To remove all bias from text

Correct Answer: B. To represent items in a space where useful relationships can be compared

Question 3

In scaled dot-product attention, why is QKᵀ divided by √d_k?

A. To remove positional information

B. To reduce excessively large dot-product scores

C. To convert text into tokens

D. To calculate recall

Correct Answer: B. To reduce excessively large dot-product scores

Question 4

Which component should normally enforce access permissions in an enterprise RAG system?

A. Only the final generated sentence

B. The access-control and retrieval pipeline

C. The model’s writing style

D. The temperature parameter

Correct Answer: B. The access-control and retrieval pipeline

Question 5

What is a hallucination in generative AI?

A. A model refusing every question

B. A response unsupported by evidence or inconsistent with reality

C. A correctly cited quotation

D. A reduction in inference cost

Correct Answer: B. A response unsupported by evidence or inconsistent with reality

Question 6

Which evaluation measure is especially concerned with whether a relevant item appears within the first k retrieved results?

A. Recall@k

B. Training loss

C. Token temperature

D. Parameter count

Correct Answer: A. Recall@k

Question 7

Why might a safety-critical system prefer recall over precision?

A. Missing a dangerous case may be more harmful than investigating a false alarm

B. Recall always costs less

C. Precision cannot be calculated

D. Recall eliminates the need for human review

Correct Answer: A. Missing a dangerous case may be more harmful than investigating a false alarm

Question 8

Which approach is most appropriate for a critical numerical calculation performed by an LLM application?

A. Require the model to estimate it from memory

B. Use a validated calculator or analytical tool and verify its output

C. Increase temperature substantially

D. Remove all retrieved evidence

Correct Answer: B. Use a validated calculator or analytical tool and verify its output

Question 9

What is the main role of abstention?

A. To increase unsupported creativity

B. To avoid answering when evidence or confidence is insufficient

C. To guarantee zero latency

D. To replace data governance

Correct Answer: B. To avoid answering when evidence or confidence is insufficient

Question 10

Which is an example of indirect prompt injection?

A. A user changes a document’s font

B. A retrieved document contains instructions intended to override the system’s rules

C. A model uses cosine similarity

D. An analyst evaluates recall

Correct Answer: B. A retrieved document contains instructions intended to override the system’s rules