Text Analytics and Contemporary AI

Introduction and Context

Text is one of the largest and most complex sources of organisational data. Customer reviews, support tickets, legal documents, clinical notes, social-media posts, research articles and internal reports contain valuable information, but they are not naturally arranged in rows and columns. Text analytics converts language into representations that computers can process for description, prediction, retrieval and decision support.

This chapter examines a modern text-analytics workflow using R. It begins with classical natural language processing (NLP)—tokenization, normalization, sentiment analysis, topic modeling and text classification—and then develops contemporary approaches based on vector embeddings, transformer models, generative artificial intelligence (AI) and large language models (LLMs). Particular attention is given to retrieval-augmented generation (RAG), evaluation, privacy, bias, explainability and hallucination risk.

Text analytics is not simply the application of a function to a character vector. It is a modelling process involving a research question, a corpus, linguistic assumptions, representation choices, validation procedures and governance controls. A sophisticated model can produce misleading results if the corpus is biased, labels are unreliable, or evaluation does not reflect operational use.

Learning Objectives

After completing this chapter, the reader should be able to:

1. Design a reproducible text-processing pipeline in R.

2. Explain tokenization, n-grams, term-frequency representations and TF-IDF.

3. Apply and critically evaluate sentiment analysis and topic modeling.

4. Compare sparse representations with dense embeddings and transformer-based models.

5. Build and assess text-classification systems.

6. Explain how R can interact with generative AI and LLM services.

7. Design a retrieval-augmented workflow and identify its failure points.

8. Evaluate hallucination, bias, privacy, security and regulatory risks.

1. Text as Data

A corpus is a collection of documents. A document may be an email, review, paragraph, web page or transcript. A document-term matrix (DTM) records the relationship between documents and terms. In a simple binary DTM, an entry is 1 when a term occurs in a document and 0 otherwise. In a count DTM, it records the number of occurrences.

Common preprocessing operations include:

• Case normalization: converting text to a consistent case.
• Punctuation and whitespace handling: removing or standardizing non-linguistic characters when appropriate.
• Tokenization: splitting text into meaningful units such as words, sentences, subwords or entities.
• Stop-word treatment: removing frequent function words, but only when they do not carry analytical meaning.
• Stemming: reducing words to crude roots, such as “connect,” “connected” and “connecting.”
• Lemmatization: mapping inflected forms to a linguistically meaningful lemma.
• Negation handling: preserving expressions such as “not useful,” which can be damaged by indiscriminate preprocessing.
• De-identification: masking names, addresses, account numbers and other personal information.

Preprocessing is task-dependent. Removing stop words may help a broad topic model but harm sentiment analysis because words such as “not” can reverse polarity. Stemming may improve vocabulary matching but reduce interpretability. Reproducibility requires documenting every transformation and its version.

In R, the `tidytext` and `dplyr` ecosystems support token-based workflows, while `quanteda` provides efficient corpus, token and document-feature-matrix objects. A minimal tidy workflow is conceptually:

```r

library(dplyr)

library(tidytext)

words <- reviews |>

unnest_tokens(word, text) |>

anti_join(stop_words, by = "word") |>

count(document_id, word, sort = TRUE)

```

The code is only a starting point. Production pipelines should include language detection, encoding checks, duplicate removal, document identifiers, timestamps and audit logs.

2. Frequency, TF-IDF and N-grams

Raw frequency is useful for corpus description but is not necessarily evidence of importance. A term appearing in every document may be frequent but uninformative. Term frequency–inverse document frequency (TF-IDF) reduces the weight of terms common across the corpus.

For term \(t\) in document \(d\), a common formulation is:

\[

\mathrm{tfidf}(t,d)=\mathrm{tf}(t,d)\times \log\left(\frac{N}{\mathrm{df}(t)}\right),

\]

where \(\mathrm{tf}(t,d)\) is the term frequency, \(N\) is the number of documents, and \(\mathrm{df}(t)\) is the number of documents containing the term. The formula assumes non-negative term counts and a defined document collection. If a term occurs in every document, its inverse-document-frequency component is zero under this formulation. Software may use logarithmic smoothing, sublinear term frequency or normalization; the exact convention must therefore be recorded.

N-grams preserve limited word order. A unigram is one token, a bigram contains two consecutive tokens, and a trigram contains three. The phrase “credit card” is more informative than either word alone in many applications. N-grams increase dimensionality and may be sparse, so they should be selected according to the task and corpus size.

A major limitation of bag-of-words and TF-IDF is that they treat words as largely independent features. “The company acquired the supplier” and “the supplier acquired the company” share many terms despite having opposite subject–object relationships. Dense embeddings address some, but not all, of this limitation.

3. Sentiment Analysis and Emotion Detection

Sentiment analysis estimates affective orientation, often as positive, negative or neutral, or as a continuous score. There are three broad approaches:

1. Lexicon-based methods: words are matched to a dictionary containing polarity or emotion values.

2. Supervised classification: a model learns from labelled examples.

3. Transformer-based inference: a pretrained language model is adapted or prompted for the target task.

Lexicon methods are transparent and inexpensive. In R, packages such as `sentimentr` and `syuzhet` provide alternative approaches, while custom lexicons can be joined to token data. However, lexicons are sensitive to domain and context. “Unpredictable” may be negative in a service review but positive in a creativity study. Sarcasm, negation, code-switching and emojis create further difficulty.

A useful evaluation design is to create a stratified, manually labelled sample and compare model output with human labels. Accuracy alone can be misleading under class imbalance. For a binary classifier, precision is:

\[

\mathrm{Precision}=\frac{TP}{TP+FP},

\]

and recall is:

\[

\mathrm{Recall}=\frac{TP}{TP+FN},

\]

where TP, FP and FN denote true positives, false positives and false negatives. The F1 score is the harmonic mean of precision and recall:

\[

F_1=2\frac{\mathrm{Precision}\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}},

\]

provided the denominator is non-zero. These measures should be reported with the positive class explicitly defined. For operational use, the cost of missing a severe complaint may be much greater than the cost of reviewing a false alert.

4. Topic Modeling

Topic modeling is an unsupervised method for discovering recurring patterns of words. Latent Dirichlet allocation (LDA) represents each document as a mixture of topics and each topic as a probability distribution over words. In the generative interpretation, a document’s topic proportions are drawn from a document-level distribution, and each token is generated by selecting a topic followed by a word from that topic. The Dirichlet prior encourages particular forms of sparsity, but the model’s assumptions are approximations rather than facts about how authors actually write.

The number of topics, \(K\), must be selected. Statistical criteria such as perplexity, topic coherence and held-out likelihood can assist, but human interpretability and usefulness for the decision problem are equally important. Topics are not automatically meaningful labels; analysts must inspect high-probability and highly associated words, representative documents and stability across random seeds or samples.

In R, `topicmodels`, `stm` and `tidytext` support topic analysis. The Structural Topic Model (STM) can incorporate document metadata, such as date or source, and estimate how topic prevalence varies with covariates. This is valuable in longitudinal studies but does not establish causal effects merely because a covariate is included.

Topic models can fail when documents are very short, vocabulary is highly domain-specific, or topics are strongly correlated. Dynamic corpora may require time-aware analysis. Topic labels should be treated as analyst interpretations, not ground truth.

5. Text Classification

Text classification assigns documents to predefined categories such as complaint type, fraud status, urgency or scientific field. A conventional workflow is:

1. Define mutually understandable labels and annotation rules.

2. Sample and label data, measuring inter-annotator agreement where feasible.

3. Split data into training, validation and test sets without leakage.

4. Transform text into features.

5. Train a model and tune hyperparameters using the training data.

6. Evaluate once on an untouched test set.

7. Monitor performance after deployment.

A document split is safer than a random token split. When documents from the same customer, case or time period are related, grouped or temporal splitting may be required. Otherwise, near-duplicates can appear in both training and test data, producing overoptimistic results.

Useful models include multinomial Naive Bayes, logistic regression, support vector machines, random forests and gradient boosting. Linear models with TF-IDF are often strong baselines. Their coefficients can support interpretation, although correlated features complicate coefficient meaning. Class weights, resampling or threshold adjustment can address imbalance, but synthetic or duplicated minority data must be validated carefully.

For multiclass tasks, macro-averaged metrics give each class equal weight, whereas weighted averages reflect class prevalence. Confusion matrices reveal whether errors are concentrated in a particular category. Calibration analysis is important when predicted probabilities guide human or financial decisions.

6. Embeddings and Transformer Models

An embedding maps text to a numerical vector in a continuous space. Similar meanings may be represented by nearby vectors according to a chosen similarity measure, often cosine similarity:

\[

\cos(\theta)=\frac{x\cdot y}{\|x\|\|y\|},

\]

where \(x\) and \(y\) are non-zero vectors, \(x\cdot y\) is their dot product and the norms are Euclidean lengths. Cosine similarity is scale-insensitive, but “nearby” remains model- and domain-dependent.

Earlier word embeddings assign one vector to a word and therefore struggle with polysemy. Transformer encoders generate context-sensitive representations. Self-attention allows a token representation to incorporate information from other tokens in the sequence. Pretrained models can be fine-tuned for classification, or their embeddings can be indexed for semantic search.

Embedding systems introduce new concerns. Similarity does not guarantee factual equivalence. General-purpose models may perform poorly on specialist terminology, minority languages or local organisational language. Embeddings can encode social biases found in training data and may make sensitive attributes inferable. Dimensionality reduction plots such as PCA or UMAP are useful exploratory aids but can distort distances and should not be treated as definitive evidence of semantic structure.

R users can access transformer models through packages such as `torch`, `reticulate`, `text2vec`, and APIs or vector databases exposed through HTTP interfaces. An enterprise design should specify model version, embedding dimension, distance metric, chunking method and index-refresh policy.

7. Generative AI and LLM Integration with R

An LLM predicts likely continuations based on learned statistical representations. It can summarize, classify, extract structured fields, generate code and answer questions. It does not possess a guaranteed database of truth, and fluent language is not evidence of correctness.

R can integrate with LLMs through REST APIs, SDKs, local inference servers or Python interoperability using `reticulate`. A robust request should control the system instruction, user input, model identifier, maximum output length, temperature or sampling settings, timeout, retry policy and logging. Structured output should be validated against a schema rather than assumed to be valid because it resembles JSON.

Prompt engineering is useful but not a substitute for evaluation. Effective prompts define the task, permitted evidence, output format, uncertainty behaviour and examples where appropriate. Few-shot examples can improve consistency but may introduce hidden bias or consume context space. Lower randomness may improve reproducibility, yet deterministic-looking output can still be wrong.

LLM use cases should be divided into low-risk assistance, human-reviewed decision support and high-risk automated decisions. The model’s role, authority and escalation path must be explicit. Sensitive information should not be sent to an external provider without an appropriate legal, contractual and technical basis.

8. Retrieval-Augmented Generation

Retrieval-augmented generation combines information retrieval with language generation. A typical workflow is:

1. Collect authoritative documents.

2. Remove duplicates and apply access controls.

3. Split documents into semantically useful chunks.

4. Generate and store embeddings with metadata.

5. Retrieve relevant chunks for a user query.

6. Construct a prompt containing the query and retrieved evidence.

7. Generate an answer constrained to that evidence.

8. Return citations, confidence or an abstention message where appropriate.

Retrieval can be lexical, semantic or hybrid. Lexical retrieval is strong for exact names and identifiers; semantic retrieval is useful for paraphrases. Hybrid retrieval and reranking often provide better coverage. Chunk size is a design trade-off: small chunks improve precision but may remove context, whereas large chunks increase context and cost.

RAG does not eliminate hallucination. The retriever may fail to find the relevant document, retrieve contradictory material, expose information the user is not authorised to see, or provide outdated evidence. The generator may misread, combine or contradict the retrieved passages. Evaluation should therefore separate retrieval quality from answer quality. Measures may include recall of relevant evidence, citation correctness, answer faithfulness, abstention quality, latency and cost.

A secure RAG system applies document-level and row-level access controls before retrieval, not merely after generation. Prompt injection within a retrieved document is another threat: untrusted text may contain instructions designed to override the system’s task. Retrieved content should be treated as data, not authority.

9. Hallucination, Bias and Evaluation

A hallucination is an output that presents unsupported or false content as if it were reliable. Causes include incomplete context, ambiguous prompts, model uncertainty, training-data gaps, retrieval errors and pressure to produce an answer even when evidence is absent.

Risk reduction requires layered controls:

• restrict answers to approved evidence;
• require citations and verify them automatically where possible;
• allow “insufficient evidence” responses;
• use retrieval and generation test sets containing adversarial examples;
• conduct human review for high-impact outputs;
• log prompts, retrieved passages, model versions and outputs;
• monitor drift and incident rates;
• protect against prompt injection and data exfiltration.

Bias can arise from data representation, annotation, language variety, feature selection, model pretraining and deployment thresholds. Evaluation should be disaggregated by relevant, lawful and ethically justified groups. Fairness metrics may conflict, particularly where base rates differ; no single metric resolves the underlying social and institutional question.

Privacy risks include memorization, re-identification, sensitive inference and accidental inclusion of confidential text in prompts or logs. Data minimization, retention limits, encryption, access controls and redaction are essential. Depending on jurisdiction and use case, data-protection, intellectual-property, sectoral and emerging AI regulations may apply. Governance should include a risk assessment, accountable owner, vendor review, incident response and documentation such as model cards or system cards.

10. Illustrative Case: University IT-Service Analytics

Background

A university receives approximately 40,000 annual IT-service tickets. Managers want to identify recurring problems, route tickets automatically and detect urgent security-related language. Tickets include free text, service category, timestamps and resolution status.

Problem

Existing categories are inconsistent. A keyword rule identifies many routine password requests as security incidents, while some genuine phishing reports are missed. The university also must protect student and staff information.

Analysis

The analytics team first removes names, phone numbers, email addresses and credentials using deterministic patterns and named-entity checks. A sample of tickets is independently labelled by two trained analysts using a written rubric. Disagreements are reviewed, and the final dataset is split chronologically so that future performance is not estimated using past duplicates.

A TF-IDF logistic-regression model is established as a transparent baseline. A transformer classifier and a hybrid retrieval system are then evaluated. Metrics include macro-F1 for routing, recall for security reports, false-positive workload and calibration. Results are reported separately for short tickets, multilingual tickets and service categories.

Approach and Solution

The final design uses the linear model for routine routing and a high-recall screening model for possible security incidents. The latter does not close or escalate a case automatically; it sends candidates to a trained analyst. A RAG assistant retrieves approved troubleshooting articles and produces a draft response with citations. It cannot access restricted case notes, and it must state when no suitable article is found.

Outcome

The system reduces manual routing effort while preserving human control over security decisions. Its performance is monitored monthly, with new labels added for emerging software and attack patterns. The university records model versions, evidence citations and analyst overrides.

Learning Points

The case demonstrates that a strong solution is a socio-technical system rather than a single model. Baselines, careful labels, temporal validation, privacy protection, role-based access and escalation rules are as important as model architecture. Automation is most defensible where errors are reversible and human review is meaningful.

11. Best-Practice Design Pattern

A professional text-analytics project should:

1. Begin with a measurable decision or research question.

2. Establish data ownership, lawful use, retention and access rules.

3. Inspect language, duplication, missingness, imbalance and temporal structure.

4. Build an interpretable baseline before adopting a complex model.

5. Prevent leakage through document-, user- and time-aware splitting.

6. Evaluate with task-relevant metrics and subgroup analysis.

7. Preserve provenance for preprocessing, models, prompts and evidence.

8. Treat LLM output as a probabilistic artefact requiring verification.

9. Design abstention, escalation and correction processes.

10. Monitor deployed systems for drift, bias, security incidents and changing user behaviour.

Contemporary Developments

Current research is moving toward multimodal models, multilingual and domain-specific LLMs, parameter-efficient fine-tuning, retrieval over graphs and structured databases, smaller models deployable at the edge, and agents capable of tool use. These developments may improve capability but increase system complexity and the number of possible failure paths. Research questions include how to evaluate factuality reliably, how to measure the environmental and financial cost of inference, how to preserve privacy during adaptation, and how to establish accountability when many models and data sources interact.

Chapter Summary

Text analytics transforms unstructured language into representations that support description, prediction, retrieval and decision-making. Tokenization, normalization, n-grams, TF-IDF, lexicons, topic models and supervised classifiers remain useful foundations. Embeddings and transformer models capture richer contextual relationships but introduce model, bias and governance risks. LLMs can assist with generation, extraction and question answering, yet fluent output is not proof of truth. RAG improves access to organisational evidence but can still fail through poor retrieval, stale documents, access-control errors or unsupported generation. Reliable practice requires representative data, leakage-resistant evaluation, baseline comparison, human oversight, privacy protection, auditability and continuous monitoring.

Key Terms

Corpus: A collection of text documents used for analysis.

Tokenization: Division of text into units such as words, sentences or subwords.

Lemmatization: Reduction of an inflected word to its linguistic base form.

N-gram: A sequence of n consecutive tokens.

Document-term matrix: Matrix representing term-document relationships.

TF-IDF: Weighting that emphasizes terms frequent in a document but uncommon across documents.

Sentiment analysis: Estimation of affective polarity or emotion in text.

Lexicon: A dictionary associating terms with linguistic or sentiment properties.

Topic model: Unsupervised model identifying recurring word distributions or themes.

Text classification: Assignment of documents to predefined categories.

Embedding: Numerical vector representation of text.

Transformer: Neural architecture based substantially on attention mechanisms.

LLM: Large language model trained to model and generate language.

Prompt injection: Malicious or unintended instructions embedded in input or retrieved content.

RAG: Retrieval-augmented generation combining search with language generation.

Hallucination: Unsupported or false generated content presented as factual.

Precision: Proportion of predicted positives that are correct.

Recall: Proportion of actual positives that are detected.

Data leakage: Unintended use of information that would not be available at prediction time.

Model drift: Change in data or relationships that reduces deployed performance.

Review Questions

Short Answer Questions

1. Why is preprocessing task-dependent in text analytics?

2. Define TF-IDF and explain the role of document frequency.

3. Distinguish lexicon-based sentiment analysis from supervised classification.

4. What is an embedding, and why can cosine similarity be useful?

5. Why does RAG not eliminate hallucination?

Long Answer Questions

1. Explain the complete workflow for developing and evaluating a text-classification system in R.

2. Compare bag-of-words, TF-IDF, word embeddings and transformer representations.

3. Describe LDA topic modeling, including topic-number selection and interpretability limitations.

4. Design a secure RAG architecture for an organisation with confidential documents.

5. Discuss privacy, bias, accountability and regulatory issues in LLM-supported text analytics.

Analytical / Application Questions

1. A sentiment classifier has high accuracy but misses most negative reviews in an imbalanced dataset. Diagnose the problem and propose an evaluation and improvement plan.

2. Design an experiment to compare a TF-IDF logistic-regression classifier with a transformer classifier for multilingual support tickets.

3. A RAG assistant produces a confident answer citing an outdated policy. Identify the retrieval, generation and governance failures, and specify controls to prevent recurrence.

Multiple Choice Questions with Answers

Question 1

Which statement best describes TF-IDF?

A. It gives every word the same weight in every document

B. It increases the weight of terms common across all documents

C. It emphasizes terms frequent in a document but uncommon in the corpus

D. It converts every document into a sentiment score

Correct Answer: C. It emphasizes terms frequent in a document but uncommon in the corpus

Question 2

Which metric is most directly concerned with the proportion of actual positive cases detected?

A. Recall

B. Precision

C. Specificity

D. Accuracy

Correct Answer: A. Recall

Question 3

What is a principal limitation of a basic bag-of-words representation?

A. It cannot count word occurrences

B. It usually ignores most word order and context

C. It requires labelled data in every application

D. It cannot represent documents numerically

Correct Answer: B. It usually ignores most word order and context

Question 4

What is the primary purpose of topic modeling?

A. To discover recurring patterns of words without predefined labels

B. To encrypt confidential documents

C. To guarantee factuality of generated answers

D. To remove all rare words from a corpus

Correct Answer: A. To discover recurring patterns of words without predefined labels

Question 5

Why should a classification test set remain untouched during model tuning?

A. To increase the vocabulary size

B. To estimate generalisation on unseen data without tuning bias

C. To guarantee balanced classes

D. To eliminate the need for labels

Correct Answer: B. To estimate generalisation on unseen data without tuning bias

Question 6

What does an embedding represent?

A. A numerical vector representation of text

B. A legal agreement for data processing

C. A list of stop words only

D. A confusion matrix

Correct Answer: A. A numerical vector representation of text

Question 7

Which sequence best represents a typical RAG workflow?

A. Generate first, then delete the source documents

B. Retrieve relevant evidence, place it in context, and generate an answer

C. Translate every document into code before retrieval

D. Classify labels without using documents

Correct Answer: B. Retrieve relevant evidence, place it in context, and generate an answer

Question 8

Which is a realistic cause of LLM hallucination?

A. The model has guaranteed access to a verified database

B. The model generates unsupported content when evidence is incomplete

C. The model always refuses uncertain questions

D. The model only processes numerical data

Correct Answer: B. The model generates unsupported content when evidence is incomplete

Question 9

Why can random splitting be problematic for related text documents?

A. It may place near-duplicates or records from the same case in both training and test sets

B. It always reduces the number of classes to one

C. It prevents tokenization

D. It makes all models deterministic

Correct Answer: A. It may place near-duplicates or records from the same case in both training and test sets

Question 10

Which control is most appropriate for a high-impact automated text decision?

A. Remove all audit logs

B. Use human review, documented evaluation and an escalation process

C. Increase temperature to maximize creativity

D. Ignore subgroup performance

Correct Answer: B. Use human review, documented evaluation and an escalation process