Natural Language Generation and Translation

Introduction

Natural language generation (NLG) is the computational production of human-readable language from data, representations, or communicative intentions. Machine translation (MT) is a specialized NLG task in which a system produces a text in one language that preserves the meaning and function of a source text in another language. Related applications include summarization, dialogue, report generation, question answering, data-to-text generation, and controlled text rewriting.

Modern systems are predominantly neural. They use an encoder to represent input, a decoder to generate output, and—frequently—attention or cross-attention to connect generated tokens with relevant input information. Large language models extend this framework by predicting tokens from broad contextual representations. Their fluency, however, does not guarantee factuality, faithful translation, or appropriate style. A professional system must therefore be designed and evaluated as a socio-technical pipeline involving data, modeling, decoding, verification, interfaces, and human review.

This chapter examines the principles, algorithms, evaluation methods, risks, and deployment practices required to create reliable translation, summarization, and controlled-generation systems.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the architecture and training objective of neural NLG and MT systems.
2. Compare autoregressive, non-autoregressive, extractive, abstractive, and retrieval-augmented approaches.
3. Apply decoding methods such as greedy search, beam search, sampling, and constrained decoding.
4. Evaluate fluency, adequacy, coverage, factuality, controllability, and human usefulness.
5. Diagnose hallucination, omission, mistranslation, bias, and unsafe generation.
6. Design a human-in-the-loop workflow for a high-stakes language application.

1. The NLG Problem

Let an input representation be denoted by x and an output sequence by y = (y₁, y₂, ..., y_T), where T is the number of output tokens. An autoregressive generator models the conditional probability of the sequence as:

Here, y_<t is the sequence generated before position t. The factorization follows the chain rule of probability and is applicable when the system generates tokens sequentially. During training, the observed previous tokens are normally supplied to the model; this is called teacher forcing. During inference, the model must condition on its own earlier outputs, creating the possibility of error accumulation.

The standard loss for a dataset of paired examples is negative log-likelihood, or cross-entropy:

θ denotes model parameters and D the training dataset. The loss is dimensionless when probabilities are used inside the logarithm. Averaging over tokens or examples is often used in implementation. Minimizing this objective encourages probability mass on reference tokens, but it does not directly optimize factuality, usefulness, or human preference.

Core terminology

• Token: a unit processed by a language model, such as a word, subword, character, or byte.
• Source: the input text or structured information.
• Reference: a human-produced target considered acceptable for evaluation.
• Hypothesis: the system-generated output.
• Adequacy: preservation of source meaning.
• Fluency: grammaticality and naturalness in the target language.
• Faithfulness: consistency with the source or supplied evidence.
• Hallucination: unsupported, fabricated, or contradicted content.
• Controllability: the degree to which specified attributes are reliably followed.
• Grounding: generation constrained by trusted input evidence.

2. Architectures for Generation and Translation

2.1 Encoder–decoder transformers

In a transformer encoder–decoder, the encoder processes the source using self-attention. The decoder uses masked self-attention over previously generated tokens and cross-attention over encoder representations. Each attention head computes compatibility between a query and keys, then forms a weighted combination of values:

Q, K, and V are query, key, and value matrices; d_k is the key-vector dimension. The scaling factor reduces excessively large dot products and helps stabilize optimization. Self-attention connects tokens within one sequence, whereas cross-attention connects target generation to the source.

The encoder–decoder design is natural for translation and conditional generation. Decoder-only models instead predict continuations from a prompt and can perform translation or summarization through instruction prompting or fine-tuning. Encoder-only models are generally better suited to representation tasks, classification, and retrieval than unconstrained generation.

2.2 Translation paradigms

Rule-based MT uses dictionaries, grammatical rules, and manually engineered transfer procedures. It offers interpretability and predictable terminology but is expensive to maintain and weak on linguistic variation.

Statistical MT learns translation correspondences from aligned corpora. Phrase-based systems search for probable target sequences using translation and language models. They enabled data-driven translation but often produced fragmented output and required substantial feature engineering.

Neural MT learns continuous representations and translates using an end-to-end neural model. It usually provides better fluency and long-range modeling, although it can generate confident errors, especially for rare names, domain terms, and low-resource languages.

Multilingual MT shares parameters across languages. Positive transfer can improve low-resource translation, but unrelated languages, scripts, or domain mismatches may cause interference. Language tags, adapters, routing, and carefully balanced data can improve performance.

2.3 Summarization

Extractive summarization selects spans or sentences from the source. It tends to preserve source wording and is comparatively easy to verify, but it may be redundant or unable to synthesize information. Abstractive summarization generates new wording and can be concise and coherent, but it introduces risks of omission, distortion, and unsupported claims.

A robust summarizer should distinguish salience from factuality. A sentence can be important yet incorrectly generated. Conversely, a factually correct sentence may be irrelevant to the user’s purpose. In high-stakes settings, extractive evidence links, source citations, and sentence-level verification are valuable safeguards.

3. Training and Adaptation

A production model may be pretrained on large text collections and then adapted using supervised fine-tuning on task-specific examples. Instruction tuning teaches a model to follow natural-language requests. Preference optimization or reinforcement learning from human feedback can improve helpfulness and style, but preference data may reward persuasive fluency over truth.

Domain adaptation is necessary when the target application contains specialized vocabulary, legal formulations, medical terminology, or organizational style. Useful methods include:

• continued pretraining on permitted domain text;
• supervised fine-tuning with representative parallel or task examples;
• parameter-efficient adapters or low-rank updates;
• terminology dictionaries and translation memories;
• retrieval of authoritative documents at inference time;
• prompt templates with explicit output schemas.

Data quality is often more important than raw volume. Training and test splits must prevent near-duplicate leakage. Parallel corpora should be checked for alignment, language identification, personal information, licensing, and machine-generated contamination. For summarization, references should reflect the intended audience, length, and factual standards.

4. Decoding: From Probabilities to Text

A model produces a probability distribution, but an application needs a single output. Decoding is therefore a decision procedure, not a trivial implementation detail.

4.1 Greedy and beam search

Greedy decoding selects the highest-probability token at every step. It is fast but can make a locally attractive choice that prevents a better complete sequence. Beam search retains the best B partial hypotheses, where B is the beam width. A length-normalized objective is commonly used because the sum of log probabilities naturally favors short sequences:

T is output length and α is a tunable length-normalization parameter. This is a decoding heuristic, not a universally optimal correction. Larger beams increase computation and do not necessarily improve quality; they can increase generic or repetitive outputs.

4.2 Sampling

Sampling generates a token from the model distribution rather than always selecting the maximum. Temperature modifies the distribution:

z_i is the logit for token i, and τ > 0 is temperature. Lower temperature makes the distribution sharper; higher temperature increases diversity. Top-k sampling restricts choices to the k highest-probability tokens. Nucleus, or top-p, sampling selects the smallest candidate set whose cumulative probability is at least p. Sampling is useful for creative generation but can be unsuitable where exactness is required.

4.3 Constraints and structural control

Controlled generation may require a fixed language, terminology, format, length range, reading level, sentiment, or prohibited-content policy. Control mechanisms include:

• lexical constraints requiring specified words or phrases;
• constrained beam search;
• finite-state grammars for valid formats;
• JSON or schema-constrained decoding;
• copy mechanisms for names and numbers;
• terminology-aware reranking;
• post-generation validators;
• retrieval and citation requirements.

Prompt instructions alone are not reliable guarantees. A system that must output valid structured data should use a parser, validator, retry policy, and safe fallback. A system producing medical or legal text should not treat style control as evidence of correctness.

5. Evaluation of Generation and Translation

No single metric measures all relevant qualities. Evaluation should combine automatic metrics, targeted tests, expert review, and operational monitoring.

5.1 Translation metrics

BLEU measures modified n-gram precision with a brevity penalty. A simplified corpus-level form is:

Here, p_n is modified n-gram precision, w_n are weights whose sum is 1, and BP is the brevity penalty. BLEU is useful for comparing systems on the same benchmark, but it can undervalue valid paraphrases and is less informative for a single short sentence.

COMET-like learned metrics use multilingual representations and human judgment signals to estimate translation quality. They often correlate better with human assessment than surface-overlap metrics, but they inherit biases from their training data and should not be treated as ground truth. Human evaluation should separately assess adequacy, fluency, terminology, named entities, and harmful errors.

5.2 Summarization metrics

ROUGE measures overlap between generated and reference n-grams or sequences. It is useful when references share wording with acceptable summaries, but it may penalize legitimate paraphrases. Semantic similarity metrics can detect meaning overlap, but similarity alone does not establish factual entailment.

Factuality evaluation should ask whether each claim is supported by the source. Methods include expert annotation, entailment models, question-answering consistency checks, citation verification, and claim extraction followed by evidence retrieval. These methods can fail on numerical reasoning, negation, ambiguity, and domain-specific knowledge, so sampled human auditing remains necessary.

5.3 Quality dimensions and test design

A mature evaluation report should include:

For pairwise human evaluation, evaluators compare two outputs under a defined rubric. Inter-rater agreement should be reported where feasible, since disagreement may reveal ambiguous instructions or genuinely difficult cases. Evaluation sets should include ordinary examples and challenge cases: negation, dates, quantities, idioms, code-switching, rare names, long documents, and contradictory evidence.

6. Hallucination, Factuality, and Reliability

Hallucination has several forms. Intrinsic hallucination contradicts the source; extrinsic hallucination adds information not supported by the source. Translation may hallucinate by deleting a qualification, changing a number, inventing a fluent phrase, or substituting a culturally familiar entity. Summarization may merge events, assign an action to the wrong person, or present a possibility as a fact.

Risk increases when the model lacks relevant evidence, when decoding rewards generic fluency, when prompts are underspecified, and when users overtrust confident language. Retrieval-augmented generation can reduce unsupported claims by supplying evidence, but retrieval errors, stale documents, and incorrect synthesis remain possible.

Reliability practices include:

1. Retrieve from versioned, authoritative sources.
2. Display evidence or citations where users need verification.
3. Require the model to abstain when evidence is insufficient.
4. Validate names, dates, units, and numerical values with deterministic tools.
5. Use terminology constraints for specialized translation.
6. Separate generation from verification.
7. Log prompts, model versions, retrieved evidence, outputs, and reviewer actions.
8. Escalate high-impact or low-confidence cases to qualified humans.

A confidence score from the model is not automatically a calibrated probability of correctness. Calibration must be measured against labeled outcomes. Selective prediction can be evaluated by allowing the system to abstain and reporting quality as the accepted fraction changes.

7. Human Review and Governance

Human review is most effective when designed around identifiable failure modes. Reviewers need the source, generated text, evidence, terminology guidance, and a structured error taxonomy. They should not merely approve fluent text. For translation, bilingual review is essential; for specialized content, subject-matter expertise may matter more than general language expertise.

A practical workflow is:

1. classify the document’s risk and intended use;
2. generate a draft with versioned model settings;
3. run automated checks for format, terminology, entities, numbers, and prohibited content;
4. route uncertain or high-risk cases to reviewers;
5. record corrections and error categories;
6. sample accepted outputs for post-deployment audit;
7. update data, prompts, constraints, or routing policies based on evidence.

Governance must address privacy, copyright, licensing, bias, accessibility, and accountability. Personal data should be minimized and protected. Organizations should document the source and permissions of training and retrieval data. Users should be told when text is machine-generated where that fact affects informed decision-making. Automated translation must not remove a person’s access to qualified interpretation in contexts such as healthcare, immigration, courts, or emergency services.

8. Applied Case: Multilingual Public-Health Bulletin System

This is an illustrative case. A public-health agency wants to translate weekly disease-prevention bulletins from English into six languages and produce short summaries for mobile alerts.

Background and problem

The bulletins contain dates, dosage instructions, geographic names, uncertainty statements, and links to official guidance. Earlier manual translation was accurate but slow. A pilot neural system was fast but occasionally dropped a negation or changed a numerical interval. A fluent but incorrect alert could cause direct harm.

Analysis

The team defines two separate tasks: faithful translation of the full bulletin and controlled summarization for alerts. The risk analysis classifies medication quantities, dates, warnings, and eligibility conditions as critical fields. A glossary is created and approved by public-health experts. A test set is stratified by language, topic, and error type, including negation, numbers, names, and low-resource language varieties.

Approach and solution

The system retrieves the current approved bulletin and uses a domain-adapted translation model with terminology constraints. Named entities, dates, and numerical expressions are extracted before generation and compared after generation. The summarizer is instructed to use only retrieved source evidence and to include a citation link. It must return a structured object containing the alert, source span identifiers, language, and review status.

Automated checks reject outputs when critical numbers differ, glossary terms are absent, required warnings are missing, or the structured format is invalid. Every alert is reviewed by a bilingual language specialist and a public-health reviewer. Low-confidence cases are translated manually. The system stores the source version, model version, evidence, output, edits, and approval decision.

Outcome

The pilot reduces drafting time while retaining human approval for all public-facing alerts. Evaluation shows that overall fluency improves, but the most important improvement is a reduction in critical-field errors. Review data also reveals that one language requires a larger glossary and different sentence segmentation. The agency therefore treats deployment as an iterative quality-management process rather than a one-time model launch.

Learning points

The case illustrates five principles: speed is not the same as safety; critical facts require deterministic checks; retrieval does not replace review; evaluation must focus on consequential errors rather than only average scores; and human corrections are valuable evidence for system improvement.

9. Contemporary Directions and Best Practices

Current research explores multilingual foundation models, speech-to-speech translation, document-level context, multimodal translation, retrieval-grounded generation, efficient adaptation, and agentic verification. Document-level models attempt to preserve discourse consistency, pronoun resolution, and terminology across paragraphs. Multimodal systems incorporate images, layout, tables, and speech, but they introduce additional grounding and accessibility challenges.

Best practice is to match model freedom to task risk. Creative marketing text may permit sampling and broad stylistic variation. A financial disclosure, medication instruction, or legal clause requires constrained generation, evidence, deterministic validation, and expert review. Teams should establish quality thresholds before deployment, monitor drift after deployment, and maintain rollback procedures.

A useful engineering principle is defense in depth: no single metric, prompt, model, or reviewer is sufficient. Reliable NLG combines high-quality data, suitable architecture, controlled decoding, targeted evaluation, factuality checks, transparent interfaces, and accountable human judgment.

Chapter Summary

Natural language generation converts representations or instructions into text, while machine translation generates target-language text that should preserve the source meaning. Autoregressive models factor sequence probability token by token and are commonly trained with cross-entropy, but likelihood optimization alone does not ensure factuality or usefulness. Transformer encoder–decoder architectures support translation through self-attention and cross-attention; decoder-only models can also perform conditional generation through prompting or fine-tuning.

Greedy search, beam search, temperature sampling, top-k, top-p, and constrained decoding make different trade-offs between speed, diversity, structure, and reliability. Translation and summarization require multidimensional evaluation covering fluency, adequacy, coverage, terminology, faithfulness, controllability, robustness, and safety. BLEU, ROUGE, and learned metrics are useful indicators rather than complete measures of quality.

Hallucination is a central deployment risk. Retrieval, abstention, terminology control, numerical validation, citations, and human review reduce but do not eliminate it. High-stakes systems require governance for privacy, bias, licensing, accessibility, transparency, and accountability. The most dependable applications use a documented human-in-the-loop workflow and treat monitoring and evaluation as continuing operational activities.

Key Terms

Autoregressive generation: Generating each token conditional on earlier tokens and the input.

Adequacy: Preservation of the source meaning in a translation.

Abstractive summarization: Producing a summary with newly generated wording.

Beam search: Decoding that retains several high-scoring partial sequences.

Cross-attention: Attention connecting decoder states to encoder representations.

Controlled generation: Generation subject to specified attributes, constraints, or formats.

Cross-entropy: A loss measuring the negative log probability assigned to observed targets.

Faithfulness: Consistency of generated content with its source or evidence.

Fluency: Naturalness, grammaticality, and readability of output.

Hallucination: Unsupported, fabricated, or contradicted generated content.

Human-in-the-loop: A workflow in which people review, correct, approve, or override system outputs.

Machine translation: Automatic translation between human languages.

Nucleus sampling: Sampling from the smallest token set reaching a specified cumulative probability.

Retrieval-augmented generation: Generation supported by documents retrieved at inference time.

ROUGE: A family of reference-overlap metrics commonly used for summarization.

Self-attention: Attention among tokens in the same sequence.

Teacher forcing: Training with the reference prefix supplied as previous context.

Terminology constraint: A requirement to use specified words or phrases.

Token: A model-processing unit such as a word, subword, character, or byte.

Review Questions

Short Answer Questions

1. What is the difference between adequacy and fluency in machine translation?
2. Why can cross-entropy training fail to guarantee factual summarization?
3. What is the purpose of temperature in sampling-based decoding?
4. Distinguish intrinsic from extrinsic hallucination.
5. Why are terminology constraints important in specialized translation?

Long Answer Questions

1. Explain the encoder–decoder transformer architecture and the roles of self-attention and cross-attention.
2. Compare greedy decoding, beam search, and nucleus sampling, including appropriate use cases and risks.
3. Discuss why BLEU and ROUGE should be combined with human and targeted factuality evaluation.
4. Describe a complete human-in-the-loop workflow for a high-stakes NLG application.
5. Analyze the ethical, privacy, licensing, and accountability issues involved in deploying multilingual generation systems.

Analytical / Application Questions

1. Design an evaluation set for a medical translation system. Identify critical error categories, metrics, reviewers, and deployment thresholds.
2. A summarizer is fluent and achieves high ROUGE but frequently invents causes for reported events. Diagnose the problem and propose model, decoding, evaluation, and workflow interventions.
3. An organization wants a system that converts database records into compliant financial reports. Specify suitable controls for data grounding, structure, numerical accuracy, abstention, auditing, and human approval.

Multiple Choice Questions with Answers

Question 1

Which expression correctly represents the autoregressive factorization of a generated sequence?

A. P(y|x) = ∑ P(y_t|x)

B. P(y|x) = ∏ P(y_t|y_{<t}, x)

C. P(y|x) = P(x|y)

D. P(y|x) = max_t P(y_t|x)

Correct Answer: B. P(y|x) = ∏ P(y_t|y_{<t}, x)

Question 2

What is the primary purpose of cross-attention in an encoder–decoder transformer?

A. To connect decoder states with encoder representations

B. To remove all source tokens before decoding

C. To calculate corpus-level BLEU

D. To force every output sequence to have equal length

Correct Answer: A. To connect decoder states with encoder representations

Question 3

Why is length normalization sometimes used in beam-search scoring?

A. Beam search otherwise always prefers longer sequences

B. Log-probability sums naturally tend to favor shorter sequences

C. It converts translation into extractive summarization

D. It guarantees factual correctness

Correct Answer: B. Log-probability sums naturally tend to favor shorter sequences

Question 4

What does increasing the temperature generally do during token sampling?

A. Makes the distribution sharper and less diverse

B. Makes the distribution more diverse and less concentrated

C. Forces the model to copy the source exactly

D. Eliminates the need for evaluation

Correct Answer: B. Makes the distribution more diverse and less concentrated

Question 5

Which is an example of extrinsic hallucination in summarization?

A. Repeating a fact explicitly stated in the source

B. Changing a source statement from positive to negative

C. Adding a plausible but unsupported event not present in the source

D. Shortening a sentence without changing its meaning

Correct Answer: C. Adding a plausible but unsupported event not present in the source

Question 6

What is a central limitation of BLEU?

A. It cannot use reference translations

B. It may penalize valid paraphrases that have low n-gram overlap

C. It directly verifies factual entailment

D. It is designed only for speech recognition

Correct Answer: B. It may penalize valid paraphrases that have low n-gram overlap

Question 7

Which control is most appropriate when an output must conform to a strict machine-readable schema?

A. Unrestricted high-temperature sampling

B. Schema-constrained decoding followed by validation

C. Removing all input evidence

D. Evaluating only grammatical fluency

Correct Answer: B. Schema-constrained decoding followed by validation

Question 8

Why should names, dates, and numerical expressions be checked deterministically in high-stakes translation?

A. They are always irrelevant to meaning

B. Fluent neural generation can still alter or omit critical fields

C. Deterministic checks improve creative diversity

D. Such checks replace all human review in every context

Correct Answer: B. Fluent neural generation can still alter or omit critical fields

Question 9

Which statement best describes retrieval-augmented generation?

A. It generates text without using any external evidence

B. It retrieves relevant documents at inference time to support generation

C. It is a metric based only on word overlap

D. It is limited to extractive summarization

Correct Answer: B. It retrieves relevant documents at inference time to support generation

Question 10

What is the most appropriate response when a high-stakes system lacks sufficient evidence for a requested claim?

A. Generate a confident answer anyway

B. Increase temperature until an answer appears

C. Abstain or escalate the case for human review

D. Delete the source document

Correct Answer: C. Abstain or escalate the case for human review