Linguistic Foundations and Language Representation

Introduction

Natural Language Processing (NLP) is concerned with enabling computers to analyse, interpret, generate and interact through human language. Although modern systems often rely on large neural networks, successful NLP still depends on linguistic concepts. A model that represents words without recognising morphology may misunderstand inflection; a parser that ignores syntax may assign the wrong relationships; and a dialogue system that neglects pragmatics may produce a grammatically correct but socially inappropriate response.

Language is simultaneously a structured system and a social practice. It has sounds and writing systems, meaningful units, grammatical patterns, conceptual content, speaker intentions and discourse-level organisation. Computational language processing therefore requires representations at several levels rather than a single universal representation.

This chapter develops the principal linguistic foundations required for NLP and generative AI. It explains morphology, syntax, semantics, pragmatics, discourse, ambiguity and multilingual variation, and connects each area to computational methods such as tokenisation, parsing, embeddings, knowledge representation, machine translation and large language models.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish the principal levels of linguistic analysis.
2. Analyse words into morphemes and explain their computational importance.
3. Describe syntactic structures and compare major parsing approaches.
4. Explain lexical, compositional and distributional meaning.
5. Distinguish semantic ambiguity from syntactic and pragmatic ambiguity.
6. Explain how context, discourse and speaker intention affect interpretation.
7. Identify challenges created by multilingualism, code-switching and language variation.
8. Select suitable representations for practical NLP and generative-AI applications.
9. Evaluate the limitations and ethical risks of linguistic representations.

1. Language as a Multi-Level System

A useful abstraction is to view language as a set of interacting levels:

These levels are analytically distinct but not independent. The word unhappiness contains morphological information; its position in a sentence contributes syntactic information; its interpretation depends on semantic composition; and its practical force may depend on who says it, to whom and why.

NLP systems frequently simplify this structure. A text classifier may operate directly on subword tokens, while a dialogue system may construct explicit dialogue acts and discourse states. The appropriate representation depends on the task, available data, computational resources and consequences of error.

2. Morphology: The Structure of Words

Morphology studies how words are formed and how their parts express meaning or grammatical information. The smallest meaningful unit is a morpheme. In unhelpfulness, the morphemes are un-, help, -ful and -ness.

2.1 Free and bound morphemes

A free morpheme can occur independently, as in book or run. A bound morpheme must attach to another form, as in -ed, -s, un- or -tion. Affixes may be prefixes, suffixes, infixes or circumfixes, depending on the language.

Morphology includes:

• Inflection, which marks grammatical properties without normally creating a new lexical item: walk → walked, cat → cats.
• Derivation, which creates a new lexeme or changes word class: kind → unkind, teach → teacher.
• Compounding, which combines lexical units: toothbrush and data centre.
• Reduplication, found in many languages, where all or part of a form is repeated.
• Cliticisation, in which a grammatical element behaves partly like a word and partly like an affix.

Languages differ in morphological complexity. English has relatively limited inflectional morphology, whereas Turkish, Finnish and Swahili can encode several grammatical categories within a single orthographic word. Chinese has little inflectional morphology but relies heavily on word order, particles and context.

2.2 Computational morphology

Computational systems commonly perform four related operations:

• Tokenisation separates an input stream into processing units.
• Stemming removes selected affixes using heuristic rules. It may produce a non-word, such as reducing studies to studi.
• Lemmatization maps an inflected form to a dictionary base form, such as was → be.
• Morphological analysis identifies a lemma and grammatical features, such as walked → lemma walk, tense past.

A morphological analyser may represent a word as a feature bundle. For example:

walked → walk + V + TENSE:PAST

The arrow indicates analysis, not a mathematical equality. Such representations are useful for information retrieval, grammar checking, translation and language generation.

Subword tokenisation is particularly important in contemporary language models. A tokenizer may represent unfamiliar words as smaller units, allowing a model to process names, technical terms and productive word formations. However, subwords are engineering units, not necessarily morphemes. The segmentation of unhappiness may or may not correspond to the linguistically meaningful units un-, happy and -ness.

3. Syntax: Structure and Grammatical Relations

Syntax concerns how words combine into phrases and sentences. It explains why the following sequences differ in interpretation:

• The researcher examined the model.
• The model examined the researcher.

The same words occur, but their grammatical relations change.

3.1 Constituency and dependency

A constituency grammar represents nested phrases. In The experienced engineer tested the system, the noun phrase the experienced engineer functions as a unit, and the system is the object noun phrase.

A dependency grammar represents directed relations between words. In the same sentence, tested is the root; engineer is its subject; system is its object; and experienced modifies engineer. Dependency structures are often convenient for information extraction because they represent grammatical relations directly.

This is a simple context-free grammar. Here, S denotes sentence, NP noun phrase, VP verb phrase, Det determiner, N noun and V verb. The rules describe one possible structural pattern and do not capture the full grammar of English.

3.2 Parsing

Parsing assigns structure to an input. In constituency parsing, a parser builds a phrase-structure tree. In dependency parsing, it predicts a head and dependency relation for each token. Common dependency relations include subject, object, modifier and auxiliary.

Parsing approaches include:

• Rule-based parsing, based on hand-written grammatical rules.
• Statistical parsing, which estimates the likelihood of alternative structures from annotated corpora.
• Transition-based parsing, which incrementally performs actions such as shift, reduce and attach.
• Neural parsing, which uses learned representations to predict structures.

Parsing is useful in relation extraction, semantic role labelling, question answering, grammar correction and explainable language analysis. It is not infallible: informal text, speech transcripts, long-distance dependencies and domain-specific terminology can cause errors.

4. Semantics: Representing Meaning

Semantics studies meaning independently of a particular conversational situation, although the boundary between semantics and pragmatics is not always absolute.

4.1 Lexical semantics

Lexical semantics examines relations among word meanings. Important relations include:

• Synonymy: similar meaning, such as begin and commence.
• Antonymy: contrast, such as legal and illegal.
• Hyponymy: an is-a relation, such as sparrow being a hyponym of bird.
• Meronymy: a part-whole relation, such as wheel being part of a car.
• Polysemy: related senses of one form, such as paper as material and academic article.
• Homonymy: identical form with unrelated meanings, such as bank as a financial institution and river edge.

A computational lexicon may store senses, examples, grammatical categories and semantic relations. WordNet-like resources organise words into synsets and relations, while domain ontologies represent concepts and formal relationships.

4.2 Compositional semantics

The principle of compositionality states that the meaning of a complex expression is determined by the meanings of its parts and the way they are combined. It explains why a system must represent both lexical items and structure. Compare old men and women with old men and old women: the adjective may modify only men or both coordinated nouns.

Semantic role labelling identifies the roles associated with an event. In The technician repaired the server with a replacement part, technician is the agent, server is the patient or theme, and replacement part may be an instrument or means.

4.3 Distributional and vector semantics

Distributional semantics represents a word through patterns of use. The central intuition is that words occurring in similar contexts tend to have related meanings. A traditional count-based representation uses a vector of context frequencies. Neural embeddings learn dense vectors from surrounding text.

For vectors u and v, cosine similarity is commonly defined as:

Here, u · v is the dot product and ||u|| and ||v|| are vector lengths. The vectors must be non-zero. The measure is dimensionless and ranges from −1 to 1 in the usual real-valued setting. It compares orientation rather than absolute magnitude.

Static embeddings assign one vector to a word type, so bank has the same representation in financial and geographical contexts. Contextual models generate representations influenced by surrounding words and can distinguish these senses more effectively. Nevertheless, contextual representations do not guarantee genuine human-like understanding; they can encode statistical associations, stereotypes and factual errors.

5. Pragmatics and Speaker Intention

Pragmatics concerns how meaning is inferred from context, shared knowledge, social relationships and communicative goals. The utterance Can you open the window? has the grammatical form of a question about ability, but commonly functions as a request.

Important pragmatic phenomena include:

• Deixis, in which interpretation depends on context: here, you, tomorrow.
• Presupposition, an assumption treated as given: The manager stopped calling presupposes that the manager called previously.
• Implicature, an inferred meaning not explicitly stated: Some participants passed may imply that not all passed.
• Speech acts, such as asserting, requesting, promising, warning and apologising.
• Politeness, including indirectness, status sensitivity and face management.

Computational pragmatics supports intent detection, dialogue management, sentiment interpretation, moderation and socially appropriate generation. A system that interprets That was brilliant must use context to determine whether the statement is praise or sarcasm.

6. Discourse and Coherence

Discourse is language extended beyond the individual sentence. Discourse analysis examines how sentences connect, how entities are tracked and how information is organised.

6.1 Cohesion and coherence

Cohesion is signalled by linguistic devices such as pronouns, conjunctions, lexical repetition and ellipsis. Coherence concerns whether the ideas form a meaningful whole. A passage can be cohesive but incoherent if it contains grammatical links without a sensible conceptual progression.

6.2 Reference and coreference

In Aisha submitted the report because she had revised it, she refers to Aisha and it refers to the report. Coreference resolution identifies expressions that refer to the same entity. Errors in coreference can change the output of summarisation, question answering and information extraction.

6.3 Discourse relations

Discourse relations include cause, contrast, elaboration, condition and temporal sequence. In The server failed because the battery was empty, the second clause explains the first. Discourse parsing and document-level modelling help systems generate coherent summaries and maintain topic continuity.

7. Ambiguity in Natural Language

Ambiguity occurs when an expression supports more than one interpretation. It is not merely a nuisance; it is a central property of language.

Systems resolve ambiguity by using local context, sentence structure, discourse context, world knowledge, speaker information and task constraints. A translation system may use target-language agreement and lexical selection to disambiguate a source sentence. A question-answering system may use the wording of the question and evidence documents.

A critical design principle is to preserve uncertainty when evidence is insufficient. Forced disambiguation can create confident but incorrect outputs, a serious risk in medicine, law and public administration.

8. Multilingual and Cross-Linguistic Linguistics

Languages differ in script, sound inventory, word formation, grammatical categories, constituent order and conventions of politeness. These differences affect both data preparation and model design.

8.1 Typological variation

Languages may be broadly described as analytic, synthetic, agglutinative, fusional or polysynthetic, although these categories are tendencies rather than rigid boxes. English generally expresses grammatical relations through word order and function words. Turkish can attach several relatively separable suffixes to a stem. Arabic includes a rich system of derivation and inflection, while Mandarin relies strongly on word order, particles and context.

A model trained primarily on English may perform poorly on languages with freer word order, complex morphology, limited digital data or non-segmented writing systems. Word segmentation is itself a linguistic and computational problem for languages such as Chinese, Japanese and Thai.

8.2 Translation and transfer

Machine translation must account for lexical gaps, idioms, gender and number agreement, politeness, tense and discourse context. Literal word substitution is insufficient. For example, an idiom may require a culturally appropriate equivalent rather than a translation of each word.

Code-switching occurs when speakers alternate languages or varieties. It is common in multilingual communities and online communication. Systems should not automatically treat it as noise: the switch may signal identity, emphasis, topic or audience alignment.

8.3 Low-resource languages and fairness

Many languages lack large corpora, digitised dictionaries, speech resources or annotated datasets. This creates unequal system quality. Responsible development includes community consultation, data documentation, transfer learning, multilingual evaluation and support for local orthographic conventions. A system should not be labelled multilingual merely because it can produce short strings in many languages.

9. From Linguistic Structure to Language Models

Modern language models typically process a sequence of tokens. A simplified probabilistic objective is to estimate the likelihood of a token sequence by factoring it into conditional probabilities:

Here, xᵢ is the token at position i, n is sequence length, and the product applies the chain rule of probability. The formula is an identity for a joint distribution when the conditional probabilities are correctly specified; it does not imply that a model has human linguistic competence.

Generative models learn regularities involving morphology, syntax, semantics and discourse from data. Attention mechanisms allow information from different positions to influence a representation. However, linguistic structure is learned imperfectly and unevenly. A model may produce fluent syntax while hallucinating facts, mishandling negation or failing at reference resolution.

Useful representations include:

• token and subword sequences;
• morphological feature bundles;
• constituency or dependency trees;
• semantic roles and frames;
• entity and coreference graphs;
• discourse representations;
• multilingual aligned embeddings;
• structured knowledge graphs.

In professional systems, representations are often combined. A retrieval-augmented assistant may use a language model, document metadata, entity linking, access controls and a citation mechanism. The linguistic layer is therefore part of a larger socio-technical system.

10. Applied Case: Multilingual Public-Service Assistant

The following is an illustrative case.

Background

A municipal authority wishes to deploy a text-based assistant answering questions about waste collection, permits and local services. Residents communicate in English, Arabic and a regional mixed-language variety. Some users submit short messages without punctuation.

Problem

An initial prototype answers fluent English questions reasonably well but fails on Arabic morphology, code-switched messages and pronouns whose reference depends on previous turns. It also interprets indirect requests as general comments. For example, a resident writes, “Tomorrow bins?” intending to ask whether collection occurs tomorrow.

Analysis

The task requires more than keyword matching. The system must identify an intent, recover omitted grammatical material, recognise dates relative to the message time, track the relevant service and respond in the user’s language. It must also distinguish a request for information from a complaint.

Approach

The development team adopts a layered design:

1. Language identification is performed at message and span level to detect code-switching.
2. Morphological analysis supplies lemmas and grammatical features where available.
3. A multilingual encoder produces contextual representations.
4. An intent classifier predicts service category and dialogue act.
5. Entity and temporal normalisation identify locations, dates and service types.
6. Coreference tracking maintains the active bin, permit or address across turns.
7. A retrieval component selects approved municipal information.
8. A constrained generator produces a response, with human review for high-risk cases.

The team evaluates accuracy separately by language, dialect, message length, code-switching rate and demographic group. It records uncertainty and asks clarification questions when multiple interpretations remain plausible.

Outcome

The revised system improves task completion and reduces unsupported answers. Its most important improvement is not a larger model alone but the explicit treatment of morphology, pragmatics, discourse context and multilingual variation.

Learning Points

The case demonstrates that linguistic analysis guides architecture, data collection and evaluation. It also shows why aggregate accuracy can conceal poor performance for minority languages and informal users.

11. Challenges, Ethics and Best Practice

Linguistic representations are shaped by their data. Corpora may contain stereotypes, privacy-sensitive conversations, abusive language and unequal representation. Embeddings can associate occupations, emotions or social roles with demographic groups. Synthetic training data may reproduce these patterns at scale.

Best practice includes:

• documenting data sources, language varieties and annotation decisions;
• separating dialect difference from error;
• testing ambiguity and negation explicitly;
• evaluating by language, domain and user group;
• protecting personal and conversational data;
• providing escalation routes for consequential decisions;
• retaining uncertainty rather than inventing certainty;
• involving speakers and domain experts in evaluation;
• monitoring model drift after deployment;
• avoiding claims of understanding that the evidence does not support.

Annotator disagreement is often informative. Disagreement may indicate genuine ambiguity, dialect variation or an underspecified label scheme. Treating every disagreement as annotation failure can erase legitimate linguistic diversity.

Contemporary NLP increasingly combines foundation models with retrieval, tool use, structured decoding, speech technologies and multilingual adaptation. The enduring lesson is that scale does not eliminate linguistic problems. It changes where they appear: tokenisation, representation, prompting, retrieval, generation, evaluation and governance.

Chapter Summary

Language is a multi-level system involving morphology, syntax, semantics, pragmatics, discourse and social variation. Morphology explains the internal structure of words and supports lemmatization, grammatical analysis and subword processing. Syntax represents constituent structure and grammatical dependencies, enabling parsing and information extraction. Semantics addresses lexical relations, compositional meaning, semantic roles and vector representations. Pragmatics explains how context, intention, presupposition and implicature shape interpretation. Discourse analysis handles coherence, reference and relations across sentences.

Ambiguity may be lexical, morphological, syntactic, semantic, referential or pragmatic. Computational systems resolve it through context, structure and world knowledge, but should preserve uncertainty when evidence is weak. Multilingual NLP must account for typological variation, scripts, morphology, code-switching, low-resource conditions and cultural conventions. Generative language models learn statistical regularities across these levels, but fluent output is not proof of reliable understanding. Responsible NLP therefore combines appropriate representations, multilingual evaluation, human oversight, privacy protection and transparent governance.

Key Terms

Morphology: The study of word structure and formation.

Morpheme: The smallest meaningful linguistic unit.

Inflection: Grammatical modification of a word without normally creating a new lexeme.

Derivation: Word formation that creates a new lexeme or changes grammatical category.

Tokenisation: Dividing text into computational processing units.

Lemmatization: Mapping an inflected form to its dictionary base form.

Syntax: The study of how words combine into structures.

Constituency: Representation of nested phrases as structural units.

Dependency: A grammatical relation between a head and dependent word.

Semantics: The study of linguistic meaning.

Pragmatics: The study of meaning in context and language use.

Discourse: Connected language extending beyond an individual sentence.

Coreference: The relation in which expressions refer to the same entity.

Ambiguity: The availability of more than one interpretation.

Distributional semantics: Representation of meaning through patterns of contextual use.

Embedding: A numerical vector representation learned for linguistic units.

Code-switching: Alternation between languages or varieties in communication.

Low-resource language: A language with limited digital, textual or annotated resources.

Semantic role: The function an entity plays in an event, such as agent or patient.

Speech act: An action performed through language, such as requesting or promising.

Review Questions

5 Short Answer Questions

1. What is the difference between inflection and derivation?
2. Why is lemmatization generally more linguistically informed than stemming?
3. What does a dependency parser represent?
4. Distinguish semantics from pragmatics.
5. What is code-switching, and why does it matter for NLP?

5 Long Answer Questions

1. Explain the principal levels of linguistic analysis and show how they interact in an NLP system.
2. Compare constituency and dependency representations, including one practical use of each.
3. Discuss lexical, compositional and distributional semantics with suitable examples.
4. Explain how discourse context helps resolve reference and ambiguity.
5. Evaluate the challenges of building a fair multilingual language model.

3 Analytical / Application Questions

1. Design a linguistic processing pipeline for a customer-support assistant that receives informal, multilingual messages. Justify each representation.
2. Analyse the possible interpretations of The analyst reviewed the report with the consultant. Identify the ambiguity and propose computational evidence for resolving it.
3. A language model produces grammatically fluent answers but frequently misunderstands pronouns and indirect requests. Diagnose the linguistic causes and recommend evaluation and system improvements.

Multiple Choice Questions with Answers

Question 1

Which operation maps an inflected word to its dictionary base form?

A. Tokenisation

B. Lemmatization

C. Parsing

D. Coreference resolution

Correct Answer: B. Lemmatization

Question 2

In the sentence The engineer repaired the server, what is the grammatical role of the engineer?

A. Object

B. Instrument

C. Subject or agent

D. Modifier of server

Correct Answer: C. Subject or agent

Question 3

Which example best illustrates pragmatics?

A. The suffix -ed marks past tense

B. A noun phrase contains a determiner and noun

C. Bank has financial and geographical senses

D. Can you close the door? functions as a request

Correct Answer: D. Can you close the door? functions as a request

Question 4

What does cosine similarity compare between two vectors?

A. Their alphabetical order

B. The angle or orientation between them

C. Their number of dimensions only

D. Their grammatical categories

Correct Answer: B. The angle or orientation between them

Question 5

Which relation is represented when sparrow is described as a kind of bird?

A. Hyponymy

B. Antonymy

C. Homonymy

D. Coreference

Correct Answer: A. Hyponymy

Question 6

What is the main purpose of coreference resolution?

A. To identify words with the same spelling

B. To identify expressions referring to the same entity

C. To divide text into subwords

D. To translate an idiom literally

Correct Answer: B. To identify expressions referring to the same entity

Question 7

Which issue is especially associated with multilingual NLP?

A. All languages use identical word order

B. Every language has equally large annotated corpora

C. Scripts, morphology and grammatical categories vary across languages

D. Pragmatics is absent from multilingual communication

Correct Answer: C. Scripts, morphology and grammatical categories vary across languages

Question 8

The phrase I saw the person with a telescope primarily illustrates which type of ambiguity?

A. Syntactic

B. Orthographic

C. Phonological

D. Inflectional

Correct Answer: A. Syntactic

Question 9

Why can subword tokens be useful in language models?

A. They guarantee correct semantic interpretation

B. They allow processing of unfamiliar or rare word forms

C. They eliminate the need for context

D. They are always identical to morphemes

Correct Answer: B. They allow processing of unfamiliar or rare word forms

Question 10

What is a responsible response when available evidence does not resolve an ambiguity in a high-risk application?

A. Select an interpretation and express complete certainty

B. Ignore the ambiguity

C. Preserve uncertainty or request clarification

D. Replace the input with a more common sentence

Correct Answer: C. Preserve uncertainty or request clarification