Language Data, Corpora and Text Preparation

Introduction

Natural language processing (NLP) systems learn from language data. Whether the task is sentiment classification, machine translation, information extraction, question answering, speech transcription, or large-language-model pretraining, system quality depends strongly on the quality and representativeness of the underlying corpus. A sophisticated model trained on incomplete, noisy, culturally narrow, or poorly documented data may produce unreliable and unfair results.

Text preparation is therefore not merely a preliminary programming step. It is a methodological process involving data acquisition, linguistic analysis, computational representation, quality control, annotation, sampling, and governance. Decisions made during preparation—such as whether to lowercase text, remove punctuation, preserve emojis, split documents by author, or include code-switched language—can change the meaning of the data and the behaviour of the resulting model.

This chapter presents a rigorous workflow for acquiring, cleaning, normalizing, tokenizing, annotating, and evaluating multilingual text data. It also examines corpus design, metadata, class imbalance, augmentation, data leakage, and train-validation-test splitting.

Learning Objectives

After studying this chapter, you should be able to:

1. Explain the purpose and structure of a corpus and distinguish major corpus types.
2. Design a representative and well-documented multilingual text collection.
3. Acquire text lawfully and assess its provenance, quality, and bias.
4. Apply cleaning, normalization, segmentation, and tokenization methods appropriately.
5. Explain why multilingual and code-switched data require language-sensitive processing.
6. Construct annotation schemes, measure agreement, and manage labels and metadata.
7. Diagnose imbalance, duplication, leakage, and sampling problems.
8. Create defensible train-validation-test splits.
9. Select suitable augmentation strategies without changing the target meaning.
10. Evaluate the ethical, privacy, and governance implications of language data.

1. Language Data and the Role of a Corpus

A corpus is a structured collection of language samples assembled for a particular linguistic, computational, or research purpose. A corpus may consist of documents, sentences, conversational turns, social-media posts, transcripts, web pages, or multimodal records with associated text. The plural is corpora.

A corpus is not simply a large pile of text. It has at least four defining properties:

• Purpose: the tasks or phenomena it is intended to support.
• Composition: the sources, genres, domains, languages, time periods, and populations represented.
• Sampling procedure: how instances were selected and what was excluded.
• Documentation: records describing provenance, processing, annotation, and limitations.

For example, a corpus for detecting abusive content in online gaming chats should not be evaluated only by its number of messages. It should also describe the games, languages, age groups, moderation policies, time period, sampling method, and definition of abuse.

1.1 Types of corpora

A corpus can belong to several categories simultaneously. A medical parallel corpus, for example, is both domain-specific and multilingual.

1.2 Corpus balance and representativeness

Representativeness means that the corpus reflects the population, linguistic varieties, or task conditions about which conclusions will be drawn. Balance refers to the deliberate distribution of categories such as genre, class, language, region, or document length.

These concepts are related but not identical. A balanced corpus may contain equal numbers of documents from five genres, yet still be unrepresentative if the genres are sampled from one country or one demographic group. Conversely, a representative corpus may be naturally imbalanced because some categories occur more frequently in real use.

Corpus design should begin with a target population. Let the target distribution over categories be p(c), and the corpus distribution be q(c), where c denotes a category such as language, topic, or dialect. A simple distributional comparison is the total variation distance:

Here, the summation covers all categories, and the result lies between 0 and 1 for probability distributions. A value of 0 indicates identical distributions. This measure is useful when the target distribution is known, although it does not capture every form of linguistic mismatch.

2. Data Acquisition and Provenance

Text may be collected from institutional databases, licensed repositories, public websites, application logs, surveys, interviews, social platforms, or generated interactions. Each source introduces different technical, legal, and ethical conditions.

A professional acquisition workflow should record:

• source identity and URL or repository reference;
• collection date and time zone;
• language and country or region, where known;
• licence, consent status, and permitted uses;
• collection method and software version;
• filtering and exclusion rules;
• document or speaker identifiers;
• known demographic or domain limitations;
• hashes or checksums for reproducibility;
• version history and correction procedures.

Data provenance is the traceable history of where an item came from and what happened to it. Provenance supports reproducibility, auditing, removal requests, and diagnosis of unexpected model behaviour.

2.1 Legal and ethical considerations

Public availability does not automatically imply unrestricted reuse. Copyright, database rights, terms of service, data-protection legislation, confidentiality obligations, and research ethics may all apply. Personal data may include names, contact details, locations, identifiers, health information, political opinions, or indirect combinations that enable re-identification.

Useful safeguards include:

• collecting only data necessary for the stated purpose;
• obtaining informed consent where required;
• removing or masking direct identifiers;
• limiting access through role-based controls;
• encrypting data at rest and in transit;
• defining retention and deletion policies;
• documenting lawful basis and licence restrictions;
• providing mechanisms for correction or withdrawal where applicable;
• evaluating whether annotation exposes workers to harmful content.

Anonymization attempts to prevent re-identification, whereas pseudonymization replaces identifiers with codes but may still allow re-identification through a separate key. Neither process guarantees complete privacy, especially in short or highly distinctive texts.

3. A Text-Preparation Pipeline

A robust pipeline is staged, versioned, and reversible where possible. A typical sequence is:

1. Acquire and inventory raw data.
2. Validate encoding and file integrity.
3. Remove exact and near duplicates when appropriate.
4. Detect language and identify code-switching.
5. Segment documents into paragraphs, sentences, and turns.
6. Normalize text while preserving task-relevant information.
7. Tokenize using a model-appropriate method.
8. Annotate labels, linguistic features, and metadata.
9. Run quality checks and document exclusions.
10. Split data into development and evaluation partitions.
11. Export versioned datasets with datasheets and processing logs.

The order is not universally fixed. For example, aggressive normalization before language identification may destroy script or diacritic evidence needed by the detector. Similarly, deduplication may be performed before or after segmentation depending on whether documents or individual passages are the unit of duplication.

3.1 Encoding and character integrity

Most contemporary systems use Unicode, which assigns abstract code points to characters across writing systems. A file encoding, such as UTF-8, specifies how those code points are stored as bytes. Encoding errors can produce replacement characters, lost diacritics, or unreadable text.

Validation should inspect:

• declared and actual encoding;
• invalid byte sequences;
• normalization form;
• control characters;
• right-to-left marks;
• zero-width joiners and non-joiners;
• script consistency;
• unexpected null bytes.

Unicode normalization is important because visually identical strings may have different code-point sequences. For example, a character with a diacritic may be represented as one precomposed character or as a base character followed by a combining mark. NFC generally composes compatible sequences where possible; NFD decomposes them. The appropriate choice depends on the language and task. Normalization must not erase distinctions that carry meaning.

4. Cleaning and Normalization

Cleaning removes or corrects unwanted artefacts. Normalization maps alternative surface forms into a more consistent representation. Neither should be treated as synonymous with deleting information.

Common cleaning operations include:

• removing markup and boilerplate;
• correcting malformed HTML entities;
• replacing corrupted encoding sequences;
• collapsing accidental repeated whitespace;
• removing empty records;
• filtering spam or machine-generated noise;
• identifying truncated or damaged documents.

Common normalization operations include:

• Unicode normalization;
• standardizing quotation marks and dash characters;
• canonicalizing URLs, email addresses, and user mentions;
• converting numbers to placeholders when exact values are irrelevant;
• controlled case folding;
• spelling normalization in narrowly defined settings;
• expanding or preserving contractions according to the task.

A normalization rule should be justified by an explicit hypothesis. Lowercasing can reduce vocabulary sparsity, but it may remove information from acronyms, named entities, or German nouns. Removing punctuation may simplify some classifiers while destroying emoticons, sentence boundaries, mathematical notation, or legal structure. Removing stop words can harm syntactic or semantic tasks and is rarely appropriate for modern transformer-based systems unless empirically justified.

A useful practice is to maintain multiple representations:

• raw text for audit and future reprocessing;
• lightly cleaned text for general modelling;
• task-specific normalized text;
• tokenized and annotated forms.

4.1 Handling social and web text

Social text often includes hashtags, mentions, emojis, elongated spellings, URLs, quoted material, and informal grammar. Replacing every URL with the same token may be sensible for topic classification but harmful for phishing detection, where the domain is important. Emojis can encode sentiment, stance, or irony and should usually be preserved or converted to descriptive labels rather than discarded.

Web text requires boilerplate removal, but navigation menus, headings, captions, and hyperlinks may carry useful structure. Extraction rules should therefore be evaluated on a sample from every source rather than applied blindly.

5. Segmentation and Tokenization

Segmentation divides text into meaningful units. It may include document, paragraph, sentence, dialogue-turn, clause, or word segmentation. Tokenization converts text into a sequence of computational units called tokens.

Tokenization is language- and model-dependent. In English, whitespace and punctuation provide useful clues, but they are not sufficient for all languages. Chinese and Japanese do not generally use spaces between words. Thai has complex word-boundary issues. Arabic involves clitics and optional diacritics. Indic scripts may contain combining marks and conjuncts. Agglutinative languages can encode many grammatical functions in a single orthographic word.

5.1 Tokenization strategies

1. Whitespace tokenization: splits on spaces. It is simple but fails with punctuation, compounds, clitics, and scripts without spaces.
2. Rule-based tokenization: uses language-specific rules for punctuation, contractions, numbers, and symbols.
3. Word segmentation: predicts word boundaries where spaces are absent.
4. Subword tokenization: represents words as frequent character sequences or subword units. Byte-pair encoding, WordPiece, and unigram language-model tokenization are common examples.
5. Character or byte tokenization: represents text at very small units and can handle unseen words, but may create longer sequences.

Suppose a tokenizer maps a sequence of characters x to tokens t₁, t₂, …, tₙ. The sequence length n depends on the tokenizer, normalization rules, and language. There is no universally correct token count. Tokenization should be evaluated by downstream performance, coverage, sequence length, interpretability, and robustness to spelling variation.

Subword methods reduce the unknown-word problem by decomposing rare words. However, they may split meaningful morphemes inconsistently across languages and can produce inefficiently long sequences for languages poorly represented in training data. A multilingual tokenizer should be assessed for vocabulary allocation and fertility, where fertility is the average number of subword tokens used to represent one word or comparable linguistic unit.

5.2 Sentence segmentation challenges

Sentence boundaries are ambiguous in abbreviations, decimal numbers, quotations, bullet lists, informal messages, and multilingual text. A rule that treats every period as a sentence boundary will incorrectly split dates, URLs, and abbreviations. Sentence segmentation errors can propagate into translation alignment, document classification, and annotation.

6. Multilingual, Multiscript, and Code-Switched Text

Multilingual NLP must account for variation in script, morphology, orthography, dialect, and resource availability. A language label may be attached to a whole document, sentence, span, or token. These levels should not be confused. A document labelled “English” may contain a Spanish quotation, a Hindi name, and an Arabic greeting.

Code-switching is the use of more than one language or language variety within an interaction, sentence, or phrase. It is not necessarily noise. It may express identity, emphasis, politeness, technical precision, or social affiliation.

A multilingual preparation strategy should consider:

• script detection and transliteration;
• language identification at the appropriate granularity;
• dialect and regional variation;
• code-switching and borrowed words;
• diacritics and orthographic conventions;
• morphology and tokenization differences;
• unequal corpus size across languages;
• culturally specific names, entities, and speech acts.

Transliteration can improve cross-script search but may introduce ambiguity. For example, multiple original characters may map to the same Latin representation. Therefore, original text should be retained alongside transliterated text whenever possible.

Language identification accuracy should be measured separately by language and by text type. Short messages, named entities, and code-switched spans are particularly difficult. A single global accuracy value can hide serious failure for low-resource languages.

7. Annotation, Labels, and Metadata

Annotation adds structured information to text. Examples include sentiment labels, named-entity spans, part-of-speech tags, syntactic dependencies, dialogue acts, toxicity categories, translation alignments, and factuality judgements.

An annotation scheme should specify:

• the unit of annotation;
• the label inventory;
• definitions and decision rules;
• inclusion and exclusion criteria;
• treatment of ambiguity and uncertainty;
• examples and counterexamples;
• adjudication procedures;
• annotator qualifications and training;
• version and change history.

Labels may be binary, multiclass, multilabel, ordinal, or structured. In a multilabel task, one item may receive several labels, such as both “threat” and “insult.” Labels must be distinguished from metadata. “Positive sentiment” is usually a target label; “source platform,” “language,” and “collection date” are metadata or covariates.

7.1 Annotation quality and agreement

Agreement measures indicate how consistently annotators apply a scheme. For two annotators assigning nominal categories, Cohen’s kappa is commonly defined as:

Pₒ is the observed proportion of agreement, and Pₑ is the agreement expected by chance under the chosen marginal-distribution model. The formula assumes categorical labels and requires a defined sampling unit. It can be misleading under severe class imbalance because high raw agreement may coexist with poor recognition of a minority class. For more than two annotators, alternatives include Fleiss’ kappa or Krippendorff’s alpha, selected according to the data type and missingness pattern.

Agreement is not the same as validity. Annotators may agree consistently on a poorly defined concept. Low agreement may indicate ambiguous language, inadequate training, overlapping categories, or a fundamentally subjective task. Disagreement analysis is therefore more informative than a single coefficient. Record disputed items, adjudication decisions, and changes to the guideline.

7.2 Metadata schema

A useful metadata record may contain:

Metadata can itself create privacy risks. Store only fields that are necessary, restrict access, and assess whether combinations of fields permit re-identification.

8. Data Quality, Duplicates, and Leakage

Quality assurance should be systematic rather than anecdotal. Useful checks include:

• missing or empty text;
• invalid encoding;
• duplicate identifiers;
• exact duplicate texts;
• near duplicates and quoted reposts;
• label values outside the schema;
• impossible timestamps;
• inconsistent language labels;
• unusually long or short items;
• toxic or sensitive content requiring controlled access;
• annotation conflicts;
• train-test overlap.

Exact duplication can be detected by canonicalizing text and comparing hashes. Near-duplicate detection may use character n-gram similarity, MinHash, locality-sensitive hashing, or embedding similarity. The threshold must be validated because aggressive deduplication may remove legitimate repeated expressions.

Data leakage occurs when information unavailable at deployment time, or derived from the evaluation set, influences training. Examples include placing near-identical posts from the same conversation in both training and test sets, using future metadata, fitting a vocabulary on all partitions, or selecting features after examining test performance.

Preprocessing transformations with learned parameters—such as vocabulary construction, normalization statistics, feature selection, and imputation—should generally be fitted on the training set and then applied to validation and test data. Fixed linguistic rules may be applied consistently before splitting, provided they do not use outcome information.

9. Imbalance and Sampling

Class imbalance occurs when target classes have unequal frequencies. It can reflect the real world or result from collection and annotation procedures. Accuracy can be misleading when a majority class dominates. If 95% of instances are negative, a classifier that always predicts negative obtains 95% accuracy but has no minority-class utility.

Let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives. Important metrics include:

These formulas assume binary classification and nonzero denominators. Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives detected. F1 is their harmonic mean and is useful when both are important, but it does not reflect true negatives and may not suit every operational cost.

Responses to imbalance include:

• collecting more minority-class examples;
• stratified sampling;
• class-weighted loss functions;
• under-sampling the majority class;
• oversampling the minority class;
• threshold adjustment;
• targeted augmentation;
• reporting per-class and macro-averaged metrics.

Sampling decisions should match deployment conditions. Artificially balancing training data can improve minority recognition but may distort probability calibration. Always evaluate on a test distribution that reflects the intended use, or report clearly when the test set is deliberately balanced.

10. Data Augmentation

Augmentation creates additional training examples while attempting to preserve the target label. Suitable techniques depend on the task and language:

• back-translation;
• controlled paraphrasing;
• synonym substitution with contextual checks;
• character-level noise for spelling robustness;
• span masking or deletion;
• entity replacement with type-compatible entities;
• synthetic templates;
• perturbation of punctuation or casing in noisy-text tasks.

Augmentation can introduce label changes. Negation is especially dangerous: deleting “not” may reverse sentiment. Replacing a named entity can alter toxicity, factuality, or intent. Machine-generated examples may also reproduce stereotypes or create unnatural language. Augmented examples should be audited, deduplicated, tagged in metadata, and kept out of evaluation sets unless the evaluation explicitly concerns synthetic data.

11. Train-Validation-Test Splitting

A dataset is commonly divided into:

• training set: used to fit model parameters;
• validation set: used for model selection, hyperparameter tuning, and threshold choice;
• test set: used once, as far as practical, for final performance estimation.

A random split is appropriate only when examples are approximately independent and identically distributed with respect to the intended use. It is inappropriate when multiple items come from the same author, patient, conversation, document, organization, or time-dependent event.

For grouped data, all items from a group should normally remain in one partition. Otherwise, author-specific vocabulary or repeated conversation context can inflate performance. For temporal deployment, use a chronological split so that training precedes evaluation. For multilingual data, stratify or group by language and domain so that low-resource categories are not accidentally excluded.

A split is reproducible when the sampling rule, random seed, unit of splitting, and exclusions are recorded. No split can guarantee generalization to a population absent from the corpus; external evaluation remains important.

12. Applied Case: A Multilingual Customer-Complaint Corpus

The following is an illustrative case.

Background

A financial-services organization wants to classify customer messages into billing problem, fraud concern, technical failure, account access, and general enquiry. Messages arrive in English, Hindi, and Hindi-English code-switched text through email and chat.

Problem

The initial dataset contains 180,000 messages, but 72% are general enquiries. Many conversations contain several turns from the same customer. Agents have used inconsistent labels, and some messages include account numbers and phone numbers. The organization reports high overall accuracy but poor performance on fraud concerns.

Analysis

An audit finds four major risks:

1. Random message-level splitting places turns from the same conversation in both training and test sets.
2. The fraud class is rare and often expressed indirectly.
3. A rule-based English tokenizer damages Hindi script and emoji sequences.
4. Personally identifying information has not been consistently masked.

The team defines the customer conversation, rather than the individual message, as the grouping unit. It develops a label guide with examples, allows a secondary “uncertain” status during annotation, and conducts bilingual annotator training. Language is annotated at message and span level to capture code-switching.

Approach and solution

The revised pipeline preserves raw text in a restricted store and creates a de-identified working version. Account numbers are replaced with typed placeholders such as ACCOUNT_ID, while dates and monetary amounts are retained because they support the task. Unicode validation is applied before language identification. A multilingual subword tokenizer is evaluated against a language-aware baseline.

Annotators independently label a stratified sample. Disagreements are reviewed by a senior adjudicator, and the guidelines are revised when recurring ambiguity is identified. The data is split by conversation: 70% training, 15% validation, and 15% test, with approximate language and class stratification where feasible. A chronological test subset is also held out to assess future performance.

Instead of simply duplicating fraud examples, the team collects additional fraud-related messages and uses carefully reviewed paraphrases. Evaluation reports per-class precision, recall, F1, confusion matrices, calibration, language-specific results, and performance on code-switched messages.

Outcome

Overall accuracy decreases slightly, but fraud recall improves substantially and test performance becomes more realistic. The organization discovers that the model performs well in English but misses indirect Hindi expressions. This leads to targeted data collection and revised annotation guidance rather than premature model deployment.

Learning points

The case demonstrates that corpus design, privacy protection, group-aware splitting, multilingual processing, and label quality can matter more than selecting a larger model. Performance numbers are meaningful only when the evaluation design reflects deployment conditions.

13. Best Practices and Contemporary Developments

Current NLP systems increasingly use pretrained multilingual language models, instruction-tuned models, retrieval systems, and data-centric evaluation tools. These technologies do not remove the need for careful corpora. They can amplify biases in source data, tokenize low-resource languages inefficiently, and generate plausible but incorrect synthetic text.

Recommended practice includes:

• preserve raw data separately from processed derivatives;
• make every transformation versioned and testable;
• use data cards or datasheets describing composition and limitations;
• report language, dialect, domain, and demographic coverage;
• evaluate both average and subgroup performance;
• inspect false positives and false negatives manually;
• separate development from final evaluation;
• use group- or time-aware splits when appropriate;
• treat annotators fairly and protect them from harmful content;
• document licensing, consent, retention, and deletion processes;
• audit pretrained tokenizers and models for low-resource languages;
• avoid claiming generalization beyond the sampled population.

A mature NLP project treats the corpus as a governed research asset rather than a disposable input file. Reproducibility requires not only code but also stable data versions, processing specifications, annotation decisions, and an account of what could not be collected.

Challenges and Limitations

No corpus can represent all language use. Sampling frames exclude undocumented populations, private communication, rare dialects, and rapidly changing vocabulary. Labels simplify phenomena that may be ambiguous or context-dependent. Privacy protection can remove information needed for disambiguation. Deduplication may eliminate legitimate recurring language, while insufficient deduplication causes leakage. Translation and augmentation may introduce artefacts. Finally, performance measured on a carefully curated test set may not predict behaviour under distribution shift.

These limitations should be stated explicitly. The purpose of corpus documentation is not to claim perfection but to make assumptions visible and enable responsible interpretation.

Chapter Summary

Language data forms the empirical foundation of NLP. A corpus must be designed for a purpose, sampled deliberately, documented, and governed responsibly. Corpus quality depends on provenance, representativeness, linguistic coverage, annotation validity, and protection against duplication and leakage.

Text preparation includes encoding validation, cleaning, normalization, segmentation, tokenization, multilingual processing, annotation, and quality control. These operations are not neutral: removing case, punctuation, emojis, diacritics, or code-switching may destroy task-relevant information. Tokenization must be evaluated for the languages and scripts in use.

Annotation schemes require precise definitions, training, disagreement analysis, and suitable agreement measures. Metadata supports auditing and subgroup analysis but can create privacy risks. Imbalance should be addressed through appropriate sampling, collection, weighting, augmentation, and class-sensitive evaluation. Train-validation-test splits must respect authorship, conversations, organizations, and time when observations are dependent.

The central professional principle is that data preparation is part of modelling. Transparent, lawful, multilingual, and deployment-aware corpus construction produces more reliable and equitable NLP systems than model selection alone.

Key Terms

Corpus: A structured collection of language data assembled for a defined purpose.

Corpus design: Planning the composition, sampling, documentation, and governance of a corpus.

Representativeness: The extent to which data reflects the population or conditions of intended use.

Balance: Deliberate distribution of categories within a dataset.

Provenance: The documented origin and processing history of a data item.

Unicode: A standard for representing characters across writing systems.

Normalization: Mapping equivalent or variant forms into a controlled representation.

Segmentation: Dividing text into documents, sentences, turns, or other units.

Tokenization: Converting text into computational units called tokens.

Subword tokenization: Representing words as frequently occurring character sequences.

Code-switching: Alternating between languages or varieties within communication.

Annotation: Adding structured linguistic or task-related information to text.

Metadata: Descriptive information about an item, its source, context, or processing.

Data leakage: Unintended use of information from evaluation data or deployment-inaccessible sources.

Class imbalance: Unequal frequencies of target classes.

Data augmentation: Creating additional training examples while attempting to preserve labels.

Stratified sampling: Sampling that preserves selected category proportions.

Group-aware splitting: Partitioning data so related items remain in one split.

Inter-annotator agreement: The degree to which annotators assign consistent labels.

Datasheet: Documentation describing a dataset’s purpose, composition, procedures, and limitations.

Review Questions

Short Answer Questions

1. What distinguishes a corpus from an unstructured collection of text?
2. Why can lowercasing be harmful in some NLP tasks?
3. Define code-switching and explain one processing challenge it creates.
4. What is data leakage? Give one example involving train-test splitting.
5. Why should metadata be documented and protected?

Long Answer Questions

1. Explain the stages of a professional multilingual text-preparation pipeline and discuss how errors can propagate between stages.
2. Compare whitespace, rule-based, word-level, subword, character, and byte tokenization. Identify appropriate use cases and limitations.
3. Design an annotation scheme for a multilingual customer-support intent corpus. Include label definitions, annotator training, disagreement handling, and quality measurement.
4. Discuss the legal, ethical, and privacy considerations involved in collecting social-media text for NLP research.
5. Explain why group-aware and chronological data splits may be preferable to random splitting. Include examples of misleading evaluation results.

Analytical / Application Questions

1. A sentiment dataset is 90% neutral, 7% positive, and 3% negative. Propose an evaluation and sampling strategy that would reveal minority-class performance without misrepresenting deployment conditions.
2. A multilingual tokenizer produces very long sequences for one low-resource language but short sequences for English. Analyse possible causes and recommend an investigation plan.
3. Design a corpus audit for a fraud-detection system containing conversations in English and code-switched Hindi-English. Specify checks for privacy, duplicates, leakage, annotation quality, language coverage, and temporal generalization.

Multiple Choice Questions with Answers

Question 1

Which property most directly distinguishes a well-designed corpus from a large unstructured text collection?

A. It contains only digitized books

B. It has a defined purpose, sampling procedure, and documentation

C. It uses exactly one tokenization method

D. It contains equal numbers of documents in every category

Correct Answer: B. It has a defined purpose, sampling procedure, and documentation

Question 2

Why should raw text usually be retained separately from normalized text?

A. Raw text always gives better model accuracy

B. It permits auditing and future reprocessing

C. It eliminates the need for metadata

D. It guarantees anonymity

Correct Answer: B. It permits auditing and future reprocessing

Question 3

Which operation is most likely to damage sentiment information in social-media text if applied indiscriminately?

A. Preserving message identifiers

B. Recording collection dates

C. Removing emojis and expressive punctuation

D. Checking file encoding

Correct Answer: C. Removing emojis and expressive punctuation

Question 4

What is a principal advantage of subword tokenization?

A. It guarantees one token per linguistic word

B. It removes the need for language identification

C. It can represent rare or unseen words through smaller units

D. It prevents all sequence-length problems

Correct Answer: C. It can represent rare or unseen words through smaller units

Question 5

In annotation, what is the main purpose of an adjudication procedure?

A. To increase the number of labels automatically

B. To resolve disagreements and refine interpretation of guidelines

C. To replace all human annotators with a tokenizer

D. To ensure every class has equal frequency

Correct Answer: B. To resolve disagreements and refine interpretation of guidelines

Question 6

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0. cinquante

C. 0.80

D. 0.50

Correct Answer: C. 0.80

Question 7

Which splitting strategy best prevents messages from the same conversation appearing in both training and test sets?

A. Character-level splitting

B. Group-aware splitting by conversation

C. Alphabetical splitting by first word

D. Splitting after removing all punctuation

Correct Answer: B. Group-aware splitting by conversation

Question 8

Why can overall accuracy be misleading on a highly imbalanced classification task?

A. Accuracy cannot be calculated for text

B. A majority-class classifier may achieve high accuracy while missing minority cases

C. Accuracy measures only the minority class

D. Accuracy automatically corrects annotation errors

Correct Answer: B. A majority-class classifier may achieve high accuracy while missing minority cases

Question 9

Which statement about Cohen’s kappa is correct?

A. It measures agreement while accounting for an expected-chance component

B. It is a measure of vocabulary size

C. It can only be used for continuous numerical targets

D. It proves that an annotation scheme is valid

Correct Answer: A. It measures agreement while accounting for an expected-chance component

Question 10

Which practice is most appropriate when collecting multilingual customer messages containing account numbers?

A. Publish all identifiers to improve reproducibility

B. Delete all messages regardless of task relevance

C. Mask identifiers, restrict access, and document the protection process

D. Translate identifiers into another language

Correct Answer: C. Mask identifiers, restrict access, and document the protection process