Language Data Acquisition and Preparation

Introduction

Deep-learning and natural-language-processing systems learn patterns from language data. Their performance therefore depends not only on model architecture, but also on how text is acquired, labelled, cleaned, represented and divided for evaluation. A sophisticated transformer trained on unreliable, duplicated or improperly split data may produce misleadingly strong results while failing in deployment.

Language data preparation is a process of transforming raw communication into a documented, analysable corpus. The process includes defining the task, acquiring data legally and ethically, designing annotation procedures, cleaning and normalising text, segmenting it into tokens, and controlling quality, imbalance and information leakage. These operations are not merely mechanical preprocessing steps: each one encodes assumptions about language and can affect what a model learns.

Learning Objectives

By the end of this chapter, you should be able to:

• define a corpus and select suitable sources for an NLP task;
• distinguish documents, instances, labels, metadata and annotations;
• design a practical labelling scheme and assess annotation quality;
• clean and normalise text without destroying task-relevant information;
• compare tokenisation, stemming, lemmatisation and subword segmentation;
• identify duplication, class imbalance and train–test leakage;
• construct reproducible data-management workflows;
• evaluate ethical, legal and governance risks in language datasets.

1. Language Corpora and Data Sources

A corpus is a structured collection of language samples assembled for a particular purpose. A sample may be a document, sentence, social-media post, speech transcript, support ticket or dialogue turn. A corpus is more useful when its scope, origin, time period, language varieties, collection method and intended task are documented.

A corpus can be:

• raw, containing text without labels;
• annotated, containing human- or machine-assigned labels;
• parallel, containing aligned texts in two or more languages;
• domain-specific, such as medical reports or legal contracts;
• balanced, designed to represent categories or demographic groups deliberately;
• continual, updated over time as new language data arrive.

Common acquisition sources include public web pages, government publications, books obtained under licence, internal organisational records, product reviews, messaging platforms, speech transcripts and crowdsourcing platforms. Each source has different coverage and risks. Web data are broad but may contain spam, personal information, duplicated pages and uncertain copyright status. Internal data may be highly relevant but require access controls, de-identification and contractual review.

Before collecting data, specify a data statement. It should record the task, population, languages, domains, time range, sampling method, inclusion and exclusion criteria, licence, known limitations and responsible contact. A useful target is not simply “more text,” but text that is relevant, representative, legally usable and sufficiently documented.

Corpus units and metadata

A corpus normally contains:

1. a unique identifier for each instance;
2. the original text, preserved separately from processed versions;
3. labels or target values;
4. metadata such as source, date, language and domain;
5. provenance information describing how the instance was obtained;
6. processing history, including software versions and transformations.

The original text should be treated as immutable. Cleaned text should be stored as a derived field so that errors can be traced and preprocessing can be repeated.

For a labelled corpus with class counts n₁, n₂, …, nₖ, the total number of instances is:

Here, N is the total number of instances, k is the number of classes, and nᵢ is the count in class i. The class proportion is:

This proportion is meaningful only when instances are counted consistently and each instance belongs to one class. For multilabel tasks, where an instance may have several labels, the class counts need not sum to N.

2. Labelling and Annotation Design

Labelling converts a linguistic task into an operational decision. For sentiment classification, labels might be positive, negative and neutral. For named-entity recognition, labels identify spans such as person, organisation and location. For summarisation, labels may be reference summaries rather than simple categories.

A sound annotation scheme contains:

• a precise definition for every label;
• inclusion and exclusion rules;
• positive and borderline examples;
• instructions for overlapping or nested cases;
• a policy for missing, uncertain or unclassifiable instances;
• a procedure for resolving disagreements.

Instructions should be written before large-scale annotation begins. A pilot set allows annotators to identify ambiguous categories and revise the scheme. It is usually preferable to label a smaller, consistently defined dataset than a larger dataset with unstable labels.

Human agreement

When two annotators independently label the same nominal-category items, observed agreement is:

However, agreement can appear high when one class is very common. Cohen’s kappa adjusts for agreement expected by chance:

Pₒ is observed agreement and Pₑ is expected agreement calculated from the annotators’ marginal label frequencies. The statistic assumes two annotators, categorical labels and independent item-level decisions. It is not automatically suitable for ordinal ratings, multilabel annotation or span-level sequence annotation. For those settings, alternatives such as weighted kappa, Krippendorff’s alpha or span-level precision and recall may be more appropriate.

Agreement statistics are diagnostic rather than absolute proof of quality. Low agreement may indicate unclear instructions, genuinely ambiguous language or insufficient annotator training. High agreement may reflect an easy task or an overly narrow sample.

Labelling workflow

A professional workflow is:

1. define the task and unit of annotation;
2. draft the label ontology and guidelines;
3. train annotators with worked examples;
4. conduct a pilot annotation;
5. measure disagreements and revise instructions;
6. annotate the production set, preferably with overlap on a quality sample;
7. adjudicate disagreements and record reasons;
8. version the labels and preserve the annotation history.

Automated or weak labelling can reduce cost, but its errors must be measured. A model-generated label should not silently be treated as equivalent to an expert label. Confidence scores, provenance and review status should be retained.

3. Data Cleaning and Normalisation

Cleaning removes corrupt, irrelevant or unusable material. Normalisation transforms alternative forms into a more consistent representation. Neither should be applied blindly: a transformation that helps topic classification may harm authorship attribution, sentiment analysis or offensive-language detection.

Typical cleaning operations

Common operations include:

• removing exact duplicate records;
• detecting near-duplicate documents;
• repairing encoding errors;
• extracting text from structured files;
• removing boilerplate, navigation menus and markup when irrelevant;
• standardising line breaks and whitespace;
• identifying empty, extremely short or corrupted instances;
• filtering unwanted languages or domains;
• redacting or replacing personal identifiers where required.

Unicode deserves particular attention. Visually identical text may have different underlying code-point sequences. Unicode normalisation, such as NFC, can combine compatible character representations, while NFKC performs broader compatibility transformations. NFKC can change distinctions that matter, so the chosen form must be documented and tested. For example, mathematical symbols, full-width characters and special typography may carry useful information in some domains.

Case folding converts text to a common case, often lowercase. This can reduce vocabulary size, but it removes distinctions such as “US” versus “us” and may affect named-entity recognition. Retain the original text whenever possible and create a separate normalised field.

Punctuation, digits, URLs, emojis and hashtags should be handled according to the task. Removing punctuation may damage sentiment cues, code, financial expressions or sentence boundaries. URLs may be replaced with a placeholder such as <URL> when their identity is irrelevant, but domain names may be predictive in phishing detection. Emojis can be retained, mapped to descriptions or tokenised separately.

Quality checks

Quality control should combine rules, sampling and statistics. Useful checks include:

• percentage of empty records;
• language-identification confidence;
• character and token length distributions;
• duplicate and near-duplicate rates;
• invalid labels or impossible metadata values;
• unexpected personal information;
• label frequency by source, date and subgroup.

A cleaning script should produce logs: records read, records removed, reasons for removal and records written. Silent deletion makes later auditing difficult.

4. Tokenisation and Text Representation

Tokenisation divides text into units that a model can process. Units may be paragraphs, sentences, words, punctuation marks, characters or subword pieces. The best choice depends on the language, model and task.

Sentence and word tokenisation

Sentence tokenisation identifies sentence boundaries. A simple rule based on full stops can fail with abbreviations, decimal numbers, web addresses and informal writing. Word tokenisation based on whitespace also fails with punctuation, contractions, clitics and languages that do not use spaces consistently.

A tokenizer should define how it handles examples such as “don’t,” “U.K.,” “state-of-the-art,” email addresses and emojis. Reproducibility requires the exact tokenizer and version to be recorded.

Stemming

Stemming removes or replaces word endings using heuristic rules. The words “connect,” “connected” and “connecting” may be reduced to a common stem, but the result need not be a valid word. Stemming is fast and sometimes useful for information retrieval, but aggressive stemming can merge unrelated words or fail with irregular morphology.

Lemmatisation

Lemmatisation maps an inflected word to a dictionary or morphological base form, called a lemma. “Was” may map to “be,” and “mice” to “mouse,” depending on the analyser. Lemmatisation generally requires part-of-speech and language-specific resources. It is more linguistically informed than stemming but slower and imperfect, especially for slang, names and emerging vocabulary.

Pretrained neural models often expect their own tokenizer. Applying an unrelated stemming or lowercasing procedure before that tokenizer can make the input distribution unlike the one used during pretraining.

5. Subword Segmentation

Word-level vocabularies struggle with names, spelling variations and newly created words. Character-level models avoid unknown words but may require long sequences. Subword methods provide a compromise by representing frequent words as single units and rare words as combinations of smaller pieces.

Popular approaches include byte-pair encoding (BPE), WordPiece and unigram language-model tokenisation. In a simplified BPE procedure, the algorithm begins with character symbols, counts adjacent symbol pairs in a training corpus, merges the most frequent pair, and repeats until the desired vocabulary size is reached. The learned merge rules are then applied consistently to new text.

For example, a word such as “unhappiness” might be represented by pieces corresponding approximately to “un,” “happi” and “ness.” The exact pieces depend on the vocabulary and algorithm. Subword segmentation is not the same as linguistic morpheme analysis: a frequent piece may be statistically useful without being a meaningful morpheme.

Vocabulary size involves a trade-off. A larger vocabulary produces fewer pieces per sentence but requires more embedding parameters and may retain more rare forms. A smaller vocabulary reduces the vocabulary table but creates longer sequences. Tokenisation should be evaluated by downstream performance, sequence length, memory use and behaviour on domain-specific terms.

Tokenisation must be trained without using held-out test text when the tokenizer learns vocabulary, merges or statistics. Otherwise, information from the test set can influence the representation.

6. Data Splitting, Imbalance and Leakage

Splitting correctly

A dataset is commonly divided into training, validation and test sets. The training set is used to fit model parameters; the validation set supports model selection and hyperparameter decisions; the test set is reserved for final evaluation. A random split is not always valid. Documents from the same author, patient, customer, conversation or source event should usually remain in one partition.

For temporal applications, training on earlier records and testing on later records better reflects deployment. For cross-user generalisation, group-based splitting prevents the model from memorising user-specific language.

Imbalance

Class imbalance occurs when categories have substantially different frequencies. A classifier that always predicts the majority class may obtain high accuracy while failing the minority class. For binary classification, accuracy is:

Here TP and TN are true positives and true negatives; FP and FN are false positives and false negatives. Precision and recall are:

The formulas assume a defined positive class and nonzero denominators. Precision measures the proportion of predicted positives that are correct; recall measures the proportion of actual positives that are found. The F1 score is the harmonic mean:

It is useful when both precision and recall matter, but it does not represent all operational costs.

Responses to imbalance include targeted data collection, class-weighted loss, careful oversampling of the training set, undersampling, threshold adjustment and suitable metrics. Oversampling must occur after splitting and only within the training partition. Validation and test sets should retain a realistic deployment distribution unless the evaluation question explicitly requires another distribution.

Leakage

Data leakage occurs when information unavailable at prediction time influences training or evaluation. Examples include:

• placing near-duplicate documents in both training and test sets;
• using future labels or post-outcome text as features;
• fitting a vocabulary, normaliser or feature selector on all data before splitting;
• allowing the same user, patient or conversation into multiple partitions;
• selecting the final model repeatedly against the test set;
• incorporating human annotations derived from the target outcome.

Leakage produces overoptimistic results. A leakage audit should examine identifiers, timestamps, source relationships, duplicate similarity, preprocessing fit stages and feature availability at deployment time.

7. A Reproducible Preparation Pipeline

A practical pipeline can be organised as follows:

1. Define the task: specify input, output, population, deployment context and evaluation measure.
2. Acquire and document: record licences, provenance, sampling and collection dates.
3. Preserve raw data: store an access-controlled immutable copy.
4. Validate: inspect encoding, schema, language, length and metadata.
5. Deduplicate: remove exact and near duplicates using documented rules.
6. Protect people: redact personal data where necessary and restrict access.
7. Split by the correct unit: use document, group or time-based splitting as appropriate.
8. Fit training-dependent operations: learn vocabulary, normalisation statistics or sampling rules using training data only.
9. Tokenise and encode: apply the same versioned procedure to all partitions.
10. Audit and publish documentation: report exclusions, distributions, limitations and known risks.

A data version should identify the source snapshot, code revision, configuration, tokenizer vocabulary and label-schema version. Reproducibility is not achieved merely by saving a final CSV file; it requires a traceable chain from source to model input.

8. Ethical, Legal and Governance Considerations

Language data often contain names, opinions, health information, location traces or confidential communication. Collection and use should follow applicable privacy and data-protection requirements, organisational policy and licence terms. Public availability does not automatically mean unrestricted ethical usability.

Important principles include purpose limitation, data minimisation, access control, retention limits and secure deletion. De-identification reduces risk but is not perfect: combinations of dates, locations and rare phrases can permit re-identification. Sensitive datasets require risk assessment and, where appropriate, ethics review.

Annotation work also raises labour concerns. Annotators may encounter violence, abuse or distressing content. They need clear contracts, fair compensation, training, psychological support and the ability to pause or refuse harmful tasks.

Representational bias can enter through source selection, missing groups, label definitions and annotator assumptions. Report performance by relevant subgroups where lawful and appropriate. Do not infer sensitive attributes casually from language. A model should not be deployed solely because its aggregate score is high if it fails systematically for a language variety or community.

9. Contemporary Practice: Foundation Models and Domain Adaptation

Modern language models use large-scale pretraining followed by task adaptation. Their tokenizers and vocabularies are usually fixed or learned from broad corpora. Domain-specific text may be poorly represented: chemical names, indigenous languages, code-switching, clinical abbreviations and local dialects can fragment into many subword pieces.

Domain adaptation can involve continued pretraining, supervised fine-tuning, retrieval-based methods or a newly trained tokenizer. Each option requires careful data governance. Removing personally identifiable information before adaptation is preferable to hoping that a model will not memorise it. Deduplication is especially important because repeated text can increase memorisation and distort evaluation.

Synthetic text can expand a corpus, but generated examples may reproduce errors, stereotypes or model-specific phrasing. Synthetic data should be labelled as such, checked against real distributions and evaluated for whether it improves performance on genuinely held-out human data.

10. Applied Case: Classifying Urgent Customer-Support Messages

Background

An illustrative telecommunications provider wants to classify incoming support messages as routine, urgent service failure or billing concern. The intended use is triage, not automatic denial of service.

Problem

The provider has 120,000 historical messages. Many are repeated templates, some contain account numbers, and urgent cases represent only 8% of the data. Messages from the same customer appear across several months. Initial random splitting produces an apparently excellent test score.

Analysis

An audit finds that identical outage announcements occur in both training and test sets. Customer identifiers also permit memorisation of writing patterns. The test set therefore does not represent performance on unseen customers or future incidents. The original labels are inconsistent: one team labels a message urgent when it describes a current outage, while another labels it urgent when the customer expresses anger.

Approach

The team first writes a label guide defining urgency by operational impact and time sensitivity. Two annotators review a pilot sample and adjudicate disagreements. Account numbers, phone numbers and email addresses are replaced with typed placeholders. Exact and near duplicates are removed, while the raw records are retained in a restricted store.

The data are split by customer, with the final eight weeks reserved as a temporal test set. The tokenizer is fitted only on training data. Because urgency is a minority class, the team reports precision, recall and F1 for urgent messages rather than accuracy alone. Class weighting is applied during training, while the validation and test distributions remain unchanged.

Outcome

The revised score is lower than the original score, but it better estimates deployment behaviour. Error analysis shows that short messages such as “still down” are difficult without conversation history. The provider therefore adds a controlled context field and requires human review for uncertain urgent predictions.

Learning points

The case demonstrates that data preparation is part of system design. Clear labels improve validity; group and temporal splitting reduce leakage; privacy protection reduces operational risk; and minority-class metrics align evaluation with the business purpose. A lower but credible score is more valuable than a high score produced by duplicated information.

Challenges and Limitations

No preprocessing pipeline is universally optimal. Aggressive cleaning can erase dialect, identity, emotion or domain terminology. Stemming may damage meaning; lemmatisation depends on imperfect linguistic resources; and subword tokenisation may disadvantage languages or domains underrepresented in the tokenizer’s training data. Annotation agreement does not guarantee conceptual validity, while automated language detection and toxicity filters can reproduce bias.

Data distributions also change. New products, events, slang and policies cause drift. A corpus should therefore include collection dates and support periodic monitoring. When a model is updated, compare both data distributions and error patterns rather than relying on a single headline metric.

Best-Practice Checklist

• Define the intended use before collecting text.
• Preserve raw data and maintain provenance.
• Document licences, sampling and known gaps.
• Use explicit, tested annotation guidelines.
• Measure agreement and investigate disagreement.
• Keep original and processed text separately.
• Make cleaning task-dependent and reversible where possible.
• Train tokenizers and preprocessing statistics only on training data.
• Split by person, group, source or time when random splitting is unsafe.
• Audit duplicates, imbalance, privacy risks and leakage.
• Report subgroup and error-analysis results.
• Version data, labels, code and tokenizer configuration.

Chapter Summary

Language data preparation transforms raw communication into reliable input for NLP systems. A corpus requires clear scope, provenance, metadata and legal and ethical justification. Annotation quality depends on operational definitions, pilot work, disagreement analysis and documented adjudication. Cleaning and normalisation should remove genuine defects without erasing task-relevant signals. Tokenisation may operate at word, character or subword level, while stemming and lemmatisation offer different compromises between simplicity and linguistic precision.

Imbalance makes accuracy potentially misleading, so precision, recall and F1 should be selected according to the application. Leakage can arise through duplicates, shared users, future information or preprocessing fitted on held-out data. Group- and time-aware splitting, immutable raw data, version control and documented audits are central professional practices. Finally, privacy, representational bias, annotation labour and distributional change must be considered alongside technical performance.

Key Terms

Corpus: A structured collection of language samples assembled for a purpose.

Annotation: The assignment of labels or linguistic information to data.

Provenance: A record of where data came from and how they were transformed.

Metadata: Descriptive information about an instance, such as source or date.

Normalisation: Transformation of alternative forms into a consistent representation.

Unicode normalisation: Standardisation of equivalent or compatible character sequences.

Tokenisation: Division of text into processing units called tokens.

Stemming: Heuristic reduction of words to root-like forms.

Lemmatisation: Mapping of an inflected form to a dictionary base form.

Subword segmentation: Division of words into statistically learned pieces.

BPE: Byte-pair encoding, an iterative frequent-pair merging method.

Class imbalance: Unequal representation of target classes.

Precision: The proportion of predicted positives that are correct.

Recall: The proportion of actual positives that are retrieved.

F1 score: The harmonic mean of precision and recall.

Data leakage: Unavailable or inappropriate information entering training or evaluation.

Deduplication: Identification and removal or control of repeated records.

Group split: A partition that keeps related entities in one dataset split.

Data drift: Change in data distribution over time.

Data statement: Documentation describing a dataset’s purpose, origin, composition and limitations.

Review Questions

Short Answer Questions

1. What is a corpus, and why should its provenance be documented?
2. Distinguish stemming from lemmatisation.
3. Why can accuracy be misleading for an imbalanced classification task?
4. Give two examples of train–test leakage.
5. What information should be included in annotation guidelines?

Long Answer Questions

1. Explain how you would design and validate an annotation scheme for sentiment classification.
2. Compare word-level, character-level and subword tokenisation, including their principal advantages and limitations.
3. Describe a reproducible pipeline for preparing a text corpus from acquisition through model-ready encoding.
4. Discuss the ethical and legal risks of collecting language data from public online sources.
5. Explain why group-based or temporal splitting may be preferable to random splitting.

Analytical / Application Questions

1. A dataset contains 9,000 routine messages and 1,000 urgent messages. Propose an evaluation and training strategy that avoids misleading conclusions about urgent-message performance.
2. A model achieves 96% accuracy after randomly splitting messages, but performance falls sharply when tested on messages from new customers. Diagnose the likely causes and propose an improved experiment.
3. A medical NLP team plans to lowercase all text, remove punctuation and train a tokenizer on the complete dataset before splitting. Critically evaluate this plan and recommend safer alternatives.

Multiple Choice Questions with Answers

Question 1

Which practice best prevents information leakage when learning a tokenizer vocabulary?

A. Fit the tokenizer on the training partition only

B. Fit the tokenizer on training and test text together

C. Fit the tokenizer on the test partition first

D. Fit a separate tokenizer after model evaluation

Correct Answer: A. Fit the tokenizer on the training partition only

Question 2

What does recall measure?

A. The proportion of predicted positives that are correct

B. The proportion of actual positives that are correctly retrieved

C. The proportion of all predictions that are correct

D. The proportion of predicted negatives that are correct

Correct Answer: B. The proportion of actual positives that are correctly retrieved

Question 3

Which situation is most likely to create train–test leakage?

A. Removing empty records before splitting

B. Keeping a customer’s messages in one group

C. Putting near-duplicate documents in both training and test sets

D. Reporting precision for the minority class

Correct Answer: C. Putting near-duplicate documents in both training and test sets

Question 4

What is the principal purpose of lemmatisation?

A. To divide a sentence into characters

B. To map an inflected word to a base dictionary form

C. To remove every punctuation mark

D. To merge the most frequent adjacent symbol pairs

Correct Answer: B. To map an inflected word to a base dictionary form

Question 5

Why are subword methods useful in NLP?

A. They guarantee that every token is a complete dictionary word

B. They eliminate the need for a vocabulary

C. They can represent rare or unseen words through smaller pieces

D. They always produce shorter sequences than word tokenisation

Correct Answer: C. They can represent rare or unseen words through smaller pieces

Question 6

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0.50

C. 0.80

D. 0.าษ

Correct Answer: C. 0.80

Question 7

Which action is most appropriate when labels have frequent disagreements during a pilot?

A. Delete all disputed examples without investigation

B. Revise the guidelines and examine the sources of disagreement

C. Allow each annotator to use a private label definition

D. Report only the majority label and hide disagreement

Correct Answer: B. Revise the guidelines and examine the sources of disagreement

Question 8

Why might lowercasing harm named-entity recognition?

A. It increases the number of punctuation marks

B. It can remove case distinctions that provide entity cues

C. It always creates data leakage

D. It prevents sentence segmentation

Correct Answer: B. It can remove case distinctions that provide entity cues

Question 9

Which split is most suitable for estimating performance on future records?

A. A temporal split with earlier training data and later test data

B. A split that duplicates each record in every partition

C. A test set sampled only from the training period

D. A split based solely on message length

Correct Answer: A. A temporal split with earlier training data and later test data

Question 10

What is a central limitation of stemming?

A. It always requires a large labelled corpus

B. It may produce invalid forms or merge unrelated words

C. It cannot process alphabetic text

D. It is identical to dictionary-based lemmatisation

Correct Answer: B. It may produce invalid forms or merge unrelated words