Text Data Acquisition and Preprocessing

Introduction

Natural language processing (NLP) systems learn from text corpora: organized collections of documents, utterances, messages, or other language records. The quality of a corpus strongly influences the quality, fairness, reliability, and reproducibility of the resulting model. A sophisticated neural architecture cannot compensate fully for duplicated, noisy, poorly labelled, legally unusable, or privacy-compromised data.

Text preprocessing is therefore not merely a sequence of cleaning commands. It is a set of design decisions about what constitutes a document, which linguistic distinctions should be preserved, how annotations are produced, and how data are made suitable for statistical and generative models. Different tasks require different treatments. Removing punctuation may be acceptable for some topic-classification tasks but harmful for sentiment analysis, code processing, question answering, or legal-document analysis.

This chapter develops a reproducible workflow from corpus creation to model-ready text. It covers acquisition, annotation, quality assurance, privacy, normalization, tokenization, stemming, lemmatization, and subword segmentation.

Learning Objectives

By the end of this chapter, students should be able to:

1. Design a legally and ethically defensible text-acquisition plan.
2. Construct a reproducible corpus with documented provenance and versioning.
3. Select annotation schemes and evaluate annotation agreement.
4. Apply normalization, tokenization, stemming, lemmatization, and subword segmentation appropriately.
5. Identify quality problems, leakage, bias, duplication, and privacy risks.
6. Prepare a reliable preprocessing pipeline for classical NLP and generative AI systems.

1. Corpus Design and Text Acquisition

A corpus is more than a folder of text files. It should have a defined population, sampling strategy, unit of analysis, metadata schema, and inclusion and exclusion criteria.

1.1 Defining the corpus

Important design questions include:

• What task will the corpus support: classification, retrieval, summarization, translation, dialogue, or language modelling?
• What is the unit of analysis: document, paragraph, sentence, turn, post, or token?
• Which languages, dialects, domains, time periods, and genres are represented?
• Is the corpus intended to describe a population or merely provide convenient training examples?
• Which sensitive attributes might be present?

A corpus for customer-support intent classification may consist of individual user messages. A corpus for document summarization may require complete documents paired with human-written summaries. Combining incompatible units can produce misleading labels and evaluation results.

1.2 Sources and acquisition methods

Common sources include licensed datasets, institutional archives, public websites, application logs, transcripts, surveys, and manually collected documents. Public availability does not automatically mean unrestricted reuse. Acquisition should record the source, collection date, licence, access conditions, geographical scope, and processing history.

Web collection requires particular care. A responsible collector should respect terms of service, robots guidance where relevant, copyright restrictions, rate limits, and personally identifiable information (PII) protections. Automatically downloading text without provenance makes later auditing difficult.

1.3 Sampling and representativeness

Convenience sampling is fast but can overrepresent highly active users, popular topics, or technologically connected populations. Stratified sampling can ensure coverage of predefined categories such as language, region, product type, or time period. Sampling decisions should be documented rather than hidden in code.

For supervised learning, class balance is also important. If 95% of messages are non-fraudulent, a classifier that always predicts “non-fraudulent” achieves 95% accuracy but has no practical fraud-detection value. The corpus should preserve the operational distribution for realistic testing while potentially using controlled sampling for training, with the difference explicitly recorded.

2. A Reproducible Corpus Pipeline

A professional pipeline separates immutable raw data from derived data. A typical structure is:

1. Raw layer: original files or records, retained under controlled access.
2. Ingested layer: standardized character encoding, file format, and metadata.
3. Filtered layer: records removed for legal, relevance, corruption, or duplication reasons.
4. Annotated layer: labels, spans, relations, or document-level metadata.
5. Model-ready layer: tokenized or otherwise transformed representations.
6. Evaluation layer: fixed validation and test sets, protected from training edits.

Each record should have a stable identifier. A useful record schema may include record_id, source_id, collection_time, language, text, licence, consent_status, annotator_version, and processing_version. Sensitive fields should be separated from the text whenever possible.

Reproducibility requires saving configuration files, software versions, random seeds, preprocessing code, data hashes, and inclusion/exclusion logs. A cryptographic hash can verify that a file has not changed, although a hash does not itself establish that the underlying material was lawfully obtained.

Train, validation, and test separation

Documents from the same conversation, user, patient, or near-duplicate source must not be distributed across training and test sets. Otherwise the model may memorize source-specific wording and produce an overly optimistic score. Splitting should occur at the highest level that could create dependence—for example, by user rather than message.

Preprocessing operations that learn from the corpus, such as vocabulary construction, frequency thresholds, imputation, or normalization statistics, should be fitted on the training set only. The same fitted operation is then applied to validation and test data.

3. Annotation and Labelling

Annotation converts raw language into structured information. Examples include sentiment labels, named-entity spans, part-of-speech tags, intent categories, toxicity judgements, and question-answer pairs.

A strong annotation protocol includes:

• an operational definition of each label;
• positive and negative examples;
• boundary rules for spans;
• treatment of ambiguity, code-switching, misspellings, and missing context;
• an “uncertain” or “not applicable” policy where appropriate;
• annotator training and an adjudication process.

Labels should represent the target construct rather than an easy proxy. For example, “contains a swear word” is not equivalent to “is abusive.” Annotation guidelines should state whether the task is to label the writer’s intention, the literal text, or the likely effect on a reader.

Annotation agreement

Agreement is useful for detecting unclear guidelines, but it is not identical to truth. For two annotators assigning categorical labels, Cohen’s kappa is commonly defined as:

Here, pₒ is the observed proportion of agreement and pₑ is the agreement expected by chance from the annotators’ marginal label frequencies. The statistic assumes categorical decisions and two annotators; weighted variants are appropriate for ordered categories, while multi-annotator settings may require other coefficients. Kappa can be affected strongly by class prevalence, so raw agreement and the label distribution should also be reported.

For a binary annotation task, precision and recall are often used to evaluate a system against a reference annotation:

Here, TP is the number of true positives, FP false positives, and FN false negatives. Precision measures the proportion of predicted positives that are correct; recall measures the proportion of reference positives that are found. These measures require clearly defined positive classes and should not be interpreted without considering the task’s costs.

4. Data Quality and Privacy

4.1 Quality dimensions

Important quality dimensions include:

Automated checks can detect empty documents, invalid Unicode, impossible timestamps, excessive repetition, abnormal length, encoding failures, and language mismatch. Manual sampling remains necessary because automated checks may miss semantic errors.

Deduplication should operate at more than one level. Exact duplicates can be detected using normalized string hashes. Near-duplicates may be identified with character n-gram similarity, MinHash, locality-sensitive hashing, or embedding similarity. Aggressive deduplication can remove legitimate repeated phrases, so thresholds and decisions must be documented.

4.2 Privacy-aware preparation

Personal data may include names, addresses, phone numbers, email addresses, account identifiers, health information, location traces, or indirect identifiers. Privacy preparation should follow data minimization: collect and retain only what is necessary for the stated purpose.

Possible controls include consent, access restriction, pseudonymization, redaction, aggregation, retention limits, and deletion procedures. Pseudonymization replaces identifiers with codes but does not make data anonymous if re-identification remains possible. Redaction can harm linguistic usefulness, so replacements should preserve the required structure when appropriate—for example, replacing an address with [ADDRESS] rather than deleting an entire message.

Privacy is not solved by removing obvious names. Rare combinations of facts can identify individuals, and model training may memorize sensitive passages. Access logs, encryption, role-based permissions, privacy impact assessments, and incident response procedures are part of responsible corpus governance. Applicable privacy, copyright, employment, health, and data-protection law must be reviewed by the responsible organization.

5. Unicode, Normalization, and Cleaning

Computers represent text through character encodings. UTF-8 is a dominant encoding for multilingual text, but valid UTF-8 does not guarantee linguistically consistent text. For example, visually similar characters may have different code points, and accented characters may be represented as a single code point or as a base character plus a combining mark.

Normalization may include:

• converting encoding to UTF-8;
• Unicode normalization, such as NFC or NFKC, when justified;
• standardizing line endings and whitespace;
• correcting known control characters;
• normalizing quotation marks or dashes;
• converting case where case is not informative.

NFKC can improve consistency but may reduce distinctions that matter, such as mathematical symbols, formatting, or compatibility characters. Lowercasing can harm named-entity recognition, acronym detection, and German noun interpretation. Cleaning should therefore be task-driven and reversible where possible.

HTML markup, boilerplate, navigation text, emojis, hashtags, URLs, and punctuation should not be removed automatically. Emojis may carry sentiment; punctuation may express questions or emphasis; URLs may identify a source or a harmful link. A robust pipeline either preserves such information or replaces it with explicit placeholders.

6. Tokenization

Tokenization divides text into units used by an algorithm. A token may be a word, punctuation mark, character, byte, or subword. There is no universally correct tokenizer.

Whitespace tokenization is simple but fails with punctuation, contractions, agglutinative languages, social-media text, and scripts without spaces. Rule-based tokenizers can preserve constructs such as email addresses and decimal numbers. Sentence segmentation must handle abbreviations, quotations, and multilingual punctuation.

For generative language models, tokenization is usually subword- or byte-based. The tokenizer used during training must be used during inference; changing it changes the model’s input representation. Special tokens such as beginning-of-sequence, end-of-sequence, padding, unknown, or separator markers require explicit handling.

Tokenization decisions affect sequence length, vocabulary size, memory use, and the model’s ability to represent rare words. For a fixed context window, inefficient segmentation can reduce the amount of meaningful text that fits into the input.

7. Stemming and Lemmatization

Stemming applies heuristic rules to remove affixes, often producing a non-word stem. For example, several English forms may be reduced to a common approximate form. Stemming is computationally inexpensive and can improve matching in information retrieval, but it may conflate unrelated words or produce unnatural outputs.

Lemmatization maps an inflected form to a dictionary lemma using morphological and often part-of-speech information. “Studies” may map to “study” when used as a noun or verb, while context can determine the correct analysis. Lemmatization is generally more linguistically informed but requires language resources and can be slower or less reliable for noisy text.

Neither technique should be applied automatically to every NLP task. Search systems may benefit from conflating inflections; a text-generation system normally needs the original surface form. For languages with rich morphology, language-specific analyzers may be more appropriate than English-oriented stemmers.

8. Subword Segmentation

Subword methods address the conflict between a small vocabulary and the need to represent rare or unseen words. Common strategies include byte-pair encoding (BPE), WordPiece, and unigram language-model tokenization.

BPE begins with small units, often characters or bytes, and repeatedly merges frequent adjacent units until a target vocabulary size is reached. If the merge rules are learned from a training corpus, they must be saved and applied identically at deployment. WordPiece selects units according to a likelihood-oriented vocabulary objective, while unigram tokenization begins with a larger candidate vocabulary and removes pieces according to a probabilistic criterion.

Subword segmentation improves open-vocabulary coverage, but it is not automatically fair or linguistically optimal. A tokenizer trained mainly on English may split low-resource languages into many pieces. This increases sequence length and may weaken representation quality. Evaluation should therefore examine token counts and unknown or fragmented forms across languages, domains, names, code, and dialects.

9. A Practical Preprocessing Blueprint

A robust implementation can follow this sequence:

1. Define task, population, legal basis, and data schema.
2. Acquire data through documented and permitted channels.
3. Preserve raw records under restricted access.
4. Validate encoding, metadata, and structural integrity.
5. Remove or quarantine irrelevant, corrupt, and prohibited records.
6. Detect exact and near duplicates.
7. Apply privacy controls and record their version.
8. Split data by an independence unit such as user or document source.
9. Annotate using tested guidelines and measure agreement.
10. Fit training-only vocabulary and preprocessing resources.
11. Normalize and tokenize with versioned tools.
12. Run quality reports and manually inspect samples.
13. Export model-ready data with provenance and checksums.
14. Monitor performance and revise the pipeline when data distribution changes.

A pipeline should produce reports, not only transformed files. Useful reports include document counts, language proportions, length distributions, label frequencies, duplicate rates, PII detections, token statistics, and exclusions by reason.

10. Illustrative Case: Customer-Support Intent Corpus

Background

A telecommunications provider wants a generative assistant that routes customer messages to billing, network failure, cancellation, device support, and account-security workflows. Messages contain informal spelling, screenshots converted to OCR text, names, phone numbers, and account identifiers.

Problem

An initial dataset was assembled from chat logs. It contained repeated messages from frequent users, inconsistent labels, and personally identifying information. Random message-level splitting caused messages from the same conversation to appear in both training and test sets.

Analysis

The team first created a data dictionary and identified the conversation as the independence unit. A sample audit found that “angry” messages had been labelled inconsistently: some annotators used emotional tone, while others labelled the operational intent. The dataset also contained copied bot responses, which inflated apparent performance.

Approach

Raw logs were stored separately with access controls. Account numbers, telephone numbers, email addresses, and names were replaced by typed placeholders. Conversations were grouped by a stable pseudonymous customer-session identifier before splitting. Annotators received intent definitions and examples, and disagreements were adjudicated. Bot-generated text was marked and excluded from the user-intent training set.

The team preserved punctuation and emojis for sentiment-related experiments, but used a separate intent pipeline with placeholders for URLs and identifiers. A tokenizer and vocabulary were trained only on the training partition. A report recorded every filter, replacement rule, software version, and dataset hash.

Outcome

The revised test score was lower than the original score, but it better reflected deployment performance. Error analysis showed that cancellation requests containing billing complaints were genuinely ambiguous, leading to a multi-intent escalation policy rather than forced single labels.

Learning points

The case demonstrates that data governance, split design, annotation validity, privacy protection, and preprocessing choices are part of model engineering. A lower but trustworthy evaluation is more valuable than a high score produced by leakage.

11. Limitations and Contemporary Developments

Preprocessing can introduce bias. Removing dialect markers may erase identity and meaning; aggressive profanity filtering may disproportionately affect communities whose language is misclassified; and language identification tools may perform poorly on code-switching. Annotation categories can also impose culturally narrow assumptions.

Large language models increasingly use byte-level or multilingual subword tokenizers, retrieval pipelines, structured document parsers, and data-mixture weighting. Synthetic data can expand coverage, but it may reproduce model errors, reduce diversity, or contaminate evaluation sets. Deduplication and provenance tracking are particularly important when public model outputs are included in a corpus.

Modern data-centric practice emphasizes dataset cards, model cards, data lineage, privacy review, red-team testing, and continuous monitoring. Governance frameworks should specify who may access data, for what purpose, for how long, and how individuals can request correction or deletion where applicable.

Best-Practice Checklist

• Start with a task and population definition, not with a convenient dataset.
• Preserve raw data and make transformations reproducible.
• Document licences, consent, provenance, and retention.
• Split by an independence unit to prevent leakage.
• Test annotation guidelines before large-scale labelling.
• Report agreement, class distributions, and uncertainty.
• Treat normalization as a modelling decision.
• Preserve information that may carry meaning, including punctuation and emojis.
• Fit learned preprocessing resources on training data only.
• Evaluate tokenization across languages and demographic groups.
• Minimize, protect, and de-identify personal data.
• Inspect samples manually and maintain an exclusion log.

Chapter Summary

Text data acquisition and preprocessing form a reproducible engineering and governance process. A useful corpus has a defined purpose, population, sampling strategy, provenance record, and metadata schema. Raw, derived, annotated, and model-ready layers should be separated and versioned. Train–validation–test separation must respect dependence among users, documents, and conversations.

Annotation requires operational definitions, boundary rules, training, adjudication, and agreement analysis. Quality assurance addresses validity, completeness, consistency, uniqueness, coverage, timeliness, and duplication. Privacy-aware preparation applies minimization, access control, pseudonymization or redaction, retention policies, and legal review.

Unicode normalization, cleaning, tokenization, stemming, lemmatization, and subword segmentation are task-dependent choices. Subword methods support open-vocabulary modelling but can create unequal fragmentation across languages and domains. Reliable pipelines preserve meaningful information, fit learned resources only on training data, generate audit reports, and monitor changes after deployment.

Key Terms

Corpus: An organized collection of language data for analysis or model development.

Provenance: Information describing where data originated and how they were transformed.

Metadata: Structured information about a record, such as source, time, language, and licence.

Data leakage: Unintended use of information from evaluation data during training or preprocessing.

Annotation: The process of assigning labels, spans, relations, or other structured information to text.

Adjudication: Resolution of disagreements among annotators.

Cohen’s kappa: A chance-adjusted agreement statistic for two categorical annotators.

PII: Personally identifiable information that can identify or help identify a person.

Pseudonymization: Replacing identifiers with codes while retaining possible re-identification risk.

Normalization: Transformation of variant text representations into a selected consistent form.

Unicode: A standard system for representing characters across writing systems.

Tokenization: Division of text into computational units called tokens.

Stemming: Heuristic reduction of words to approximate stems.

Lemmatization: Mapping an inflected form to a linguistically defined base lemma.

Subword: A token unit smaller than a full word but larger than, or equal to, a character or byte.

Byte-pair encoding: A frequent-pair merging method used to learn subword vocabularies.

Deduplication: Identification and removal or control of repeated records.

Data minimization: Collection and retention of only data necessary for a stated purpose.

Corpus shift: Change in the distribution of data between collection, training, and deployment.

Review Questions

Short Answer Questions

1. Why should raw and model-ready corpus layers be stored separately?
2. Distinguish pseudonymization from anonymization.
3. What is the purpose of an annotation guideline?
4. Give two reasons why lowercasing may be harmful.
5. Why must a tokenizer be versioned with a generative model?

Long Answer Questions

1. Explain how you would design a reproducible corpus for intent classification, from acquisition through evaluation.
2. Compare stemming, lemmatization, and subword segmentation in terms of linguistic information, computational cost, and suitable applications.
3. Discuss how train–test leakage can occur in conversational text and how it can be prevented.
4. Explain the role of data quality reports and manual inspection in a preprocessing pipeline.
5. Evaluate the ethical and governance requirements for preparing a corpus containing customer messages.

Analytical / Application Questions

1. A dataset contains 80 positive and 920 negative examples. Explain why accuracy alone may be misleading and propose suitable additional metrics and sampling controls.
2. Two annotators agree on 90 of 100 records, while their label frequencies are very different. Explain why observed agreement and Cohen’s kappa may tell different stories, and identify information needed for interpretation.
3. Design a preprocessing policy for multilingual social-media data containing emojis, URLs, code-switching, misspellings, and personal identifiers. Justify which information you would preserve, replace, or remove.

Multiple Choice Questions with Answers

Question 1

Which practice most directly prevents conversation-level leakage between training and test data?

A. Lowercasing every message

B. Splitting records by conversation or user

C. Removing all punctuation

D. Increasing the vocabulary size

Correct Answer: B. Splitting records by conversation or user

Question 2

In Cohen’s kappa, what does pₑ represent?

A. The observed agreement proportion

B. The proportion of missing labels

C. The agreement expected by chance from marginal frequencies

D. The proportion of correctly tokenized words

Correct Answer: C. The agreement expected by chance from marginal frequencies

Question 3

Which statement best describes pseudonymization?

A. It guarantees that re-identification is impossible

B. It replaces identifiers while possible re-identification risk remains

C. It deletes every record containing personal data

D. It converts text into subword tokens

Correct Answer: B. It replaces identifiers while possible re-identification risk remains

Question 4

Why can aggressive lowercasing harm named-entity recognition?

A. It increases file size

B. It removes distinctions carried by capitalization

C. It prevents UTF-8 encoding

D. It creates duplicate documents automatically

Correct Answer: B. It removes distinctions carried by capitalization

Question 5

Which technique is designed to represent rare or unseen words using smaller units?

A. Exact deduplication

B. Sentence segmentation

C. Subword segmentation

D. Metadata validation

Correct Answer: C. Subword segmentation

Question 6

A system predicts 40 positive cases, of which 30 are correct. What is its precision?

A. 0.25

B. 0.30

C. 0.75

D. 0.90

Correct Answer: C. 0.75

Question 7

Which resource should generally be fitted using training data only?

A. The production legal policy

B. A learned vocabulary and frequency threshold

C. The definition of UTF-8

D. The record identifier format

Correct Answer: B. A learned vocabulary and frequency threshold

Question 8

What is a principal risk of whitespace tokenization?

A. It cannot store metadata

B. It always removes privacy information

C. It handles punctuation and contractions poorly

D. It requires a neural network

Correct Answer: C. It handles punctuation and contractions poorly

Question 9

Which quality dimension asks whether repeated records have been identified?

A. Uniqueness

B. Timeliness

C. Validity

D. Completeness

Correct Answer: A. Uniqueness

Question 10

Why should emojis sometimes be preserved during preprocessing?

A. They always identify the author

B. They can carry sentiment or pragmatic meaning

C. They reduce every document to one token

D. They eliminate the need for annotation

Correct Answer: B. They can carry sentiment or pragmatic meaning