Text and social media analytics begins long before modelling. The quality, legality, representativeness and reproducibility of an analysis depend on how data are identified, collected, documented, stored and prepared. A sophisticated sentiment model cannot compensate for a biased sample, missing context, unreliable timestamps or an undocumented change in an application programming interface (API).

For business analysts, “data” is not simply a collection of text files. It is an evidence system consisting of content, contextual metadata, provenance, sampling decisions, platform rules, access conditions and governance controls. A customer review, for example, may contain the review text, rating, product identifier, language, date, verified-purchase flag, helpfulness votes and information about the collection method. Removing these fields may make modelling easier, but it can also destroy the ability to interpret or audit the result.

This chapter examines sources of text and social data; collection through APIs, web sources, enterprise systems and documents; sampling and coverage; metadata; data quality; storage architectures; privacy and governance; and reproducible workflows.

Learning Objectives

1. The Data-Source Landscape

1.1 Documents and organisational records

Documents include annual reports, contracts, policy manuals, emails, call transcripts, support tickets, survey responses, news articles and research publications. They are often rich in domain language and institutional knowledge. Their weaknesses include inconsistent formatting, uncertain authorship, outdated versions and restricted access.

Document collections should be treated as versioned records rather than undifferentiated text. A contract amendment, an updated policy and a scanned copy of the same policy may all appear to be separate documents while representing one evolving source. Useful fields include document identifier, title, author, department, creation date, effective date, version, language, confidentiality classification and source location.

Optical character recognition (OCR) can convert scanned pages into machine-readable text, but OCR errors are not random. Tables, handwriting, poor scans and specialist symbols may be disproportionately affected. OCR output should therefore retain a link to the original image and, where material decisions depend on it, undergo quality checking.

1.2 Reviews and user-generated content

Reviews provide opinions, evaluations and explanations associated with products, services or experiences. Structured ratings can be paired with unstructured text to study satisfaction, drivers of complaints and changes in customer expectations. Analysts must account for review incentives, moderation, fake reviews, survivorship bias and the fact that reviewers are not a random sample of customers.

A one-star review and a five-star review do not necessarily represent opposite attitudes about every attribute. Aspect-level analysis may reveal that a customer praises delivery but criticises product quality. Consequently, collection should preserve product, location, rating, date, response and verification fields wherever permitted.

1.3 Conversations and interactional data

Chat logs, call-centre transcripts, community forums and instant messages contain conversational structure. Meaning may depend on turn-taking, previous messages, speaker roles, response time and escalation history. Treating every message as an independent document can produce misleading conclusions because messages from one conversation are correlated and context may be distributed across turns.

Conversation data also create heightened privacy risks. Participants may disclose health, financial or identity information without expecting secondary analysis. Access controls, redaction and purpose limitation are therefore essential.

1.4 Social platforms

Social platforms provide posts, comments, replies, reposts, reactions, hashtags, profiles and network relations. They can support brand monitoring, issue detection, campaign evaluation and crisis intelligence. However, platform populations are not equivalent to the general population. Participation, visibility and algorithmic ranking vary by country, age, income, language, political context and platform design.

A social-media dataset is usually an observation of a platform’s information environment, not a direct measurement of public opinion. Deleted posts, private accounts, suspended accounts, API filtering and ranking algorithms affect what becomes observable.

1.5 APIs and web sources

An API is a controlled interface through which a provider makes data or functions available. API-based collection is usually preferable to unapproved scraping because it provides documented fields, authentication, rate limits and usage rules. Nevertheless, APIs can impose historical limits, pagination rules, quota restrictions, field-level permissions and changing schemas.

Web pages may be collected when an API is unavailable and collection is lawful and permitted. Analysts should respect terms of service, robots guidance where relevant, copyright, authentication boundaries and reasonable request rates. A web page is also a presentation layer: content may be generated dynamically, localised, personalised or altered after collection. The collector should preserve retrieval time and, where lawful, the source URL and relevant page context.

1.6 Enterprise systems

Customer relationship management systems, ticketing platforms, enterprise resource planning systems, data warehouses and knowledge bases often contain high-value text linked to operational outcomes. Enterprise data may be more actionable than public data but can reflect organisational processes and recording practices. For example, a complaint field may be completed only after escalation, creating apparent associations between complaint language and service failure.

Integration requires attention to identifiers, access roles, retention rules, master-data definitions and changes to source systems. Text analytics should not bypass established data stewardship arrangements.

2. From Business Question to Collection Design

Collection should begin with a decision, not with an attractive dataset. A useful design specifies:

The target population is the complete set about which conclusions are intended. The sampling frame is the operational list or mechanism from which observations can be selected. The sample is the selected set, whereas the observed dataset is what is actually obtained after access restrictions, non-response, deletion and technical failures.

These distinctions are crucial. A project may intend to analyse all customers in a market, use a platform’s followers as its frame, draw a sample of posts and finally obtain only posts available through an API. Each stage can introduce coverage error.

2.1 Units and dependence

The unit of analysis must be explicit. A post, author, thread, product, customer and day are different units. If ten posts are written by one highly active account, treating them as ten independent customers exaggerates the effective breadth of evidence. When observations are clustered, analysis and reporting should recognise the clustering, for example by aggregating to account, conversation or customer level where appropriate.

2.2 Sampling approaches

Probability sampling gives units a known, non-zero probability of selection. Simple random, systematic, stratified and cluster sampling are examples. It supports stronger generalisation when the frame is adequate.

Non-probability sampling includes convenience, purposive, quota, snowball and platform-availability sampling. It is often necessary for social and conversational data, but its limitations must be disclosed. Large volume does not eliminate selection bias.

Stratification can improve coverage. If reviews are collected across regions, a design may deliberately select observations by region and then apply weights when estimating population-level quantities. If stratum h has population size Nₕ and sample size nₕ, a basic design weight is:

Here, wₕ is the weight for an observation in stratum h. This formula assumes selection probabilities are equal within each stratum and that Nₕ and nₕ are known. Non-response and calibration adjustments may still be required. Weights should not be applied mechanically to machine-learning training data without considering their effect on the learning objective.

For estimating a proportion p under a simple random sample, an approximate margin of error is:

where E is the margin of error, z is the chosen standard-normal critical value, p is the population proportion or a planning value, and n is sample size. This relationship assumes a probability sample, independent observations and an appropriate approximation. For a 95% planning value with p = 0.5 and n = 400, z ≈ 1.96 and E ≈ 1.96 × √(0.25/400) = 0.049, or approximately 4.9 percentage points, before finite-population, design or non-response adjustments. In social-media data, these assumptions often fail; the calculation should not be presented as a guarantee of accuracy.

2.3 Temporal sampling and event capture

Text streams are dynamic. Sampling only during business hours may miss night-time complaints; collecting only during a crisis may overstate negative sentiment; collecting at regular intervals may miss short-lived events. A robust design defines time zones, collection frequency, backfill procedures and event windows. Every timestamp should have a documented time standard, preferably Coordinated Universal Time (UTC), alongside the original local time when available.

3. Collection Mechanisms and Platform Constraints

3.1 API collection

A production API collector should manage authentication, pagination, rate limits, retries, timeouts, duplicate responses, schema changes and logging. Cursor-based pagination is generally safer than assuming that page numbers remain stable while a stream changes. Each request should record endpoint, parameters, request time, response status, provider version and error information.

Rate-limit handling must be respectful and explicit. A collector should implement backoff rather than repeatedly retrying a denied request. If a provider returns a continuation token, that token and the retrieval state should be stored so collection can resume without silently skipping records.

API data may be incomplete because the provider excludes deleted, private, restricted or older content. “No records returned” may mean no matching records, unavailable history, an invalid query or a quota problem. These outcomes must be distinguished in logs.

3.2 Web collection

A compliant web-collection workflow identifies permitted pages, uses a reasonable request rate, avoids bypassing access controls, records retrieval dates and validates that the parser still matches the page structure. HTML parsers can fail silently when a site redesign changes class names or embedded data structures. Monitoring should therefore test expected fields, record extraction success rates and sample raw pages for audit where retention is lawful.

3.3 Documents and batch ingestion

Batch ingestion should preserve the original file, a cryptographic checksum, extraction software version and transformation history. A checksum such as SHA-256 can help detect whether a file has changed; it does not prove that the file is authentic or that its contents are correct. Text extraction should preserve page boundaries, headings, tables and character encoding when these matter to interpretation.

3.4 Enterprise integration

Enterprise integration commonly uses scheduled exports, database views, message queues or managed pipelines. A stable surrogate identifier can link text to transactions while limiting exposure of direct identifiers. Analysts should obtain data dictionaries and consult system owners about field meaning, historical changes and known operational biases.

4. Metadata, Provenance and Data Quality

4.1 Metadata as analytical evidence

Metadata describes what an observation is, where it came from, when it was created and how it was transformed. A useful metadata model includes:

Metadata category	Examples	Analytical value
Identity	record ID, conversation ID, author hash	Deduplication and linkage
Time	creation, update, retrieval time, time zone	Trend and event analysis
Source	platform, endpoint, URL, system table	Coverage and provenance
Content	language, encoding, media indicator	Processing decisions
Context	product, location, rating, reply relation	Interpretation and segmentation
Process	collector version, parser, transformation	Reproducibility
Governance	consent basis, sensitivity, retention class	Lawful and secure use

Metadata should be stored alongside the data and in a catalogue that users can search. A field such as “date” is inadequate unless its meaning—creation date, publication date, update date or retrieval date—is specified.

4.2 Provenance and lineage

Provenance answers: What was collected? From where? When? By which process? What changed? Who accessed it? A lineage record can connect raw data to cleaned data, feature tables, models and reports. This allows an analyst to reproduce a result or determine whether a conclusion was affected by a source revision.

Raw data should not be overwritten when corrections are made. Instead, a new version and transformation record should be created.

4.3 Data-quality dimensions

A missing value is not always a defect. A user may legitimately omit a location, or a platform may suppress a field for privacy. Missingness should be classified as unknown, not applicable, not collected, restricted or technically failed where possible.

4.4 Deduplication and identity

Exact duplicates can be detected with stable record IDs or hashes. Near-duplicates require comparison of normalised text, timestamps, authorship and source identifiers. Deduplication can be dangerous: syndicated news, reposts and quoted text may be analytically meaningful. The appropriate action depends on the unit of analysis and research question.

Entity resolution links records believed to refer to the same customer, product or organisation. Deterministic matching uses reliable identifiers; probabilistic matching uses evidence such as names, addresses and dates. False matches can create serious privacy and analytical harm, so uncertain links should be retained with confidence scores or excluded from high-stakes decisions.

4.5 Language, encoding and preprocessing

UTF-8 should normally be used to preserve multilingual characters. Language identification can fail on short texts, code-switching, transliteration and named entities. Do not translate automatically without recording the translation system, model version and source language. Emojis, punctuation, spelling variation, hashtags and mentions may carry meaning; aggressive cleaning can remove precisely the signals required for social analysis.

5. Storage and Reproducible Management

Text data range from small relational tables to large document and event stores. A relational database is useful for structured metadata, constraints and joins. Object storage is suitable for raw files, JSON responses and large artefacts. Search indexes support retrieval across text. A lakehouse or warehouse can combine scalable storage with governed analytical tables.

A sound design separates personally identifiable information (PII) from analytical content where feasible. Direct identifiers should be encrypted, access should follow least privilege, and sensitive fields should be masked or tokenised. Hashing an identifier is not automatically anonymous: a predictable or small identifier space may be reversible or linkable.

Backups should be tested, not merely configured. Retention schedules should specify how long raw data, derived data, logs and model outputs are kept. Deletion requests and platform takedown requirements may require deletion or suppression in downstream derivatives, although feasibility depends on the legal basis and technical architecture.

Reproducibility requires more than saving code. A reproducible project normally includes:

Full public release of raw social data may be prohibited by privacy, copyright or platform rules. Reproducibility may instead use synthetic data, aggregated outputs, shareable code, documented queries, secure enclaves or controlled-access repositories.

6. Governance, Ethics and Regulation

Text collection can affect people who did not intend to participate in research or commercial profiling. Public availability does not automatically remove ethical obligations. Key principles include purpose limitation, data minimisation, proportionality, transparency, security, accountability and respect for deletion or access rights where applicable.

The legal framework varies by jurisdiction and context. Relevant issues may include data-protection law, consumer-protection law, employment law, intellectual-property rights, sectoral confidentiality and contractual platform restrictions. Analysts should involve legal, privacy and information-security specialists for high-risk uses rather than assuming that a public URL grants unrestricted analytical rights.

Sensitive inference deserves special caution. A model may infer health status, political affiliation, ethnicity or financial vulnerability from seemingly ordinary language. Such inferences can produce discrimination even when the sensitive attribute was never explicitly collected. High-impact decisions should have human oversight, documented purposes, appeal routes and bias monitoring.

7. Applied Case: Customer-Service Conversation Intelligence

Background

A national telecommunications company receives support requests through chat, email and call-centre transcripts. Management wants to identify recurring causes of repeat contacts and reduce resolution time. The company initially proposes collecting all messages and training a sentiment classifier.

Problem

A preliminary extract contains duplicated messages, inconsistent customer IDs, missing timestamps, automated system messages and personally identifiable information. Call transcripts are available only for selected queues. Chat data are retained for different periods across regions. A simple count of negative messages would overrepresent customers who send many short messages and would confuse frustration with the technical cause of the problem.

Analysis and approach

The analytics team first defines the unit of analysis as a customer issue episode, not an individual message. Messages are grouped using conversation IDs, customer tokenisation and a specified inactivity threshold, with exceptions for explicit case closure. The team creates a data dictionary distinguishing message time, case-open time, resolution time and retrieval time.

Raw exports are stored in a restricted object store. A separate identity vault holds the mapping between customer identifiers and random analytical IDs. Automated redaction removes phone numbers, email addresses and payment details, while a quality sample is reviewed by authorised staff. Source-system coverage is documented by channel, region and month.

Sampling is stratified by channel, region and outcome so that rare escalations are not lost. The team retains sampling indicators rather than treating the resulting sample as a representative estimate of all customers. Duplicate detection uses source message IDs first and text-time comparison second. Messages generated by bots are labelled rather than silently deleted.

Analysts combine topic discovery with a supervised issue taxonomy developed by service experts. Sentiment is used as contextual evidence, not as a direct proxy for service failure. Model outputs are joined to operational measures such as repeat contact, transfer count and resolution time. Human reviewers examine errors, especially multilingual, short and code-switched messages.

Outcome

The organisation discovers that repeat contacts are associated less with general negativity than with unresolved authentication failures and unclear outage messages. It revises customer communications and routing rules. The data-management work also reveals that one region’s apparent improvement resulted from a change in logging, not better service.

Learning points

The case demonstrates that collection design determines what can be learned. Episode-level units prevent message volume from being mistaken for customer volume; metadata expose coverage changes; redaction reduces risk; and operational linkage makes text findings actionable. It also shows why sentiment alone is insufficient for managerial diagnosis.

8. Contemporary Developments and Best Practices

Large language models and embedding services can enrich retrieval, classification and summarisation, but they create additional governance requirements. Analysts should record model name, version, prompt or configuration, date, input policy and output validation. Sensitive text should not be sent to an external service without an approved data-processing arrangement. Generated summaries must be checked for omission, fabrication and disclosure of private information.

Multimodal platforms increasingly combine text, images, audio and video. Collection plans should identify whether text is primary evidence or merely a caption for another modality. Cross-platform identity graphs can improve analysis but substantially increase re-identification risk.

Chapter Summary

Text and social media analytics depends on disciplined data management. Documents, reviews, conversations, social platforms, APIs, web sources and enterprise systems differ in structure, access, context and bias. A collection design should define the target population, sampling frame, unit of analysis, time period, inclusion rules and governance requirements. Platform APIs and web sources impose limits that must be recorded rather than hidden.

Metadata and provenance make observations interpretable and analyses auditable. Data quality includes completeness, validity, accuracy, consistency, uniqueness, timeliness, representativeness and integrity. A layered architecture—raw, standardised, curated and feature/output—protects source evidence while supporting analysis. Secure storage, separation of identifiers, retention controls and versioned workflows enable responsible use and reproducibility.

Finally, public availability does not eliminate ethical responsibility. Purpose limitation, minimisation, transparency, privacy, security and human oversight are essential, particularly when text is used to infer sensitive attributes or influence high-impact decisions. The central managerial lesson is that reliable insight is produced by the entire evidence pipeline, not by the model alone.

Key Terms

Target population: The complete set about which an analysis intends to make claims.

Sampling frame: The operational source or mechanism from which observations can be selected.

Unit of analysis: The entity or event represented by one analytical observation.

Probability sampling: Sampling in which each eligible unit has a known non-zero selection probability.

Coverage error: Error caused when the sampling frame does not adequately represent the target population.

Metadata: Data describing an observation’s identity, context, source, time or processing history.

Data lineage: The trace connecting source data to derived tables, models and outputs.

API: A controlled software interface for requesting data or services from a provider.

Pagination: The mechanism used to retrieve a large result set across multiple responses.

OCR: Optical character recognition that converts images of text into machine-readable characters.

Entity resolution: Linking records that are believed to describe the same real-world entity.

Representativeness: The extent to which observed data support intended population-level claims.

PII: Personally identifiable information that can identify or help identify an individual.

Data minimisation: Collecting and retaining no more personal data than necessary.

Data drift: Change over time in data distributions, sources, language or relationships.

Reproducibility: The ability to obtain consistent analytical results using documented data and procedures.

Purpose limitation: Restricting collection and use to specified, legitimate purposes.

Review Questions

Short Answer Questions

Long Answer Questions

Analytical / Application Questions

Multiple Choice Questions with Answers

Question 1

Correct Answer: B. The list or mechanism from which observations are selected

Question 2

What is the main analytical risk of treating every message in a conversation as an independent customer observation?

Correct Answer: C. It can overrepresent highly active customers and ignore dependence

Question 3

In a stratum with population size Nₕ = 2,000 and sample size nₕ = 100, what is the basic design weight?

Question 4

Question 5

Correct Answer: B. Record query parameters, retrieval time, endpoint and API version

Question 6

Why is a platform’s follower population not necessarily a representative sample of all customers?

B. Platform participation and visibility can differ systematically across groups

Correct Answer: B. Platform participation and visibility can differ systematically across groups

Question 7

Which storage arrangement best protects direct identifiers while permitting analysis?

Correct Answer: C. Separate the identity vault from analytical data and restrict access

Question 8

Question 9

A 95% margin-of-error calculation based on a simple random sample may be misleading for social-media data primarily because

B. the assumptions of probability sampling and independent observations may not hold

Correct Answer: B. the assumptions of probability sampling and independent observations may not hold

Question 10

Which principle recommends collecting no more personal data than necessary for a stated purpose?

Chapter 3 Data Sources, Collection and Management

Data Sources, Collection and Management

Introduction