Data Wrangling and Visualization Foundations

Introduction

Data science rarely begins with a clean spreadsheet and a clearly defined question. In professional settings, data may be distributed across databases, application logs, spreadsheets, surveys, sensors, documents and external services. It may contain missing values, inconsistent labels, duplicate records, incompatible formats and measurements collected for purposes different from the current analysis. Before data can support a trustworthy decision, it must be understood, prepared and communicated.

Data wrangling and visualization are therefore foundational capabilities in data science. Data wrangling is the systematic process of discovering, acquiring, structuring, cleaning, transforming, validating and documenting data. Data visualization is the use of graphical or visual representations to explore data, communicate evidence and support reasoning. These activities are closely connected: wrangling makes patterns more reliable, while visualization helps reveal errors, relationships and useful questions.

This chapter introduces the terminology, lifecycle, principles and professional responsibilities that underpin later work with tools such as spreadsheets, SQL, Python, R and business-intelligence platforms.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish data, information, metadata, datasets and variables.
2. Explain the main stages of a data lifecycle.
3. Apply the principles of tidy data to simple datasets.
4. Describe the purposes and limitations of visualization.
5. Translate stakeholder needs into data and visualization requirements.
6. Explain how wrangling contributes to analytics and digital transformation.
7. Identify quality, ethical, governance and reproducibility risks.

1. Essential Terminology

A datum is a recorded observation, while data is the plural term commonly used for a collection of observations. Data can be quantitative, such as revenue or temperature, or qualitative, such as customer comments and product categories. Data can also be structured, semi-structured or unstructured.

A dataset is an organized collection of related observations. In a relational table, a row usually represents an observation or entity instance, and a column represents an attribute or variable. A variable is a characteristic that can take different values. For example, in a student dataset, student_id, programme, age and assessment_score may be variables.

A record is a collection of fields describing one entity or event. An identifier is a variable intended to distinguish records. A primary key uniquely identifies a row within a database table. A foreign key links a record to a related table. A measurement is a value produced by an observation process; it should be interpreted together with its unit, time and method.

Metadata is data about data. It may describe definitions, units, permitted values, collection dates, ownership, provenance, privacy classification and transformation history. Without metadata, a value such as “120” may be impossible to interpret: it could mean dollars, kilograms, minutes or cases.

Data quality refers to the fitness of data for a stated purpose. Common dimensions include accuracy, completeness, consistency, validity, uniqueness, timeliness and relevance. Quality is not an absolute property. A dataset can be adequate for exploratory analysis but unsuitable for regulatory reporting.

Wrangling, data preparation and data preprocessing are related terms. Wrangling emphasizes practical discovery and transformation; preprocessing often refers to preparation before modelling, including encoding and scaling. Typical wrangling operations include filtering, joining, reshaping, parsing, deduplicating and validating.

A visualization encodes data using position, length, colour, shape, size or other visual attributes. A dashboard combines several coordinated visualizations, often with filters and summary indicators, to support monitoring or decision-making. An insight is a meaningful interpretation supported by evidence, not merely an interesting visual pattern.

2. The Data Lifecycle

The data lifecycle describes how data moves from an initial need through collection, use, maintenance and eventual retention or disposal. Different organizations use different labels, but a practical lifecycle contains the following stages.

2.1 Define the purpose

The process begins with a question, decision or operational need. A vague request such as “show sales performance” must be converted into measurable questions: Which products? Which period? Compared with what target? For which audience? What action might follow?

A clear purpose prevents unnecessary collection and reduces the risk of producing technically impressive but operationally irrelevant outputs.

2.2 Acquire or collect data

Data may be obtained from transactional systems, APIs, surveys, sensors, files, experiments or public sources. At this stage, analysts should record source ownership, collection method, time period, population covered, sampling method and legal basis for use where personal data is involved.

2.3 Store and manage data

Data must be stored with appropriate access controls, backups, retention rules and documentation. Storage design affects later work. For example, timestamps should be stored in an unambiguous standard and transformed for display only when required by the audience.

2.4 Profile and assess

Data profiling is the examination of structure and content before transformation. Useful checks include row counts, data types, missing-value rates, distinct-value counts, minimum and maximum values, duplicate identifiers and unexpected categories. Profiling is not a substitute for domain knowledge: a value can be technically valid but conceptually wrong.

2.5 Wrangle and transform

Analysts standardize formats, resolve inconsistencies, combine sources and create analysis-ready variables. Every substantive transformation should be documented. A useful principle is to preserve the original data and create a reproducible transformed version rather than overwriting the source.

2.6 Analyze and visualize

Analysis may involve descriptive statistics, statistical modelling, machine learning or operational reporting. Visualization supports both exploration and communication. At this stage, analysts should distinguish observed facts, assumptions and interpretations.

2.7 Share, act and evaluate

Results may be delivered through a report, dashboard, data product or operational system. The lifecycle continues when users act on the results and provide feedback. Monitoring should examine whether definitions remain appropriate, data quality changes and decisions produce intended outcomes.

2.8 Archive or dispose

Data should be retained only as long as justified by operational, legal, research or historical requirements. Secure disposal is part of responsible data management. Archiving should preserve sufficient metadata so that future users can understand the data.

3. Tidy Data Principles

Hadley Wickham’s tidy-data framework provides a practical structure for analysis. Its central principles are:

1. Each variable forms a column.
2. Each observation forms a row.
3. Each type of observational unit forms a table.

These principles are not the only valid database design rules. A normalized operational database may deliberately distribute information across related tables. Tidy data is primarily an analysis-oriented arrangement that makes transformations and visualizations more consistent.

Consider this untidy table:

Here, month values are embedded in column names. A tidy version is:

The second structure makes it easier to group by month, compare regions and create a time-series visualization. The transformation from wide to long form is often called pivoting or melting, depending on the software.

Tidy data does not mean that every table must contain one row per person. The observation could be a person-month, product-day or store-transaction. The key question is: what does one row represent? This should be explicitly documented.

Keys and granularity

Granularity is the level of detail represented by each row. A transaction-level table and a monthly-summary table may both be valid, but they answer different questions. Joining tables with different granularities can accidentally duplicate values. For example, joining one customer row to many transaction rows causes customer-level attributes to repeat. Analysts must identify keys and verify the intended relationship before joining.

Missing values

A missing value may mean “not recorded,” “not applicable,” “not yet available” or “withheld.” These meanings should not automatically be replaced by zero. Imputation or deletion may be appropriate in some analyses, but the decision must be justified and documented. A missing category and a genuine zero represent different information.

Consistent representations

Dates should use a consistent, interpretable format; categorical values should use controlled labels; numeric fields should not mix symbols and text; and units should be explicit. For example, “UK,” “U.K.” and “United Kingdom” should be harmonized only after confirming that they refer to the same concept.

4. What Data Wrangling Involves

A professional wrangling workflow is iterative rather than strictly linear.

Inspecting involves examining schemas, sample rows, metadata and distributions. Cleaning corrects or flags errors such as invalid dates, duplicate records and impossible measurements. Transforming changes representation or creates useful variables, such as extracting year from a timestamp. Combining uses joins, unions or concatenation to bring sources together. Validating checks whether the result satisfies structural and domain rules. Documenting records sources, decisions, assumptions and code versions.

Validation rules should reflect the domain. A rule may require that an assessment score lies between 0 and 100, that an order date is not later than its delivery date, or that every transaction references an existing product. A failed rule should generate an investigation or an explicit exception, not be silently ignored.

A useful quality-control pattern is to compare before and after counts. If a join unexpectedly increases the number of rows, the analyst should investigate whether the relationship is one-to-one, one-to-many or many-to-many. Reproducible scripts and version-controlled files reduce the risk of undocumented manual edits.

5. Purposes of Data Visualization

Visualization serves at least four distinct purposes.

Exploration

Exploratory visuals help analysts discover distributions, outliers, trends, clusters and possible relationships. Histograms, box plots, scatter plots and line charts are often useful. Exploration encourages questioning rather than premature certainty.

Explanation

Explanatory visuals communicate a selected finding to a defined audience. They should provide context, clear labels, appropriate scales and a concise message. A chart designed for an executive decision differs from one designed for a technical investigation.

Monitoring

Operational dashboards track key measures over time. They require reliable refresh schedules, threshold definitions, ownership and escalation procedures. A dashboard without an agreed action is often only a display, not a management instrument.

Analysis and communication of uncertainty

Visuals can show confidence intervals, ranges, distributions and scenarios. They should not imply greater precision than the data supports. A single average may conceal substantial variation; showing the distribution may lead to a better decision.

Choosing visual encodings

Position on a common scale is usually easier to compare accurately than area, volume or decorative effects. Bar charts are appropriate for comparing categories when the baseline and category definitions are clear. Line charts are useful for ordered time data. Scatter plots show relationships between quantitative variables. Maps are appropriate when geographic location is central, not merely because locations are available.

Colour should have a declared meaning. Sequential colour scales suit ordered magnitudes, while categorical palettes suit distinct groups. Excessive colour, three-dimensional effects and truncated axes can distort interpretation. Accessibility requires adequate contrast and alternatives for people with colour-vision deficiencies.

6. Stakeholder Requirements

A visualization is successful only if it supports a legitimate user need. Requirements gathering should identify:

• the decision or task;
• the intended audience and their data literacy;
• required measures and definitions;
• time period, refresh frequency and level of detail;
• filters, comparisons and alerts;
• acceptable latency and accuracy;
• privacy, security and accessibility constraints; and
• the action expected after viewing the result.

Stakeholders often use the same word differently. “Customer,” “active user,” “profit” and “on-time delivery” need operational definitions. A requirements document or data dictionary should specify numerator, denominator, unit, population, exclusions and time reference for important metrics.

A stakeholder may ask for a pie chart because it is familiar, although a bar chart may permit more accurate comparisons. The analyst’s responsibility is not simply to obey a requested format but to clarify the purpose and recommend an effective representation.

7. Worked Application: Retail Delivery Dashboard

The following is an illustrative case.

Background

A regional retailer receives orders through stores and an online platform. Managers report that delivery performance is declining, but different departments calculate “on-time delivery” differently.

Problem

The source data contains order_id, store_id, promised_date, delivered_date, status and customer_region. Dates are stored in several formats. Some cancelled orders have no delivery date. A store appears more than once in a reference table because of historical naming changes.

Analysis

The analyst first defines the grain: one row should represent one order. The business definition is that an order is on time when it is delivered on or before the promised date. Cancelled orders are excluded from the delivery-rate denominator, while undelivered but active orders remain visible in an operational backlog measure.

The analyst profiles the data, standardizes dates, maps historical store names to a controlled store identifier and checks that order_id is unique. A join to the store reference table initially increases row count, revealing duplicate reference records. The reference table is corrected using effective dates before the join is repeated.

For each eligible order, an indicator variable is created:

Here, i identifies an eligible order, and both dates must use the same calendar convention and time zone. The metric is then calculated as:

Cancelled orders are excluded according to the stated definition. The analyst also reports the number of active undelivered orders, because a high on-time rate based only on completed orders could hide a growing backlog.

Visualization solution

The dashboard contains a headline on-time rate with its denominator, a line chart by week, a bar chart comparing regions, and a table of active overdue orders. Filters allow managers to select region and date range. A definition panel explains inclusion rules and refresh time.

Outcome

Managers discover that the decline is concentrated in two regions and began after a change in the dispatch process. The dashboard does not prove causation, but it directs operational investigation. The inclusion rules prevent cancelled orders and inconsistent store mappings from creating misleading results.

Learning points

The case illustrates that visualization quality depends on definitions, grain, keys, validation and stakeholder action. Attractive charts cannot compensate for ambiguous metrics or incorrect joins.

8. Wrangling, Analytics and Digital Transformation

Analytics converts data into evidence for description, diagnosis, prediction or decision support. Wrangling is the enabling layer between raw operational records and these analytical uses. Poorly prepared data can produce biased estimates, unstable models and incorrect business decisions. In practice, substantial analytical effort is often spent understanding and preparing data rather than fitting a final model.

Digital transformation involves changing processes, services and organizational practices through digital technologies. It is not achieved merely by purchasing a dashboard or migrating files to the cloud. Transformation requires trustworthy data flows, shared definitions, interoperable systems, appropriate governance and a culture of evidence-based improvement.

Wrangling supports transformation by:

• integrating previously separated systems;
• creating consistent organizational definitions;
• exposing process bottlenecks;
• enabling automation and near-real-time monitoring;
• preparing data for predictive and AI systems; and
• creating auditable data products.

Automation should not remove accountability. A pipeline that silently changes definitions can distribute errors at scale. Automated tests, monitoring, lineage and human review are essential, particularly when data informs high-impact decisions.

9. Ethics, Governance and Limitations

Data work involves responsibilities beyond technical correctness. Personal data should be collected and used for legitimate, transparent purposes, with appropriate security and retention controls. Access should follow the principle of least privilege. Sensitive attributes require careful treatment, and de-identification does not guarantee that re-identification is impossible.

Bias can enter through sampling, missing data, measurement choices, historical practices and labels. Removing a protected attribute does not necessarily remove bias because other variables may act as proxies. Analysts should examine coverage, subgroup performance and the consequences of decisions.

Visualization can mislead through selective time periods, inappropriate scales, omitted denominators, excessive aggregation or confusing colour. Ethical communication requires showing relevant context and acknowledging uncertainty. A chart should not be designed to conceal an inconvenient result.

Important governance practices include data ownership, quality rules, lineage, access management, retention schedules, change control and incident procedures. Regulations vary by jurisdiction and sector; analysts must follow applicable privacy, records-management and professional requirements rather than assuming that publicly accessible data is unrestricted for every use.

10. Contemporary Practice and Best Practices

Modern organizations increasingly use cloud warehouses, APIs, automated pipelines, semantic layers and AI-assisted coding. These technologies can improve speed, but generated transformations and summaries must be tested against source data and domain expectations. AI systems may reproduce bias, expose confidential information or produce plausible but unsupported interpretations.

Recommended practices are:

1. Start with the decision and define the unit of analysis.
2. Preserve raw data and create documented transformed layers.
3. Use metadata, data dictionaries and controlled vocabularies.
4. Profile data before making assumptions.
5. Validate keys, ranges, dates, duplicates and row counts.
6. Make missing-value decisions explicit.
7. Separate exploration from published reporting.
8. Choose visual encodings for the task and audience.
9. Display denominators, units, dates and definitions.
10. Test accessibility and obtain stakeholder feedback.
11. Version scripts, transformations and metric definitions.
12. Monitor pipelines after deployment.

Chapter Summary

Data wrangling prepares raw, heterogeneous data for reliable analysis through inspection, cleaning, transformation, integration, validation and documentation. The data lifecycle extends from defining a purpose and acquiring data to analysis, action, archiving and disposal. Tidy data provides a useful analysis structure in which each variable is a column, each observation is a row and each observational unit has an appropriate table.

Visualization supports exploration, explanation, monitoring and communication of uncertainty. Its effectiveness depends on accurate data, appropriate encodings, clear definitions and knowledge of the stakeholder’s decision. Granularity, keys, missing-value meanings and join relationships are central to avoiding analytical errors. Wrangling enables analytics and digital transformation, but professional practice also requires privacy protection, governance, accessibility, bias awareness, reproducibility and accountability.

Key Terms

Data: Recorded observations or facts used for description, analysis or decision-making.

Dataset: An organized collection of related observations.

Variable: A characteristic that can take different values.

Record: A group of fields describing an entity or event.

Metadata: Data that describes the meaning, structure, origin or management of data.

Data quality: The fitness of data for a specified purpose.

Data wrangling: The process of discovering, cleaning, transforming, combining and validating data.

Data profiling: Examination of data structure, content and quality before use.

Tidy data: An analysis-oriented structure with variables in columns and observations in rows.

Granularity: The level of detail represented by one record.

Primary key: A field or combination of fields that uniquely identifies a record.

Foreign key: A field that references a key in another table.

Missing value: An absent, unavailable or inapplicable value whose meaning must be determined.

Visualization: A graphical representation that encodes data for exploration or communication.

Dashboard: A coordinated collection of visual displays used to monitor or support decisions.

Data lifecycle: The stages through which data is collected, managed, used, shared and retired.

Data lineage: A record of where data originated and how it was transformed.

Digital transformation: Organizational change enabled by digital technologies, data and redesigned processes.

Data governance: Policies, roles and controls that manage data quality, access, security and use.

Review Questions

Short Answer Questions

1. What is the difference between data and metadata?
2. State the three central principles of tidy data.
3. Why is granularity important when joining datasets?
4. Identify two purposes of data visualization.
5. What is the role of data profiling in a wrangling workflow?

Long Answer Questions

1. Explain the major stages of the data lifecycle and show how decisions at one stage affect later stages.
2. Compare exploratory and explanatory visualization, including suitable examples of each.
3. Explain how missing values should be interpreted and managed in a professional data project.
4. Discuss how stakeholder requirements influence metric definitions, chart selection and dashboard design.
5. Explain how data wrangling contributes to digital transformation while also creating governance risks.

Analytical / Application Questions

1. A table contains one row per region and separate columns for sales in each month. Explain how you would restructure it as tidy data and identify one analytical benefit.
2. After joining an orders table to a store table, the number of rows doubles. What possible relationship or data-quality problem could explain this result, and what checks would you perform?
3. Design a small dashboard for monitoring delivery performance. Specify the grain, key metric, denominator, at least two visualizations, one filter and two governance or accessibility requirements.

Multiple Choice Questions with Answers

Question 1

Which statement best describes tidy data?

A. Each row contains a different type of variable.

B. Each variable forms a column and each observation forms a row.

C. All data must be stored in one database table.

D. Missing values are replaced with zero.

Correct Answer: B. Each variable forms a column and each observation forms a row.

Question 2

What is metadata?

A. A duplicate copy of a dataset

B. A chart created from a dataset

C. Data that describes the meaning, origin or structure of data

D. Only numerical data collected by sensors

Correct Answer: C. Data that describes the meaning, origin or structure of data

Question 3

Which activity is part of data profiling?

A. Examining data types, missingness and distinct values

B. Designing a logo for a dashboard

C. Deleting all unusual observations

D. Publishing results without validation

Correct Answer: A. Examining data types, missingness and distinct values

Question 4

Why must granularity be considered before joining tables?

A. It determines the colour palette of a chart.

B. Different granularities can cause duplicated or misleading values.

C. It guarantees that all values are normally distributed.

D. It removes the need for primary keys.

Correct Answer: B. Different granularities can cause duplicated or misleading values.

Question 5

Which visualization is generally most suitable for showing a trend over ordered time periods?

A. Line chart

B. Pie chart

C. Word cloud

D. Treemap

Correct Answer: A. Line chart

Question 6

In the illustrative delivery case, which orders were excluded from the on-time-rate denominator?

A. All orders delivered late

B. All orders delivered early

C. Cancelled orders

D. Orders from regional stores

Correct Answer: C. Cancelled orders

Question 7

What is the primary purpose of a stakeholder requirements discussion?

A. To select the most decorative chart type

B. To identify the decision, audience, measures and constraints

C. To guarantee that no data will be missing

D. To replace data validation with user opinion

Correct Answer: B. To identify the decision, audience, measures and constraints

Question 8

Which practice most directly supports reproducibility?

A. Overwriting the original source file

B. Using undocumented manual edits

C. Version-controlling transformation code and definitions

D. Removing metadata before publication

Correct Answer: C. Version-controlling transformation code and definitions

Question 9

Why can removing a protected attribute fail to eliminate bias?

A. Charts cannot display protected attributes.

B. Other variables may act as proxies or reflect historical bias.

C. Protected attributes are always missing.

D. Removing variables always increases accuracy.

Correct Answer: B. Other variables may act as proxies or reflect historical bias.

Question 10

An on-time rate is calculated as 90 on-time eligible deliveries divided by 100 eligible delivered orders. What is the rate?

A. 0.9%

B. 9%

C. 90%

D. 190%

Correct Answer: C. 90%