Foundations of Data Analysis and Visualisation

Introduction

Modern organisations generate data continuously through transactions, websites, mobile applications, sensors, enterprise systems, social media and operational records. Data alone, however, does not automatically create knowledge or improve decisions. Data analysis is the disciplined process of preparing, examining, interpreting and communicating data so that evidence can support action. Visualisation is an important part of this process because well-designed charts and dashboards make patterns, comparisons, trends and exceptions easier to perceive.

For students of artificial intelligence and machine learning, these foundations are essential. Machine-learning models depend on appropriately defined variables, reliable datasets, suitable measurement scales and clearly stated objectives. Poorly framed questions or misleading visualisations can produce technically sophisticated but practically useless results.

This chapter introduces the main concepts of data analysis, analytics and visualisation. It distinguishes four common forms of analytics, explains data types and measurement scales, describes variables and datasets, and shows how analytical questions can be connected to measurable success criteria.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the relationships among data, information, analysis, analytics and visualisation.
2. Distinguish descriptive, diagnostic, predictive and prescriptive analytics.
3. Classify data by structure, nature and measurement scale.
4. Identify variables, observations, records, populations, samples and datasets.
5. Select suitable visualisations for common analytical purposes.
6. Frame analytical questions that are specific, answerable and decision-relevant.
7. Define appropriate success criteria for an analytical project.
8. Recognise practical, ethical and governance issues in data work.

From Data to Decision

Data, information and knowledge

Data consists of recorded observations, measurements, labels or symbols. Examples include a product code, a temperature reading, a customer complaint, a date, or the number of units sold. A single value often has little meaning without context. Information is data placed into a meaningful structure or context. For example, “1,250 units were sold in March” is more informative than the isolated value 1,250.

Analysis is the systematic examination of data to describe conditions, identify relationships, test explanations or support decisions. Analytics is the broader organisational practice of using data, statistical reasoning, computational methods and domain knowledge to produce insight and guide action. Visualisation is the graphical representation of data or analytical results using visual encodings such as position, length, colour, shape and area.

These concepts are related but not interchangeable:

A useful analytical workflow is iterative rather than strictly linear. An analyst defines a problem, obtains data, assesses quality, explores patterns, applies appropriate methods, communicates findings and evaluates whether the result supports a decision. Discoveries during exploration may require the question, data preparation or method to be revised.

Analysis and visualisation as reasoning activities

A chart is not merely decoration. It is an argument about what is important in the data. Choosing a time axis highlights change; grouping categories highlights comparison; a map highlights spatial variation. Every visualisation therefore involves design decisions that can clarify or distort meaning.

Good analysis combines three forms of understanding:

• Statistical understanding: variation, distributions, uncertainty and association.
• Subject-matter understanding: how the organisation or process actually works.
• Communication understanding: how an audience will interpret evidence and act on it.

An analyst should avoid treating a numerical association as proof of causation. If sales and advertising expenditure increase together, the relationship may reflect advertising effectiveness, seasonal demand, a third factor, or a combination of causes. Analysis can identify evidence consistent with an explanation, but causal claims usually require stronger research designs or controlled experiments.

Four Forms of Analytics

The four-part framework below is widely used to describe increasing levels of analytical purpose. In practice, projects may combine them.

Descriptive analytics: What happened?

Descriptive analytics summarises historical or current data. It answers questions such as:

• How many units were sold last month?
• What was the average response time?
• Which region had the highest number of support tickets?
• How did website traffic change over the quarter?

Common techniques include counts, percentages, averages, medians, minimum and maximum values, frequency tables, grouped summaries and time-series charts. A simple arithmetic mean for n observed values x₁, x₂, …, xₙ is:

Here, x̄ is the sample mean, xᵢ is the i-th observation, and n is the number of observations. The formula is applicable when the values are meaningfully averaged, such as income, duration or quantity. It is not generally appropriate to average nominal category codes such as 1 = red and 2 = blue.

Descriptive analytics is often the first stage of an investigation, but it does not by itself explain causes or predict future outcomes.

Diagnostic analytics: Why did it happen?

Diagnostic analytics investigates possible explanations for an observed result. It may involve filtering and drilling down, comparing segments, examining distributions, checking data quality, analysing correlations, or tracing events through a process.

For example, a fall in completed deliveries may be examined by warehouse, route, product type, day and delivery partner. A diagnostic analysis might reveal that the overall decline was concentrated in one warehouse after a system change.

Diagnostic results should be interpreted cautiously. Correlation measures the strength and direction of association between variables; it does not automatically establish a causal relationship. A useful diagnostic process combines data evidence with operational knowledge and, where possible, controlled comparisons.

Predictive analytics: What is likely to happen?

Predictive analytics uses historical data, statistical models or machine-learning methods to estimate future or unknown outcomes. Examples include:

• forecasting next month’s demand;
• estimating the probability that a customer will cancel a subscription;
• predicting whether a machine will require maintenance;
• classifying an email as likely spam.

A prediction is conditional on the available information, the modelling assumptions and the stability of the environment. Predictive performance should be evaluated on data that were not used to fit the model. Metrics must match the task: for example, classification may use precision, recall or accuracy, while numerical forecasting may use mean absolute error.

Prescriptive analytics: What should be done?

Prescriptive analytics recommends actions by combining predictions, objectives, constraints and possible decisions. Examples include selecting delivery routes, allocating staff, setting inventory levels or prioritising inspections.

A prediction does not automatically imply a recommendation. For instance, predicting high demand does not determine whether an organisation should increase production; capacity, cost, risk, contractual commitments and sustainability constraints must also be considered. Prescriptive systems may use optimisation, simulation, decision analysis or rules based on policy.

Data, Variables and Datasets

Observations and variables

A variable is a characteristic that can take different values across units, people, events or time periods. An observation is one recorded instance of those characteristics. In a rectangular dataset, rows commonly represent observations and columns represent variables.

For a customer dataset, one row might represent one customer and columns might include customer identifier, age, region, subscription type and monthly spend. For a transaction dataset, one row might represent one purchase rather than one customer. Correctly identifying the unit of observation is essential. Confusing transactions with customers can lead to incorrect counts and biased conclusions.

Variables can be classified in several complementary ways:

The distinction between independent and dependent variables depends on the analytical design. A variable used as a predictor in one project may be the outcome in another. The labels do not prove causation.

Populations, samples and records

A population is the complete set of units relevant to a study, such as all customers who used a service during a year. A sample is a subset selected for analysis. Sampling is useful when collecting or processing the entire population is impractical. Conclusions from a sample should account for how the sample was selected. A convenience sample may not represent the population.

A record is a stored representation of an observation, often a row in a table. A dataset may contain one table or multiple related tables. In a relational database, a primary key uniquely identifies a record within a table, while a foreign key links records across tables. Analysts must understand these relationships before joining data; an incorrect join can duplicate records and inflate totals.

Structured and unstructured data

Structured data follows a defined schema, such as rows and columns in a relational database. Semi-structured data has some organisation but a flexible schema, as in JSON documents or event logs. Unstructured data lacks a regular tabular structure, as in free-text documents, images, audio and video.

Unstructured data can still be analysed, but it usually requires specialised representations and methods. Natural-language processing can convert text into features; computer vision can extract information from images; speech-processing systems can analyse audio. The resulting representations should still be documented, validated and interpreted carefully.

Measurement Scales

Measurement scale determines what comparisons and mathematical operations are meaningful.

Nominal scale

Nominal variables classify observations into distinct categories with no inherent order. Examples include department, colour, country and product type. Category labels may be encoded numerically for storage, but the numbers have no arithmetic meaning. Suitable summaries include counts, proportions and the mode.

Ordinal scale

Ordinal variables have a meaningful order, but the distances between categories are not necessarily equal. Examples include satisfaction ratings such as poor, fair, good and excellent, or risk levels such as low, medium and high. Medians and rank-based comparisons may be meaningful. Treating ordinal categories as equally spaced numerical values requires justification.

Interval scale

Interval variables have ordered values with meaningful and equal differences, but zero does not represent the complete absence of the quantity. Temperature measured in degrees Celsius is a common example: a difference of 10 °C has consistent interpretation, but 20 °C is not twice as hot as 10 °C. Addition and subtraction are meaningful; ratio statements require care.

Ratio scale

Ratio variables have equal intervals and a meaningful zero, allowing meaningful ratios. Examples include mass, duration, distance, age and revenue measured from zero. A value of 20 kg is twice 10 kg under the same measurement conditions.

Measurement-scale errors are common in practice. For example, converting “high”, “medium” and “low” into 3, 2 and 1 may be useful for a model or chart, but it does not prove that the distance from low to medium equals the distance from medium to high.

Data Quality and Preparation

Analysis is only as credible as the data and process supporting it. Important quality dimensions include:

• Accuracy: Does the value represent the real-world condition?
• Completeness: Are required values present?
• Consistency: Do values agree across systems and records?
• Validity: Does each value conform to allowed rules and formats?
• Timeliness: Is the data current enough for the decision?
• Uniqueness: Are duplicate records controlled?
• Lineage: Can the origin and transformations of the data be traced?

Common preparation tasks include correcting formats, standardising category labels, resolving duplicates, handling missing values, checking outliers and documenting transformations. Missing data should not be treated automatically as zero. A missing income value, for example, may mean that a question was not asked, a respondent declined to answer, or a system failed.

Outliers require investigation rather than automatic removal. An extreme value may be a data-entry error, a rare but valid event, or evidence of a new phenomenon. The appropriate treatment depends on the purpose of the analysis and should be documented.

A data dictionary should describe variable names, definitions, units, allowed values, source systems, missing-value codes and update frequency. This supports reproducibility and reduces misunderstandings between analysts and subject experts.

Visualisation for Analytical Communication

Visualisation maps data attributes to visual properties. Position along a common scale is generally easier to compare accurately than area, angle or colour intensity. A chart should therefore be selected according to the analytical task.

A strong visualisation has a clear purpose, informative title, labelled axes, appropriate units, readable scales, meaningful ordering and a source or time reference. It should reveal relevant variation rather than hide it.

Truncated axes can exaggerate differences in bar charts, while inconsistent category definitions can invalidate comparisons. Three-dimensional effects, unnecessary decoration and excessive colour often reduce clarity. Colour should be used consistently and should remain interpretable for people with colour-vision differences. A chart should not imply precision beyond what the data support.

Dashboards are collections of visualisations designed for repeated monitoring or decision support. They should distinguish leading indicators from lagging indicators, show targets and thresholds where relevant, and allow users to identify exceptions without overwhelming them. A dashboard is not automatically effective simply because it contains many charts.

Framing Analytical Questions and Success Criteria

A project should begin with a decision or problem, not with an attractive dataset. A useful analytical question identifies the subject, outcome, population, time period and intended use of the answer.

Weak question: “Can we analyse customer data?”

Stronger question: “Among customers who began a monthly subscription between January and June, which observable factors are associated with cancellation within the first 90 days, and which retention actions should be tested?”

The stronger question defines a population, outcome, time window and decision context. It also avoids claiming that associations are automatically causal.

A practical question-framing sequence is:

1. Decision: What decision will the analysis inform?
2. Stakeholder: Who will use the result, and what authority do they have?
3. Unit of analysis: Is the unit a customer, transaction, product, visit or day?
4. Outcome: What must be measured or predicted?
5. Predictors or explanatory factors: Which variables may be relevant?
6. Scope: What population, geography and time period are included?
7. Constraints: What privacy, cost, latency, fairness or operational limits apply?
8. Success criteria: How will usefulness and technical quality be judged?

Success criteria should be measurable and connected to the decision. They commonly include:

• Business or operational criteria: reduced processing time, fewer stockouts, increased retention or lower cost.
• Analytical criteria: acceptable error, calibration, coverage, stability or explanatory usefulness.
• Adoption criteria: whether intended users understand and act on the result.
• Governance criteria: privacy, fairness, security, auditability and compliance.

For a classification model, accuracy alone may be unsuitable when one class is rare. A confusion matrix records true positives, false positives, true negatives and false negatives. Precision and recall are defined as:

TP, FP and FN denote true positives, false positives and false negatives. Precision is the proportion of predicted positives that are correct; recall is the proportion of actual positives that are detected. These measures apply when a binary classification decision and a defined positive class exist. Their usefulness depends on the relative costs of errors. In a safety inspection, missing a dangerous case may be more serious than investigating an additional safe case.

For a numerical prediction, mean absolute error is:

Here, yᵢ is the observed value, ŷᵢ is the predicted value and n is the number of evaluated cases. MAE has the same units as the outcome and treats errors linearly. It should be calculated on an evaluation set appropriate to the intended use.

Illustrative Case: Reducing Missed Clinic Appointments

Background

This fictional illustrative case concerns a community clinic experiencing a high number of missed appointments. Missed appointments reduce service capacity and may delay patient care. The clinic wants to decide whether reminder interventions should be targeted toward particular appointment groups.

Problem formulation

The initial request—“Use AI to reduce missed appointments”—is too broad. The analyst reframes it as: “For appointments scheduled during the next six months, can routinely available information identify groups with higher non-attendance risk early enough to offer an appropriate reminder, while protecting patient privacy and avoiding unfair treatment?”

The unit of analysis is an appointment, not a patient. The outcome is whether the appointment was attended, cancelled in advance or missed. Potential variables include appointment lead time, appointment type, day of week, prior attendance history and whether a reminder was sent.

Analysis approach

The team first creates a data dictionary and checks duplicate appointment identifiers, date formats, missing values and inconsistent outcome labels. It distinguishes a missed appointment from a cancellation because these outcomes have different operational implications. Descriptive analysis compares attendance rates by appointment type and lead-time group. Diagnostic analysis examines whether the apparent differences remain after considering scheduling patterns.

If a predictive model is developed, the team separates training and evaluation periods to reduce leakage from future information. It evaluates recall for high-risk appointments, precision for the reminder list and the operational workload created by additional outreach. It also checks whether performance differs across relevant patient groups. A model should not use sensitive information merely because it is available; its inclusion requires a clear ethical and operational justification.

Solution and outcome

The clinic pilots a simple, interpretable risk rule using appointment lead time and previous attendance behaviour, combined with a standard reminder process. The pilot is evaluated against a comparable period using attendance rate, outreach workload and patient feedback. The clinic does not treat the model output as a medical judgement or deny appointments. Instead, it uses the output to offer reminders and transport information.

The outcome is considered successful only if missed appointments decline without disproportionate burden on particular groups, privacy breaches or unacceptable staff workload. The case illustrates that the analytical solution includes problem definition, data governance, evaluation design and implementation—not only model selection.

Learning points

1. The unit of analysis must match the decision.
2. Outcome definitions should distinguish materially different events.
3. Predictive performance and operational usefulness are different criteria.
4. A pilot or controlled comparison is stronger than relying on before-and-after impressions alone.
5. Fairness, privacy and human oversight belong in the project design from the beginning.

Professional, Ethical and Governance Considerations

Data work affects people, resources and opportunities. Analysts should collect and use data for a defined legitimate purpose, minimise unnecessary personal information, protect access and retain data only as long as justified. Organisations should document data ownership, consent or another lawful basis where applicable, retention periods and permitted uses.

Privacy protection includes access controls, encryption, secure transfer, pseudonymisation where appropriate and careful reporting of small groups. Removing names does not guarantee anonymity if records can be linked using dates, locations or other attributes.

Fairness requires examining whether data, measurement processes or decisions produce unjustified disparities. A model can perform well on average while performing poorly for an under-represented group. Fairness cannot be solved by a single universal metric; the relevant assessment depends on the context, harms, legal requirements and decision process.

Reproducibility is a professional responsibility. Analysts should retain source references, code or procedural steps, versioned data definitions, assumptions, validation results and limitations. Stakeholders should be told what the analysis cannot establish. In high-impact settings, human review, appeal processes and audit trails are especially important.

Contemporary Developments

Cloud platforms, automated data pipelines, real-time streams and generative AI have increased the speed and scale of analysis. These technologies can support data preparation, natural-language querying and automated visualisation, but they do not remove the need for verification. Automated systems may reproduce errors, expose confidential information or generate plausible but unsupported explanations.

Modern analytical teams increasingly use reproducible notebooks, version control, data-quality checks, model monitoring and data contracts. Data contracts specify expected fields, types, definitions and service responsibilities between data producers and consumers. Monitoring is necessary because data distributions, user behaviour and business processes change over time. A model or dashboard that was valid last year may become unreliable after a policy, market or system change.

Best-Practice Checklist

Before accepting an analytical result, ask:

• Is the decision and intended user clearly identified?
• Is the unit of analysis correct?
• Are variables defined, measured and documented?
• Are the data sufficiently accurate, complete, timely and representative?
• Has the visualisation been chosen for the analytical purpose?
• Are uncertainty, limitations and alternative explanations communicated?
• Are success criteria measurable and relevant to the decision?
• Have privacy, security, fairness and regulatory responsibilities been considered?
• Can the analysis be reproduced and audited?
• Will the result be monitored after implementation?

Chapter Summary

Data analysis is the systematic examination and interpretation of recorded observations, while analytics connects analytical work to organisational decisions. Visualisation communicates patterns through carefully selected visual encodings and should clarify rather than decorate. Descriptive analytics summarises what happened; diagnostic analytics investigates why; predictive analytics estimates what may happen; and prescriptive analytics recommends action under objectives and constraints.

Variables may be categorical or numerical, discrete or continuous, and explanatory or response-oriented. Datasets contain observations whose unit must be identified correctly. Measurement scales—nominal, ordinal, interval and ratio—determine which comparisons and operations are meaningful. Data quality, documentation and preparation are prerequisites for credible results.

Effective projects begin with a well-framed decision question and define technical, operational, adoption and governance success criteria. Ethical practice includes privacy protection, fairness assessment, transparency, reproducibility and human oversight. These foundations support later study of statistics, machine learning, dashboards and advanced analytics.

Key Terms

Data: Recorded observations, measurements, labels or symbols.

Information: Data organised and interpreted in context.

Data analysis: Systematic examination and interpretation of data.

Analytics: The use of data, methods and domain knowledge to support decisions.

Visualisation: Graphical representation of data or analytical results.

Dataset: A structured or unstructured collection of related data.

Variable: A characteristic that can take different values.

Observation: One recorded instance of a unit under study.

Unit of analysis: The entity represented by one observation.

Population: The complete set of units relevant to a study.

Sample: A subset selected from a population.

Categorical variable: A variable whose values identify groups or categories.

Numerical variable: A variable representing a quantity or count.

Nominal scale: A measurement scale with categories but no order.

Ordinal scale: An ordered scale without guaranteed equal intervals.

Interval scale: An ordered scale with equal intervals but no meaningful zero.

Ratio scale: A scale with equal intervals and a meaningful zero.

Descriptive analytics: Analysis of what happened.

Predictive analytics: Estimation of future or unknown outcomes.

Prescriptive analytics: Recommendation of actions under constraints.

Review Questions

Short Answer Questions

1. Distinguish data, information and analytics.
2. What is the difference between a variable and an observation?
3. Name the four measurement scales and give one example of each.
4. Why does correlation not necessarily establish causation?
5. State two important dimensions of data quality.

Long Answer Questions

1. Explain the four forms of analytics and show how they may be combined in one organisational project.
2. Discuss how the unit of analysis and measurement scale affect analytical conclusions.
3. Compare bar charts, line charts, histograms and scatter plots, including a suitable use for each.
4. Explain the process of converting a broad business request into a well-framed analytical question.
5. Discuss the ethical and governance responsibilities involved in using predictive analytics.

Analytical / Application Questions

1. A retailer reports that revenue increased after a new advertising campaign. Design a diagnostic analysis that would investigate whether the campaign plausibly contributed to the increase.
2. A hospital wants to predict missed appointments. Specify the unit of analysis, outcome variable, at least four possible predictors, two success criteria and two governance risks.
3. A dataset codes satisfaction as 1 = poor, 2 = fair, 3 = good and 4 = excellent. Explain which calculations and visualisations are defensible, and identify one potentially misleading practice.

Multiple Choice Questions with Answers

Question 1

Which type of analytics primarily answers the question “What happened?”

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: A. Descriptive analytics

Question 2

Which measurement scale has ordered categories but does not guarantee equal distances between categories?

A. Nominal

B. Ordinal

C. Interval

D. Ratio

Correct Answer: B. Ordinal

Question 3

In a rectangular dataset, what does a row commonly represent?

A. A variable definition

B. A measurement scale

C. An observation or record

D. A visual encoding

Correct Answer: C. An observation or record

Question 4

Which visualisation is generally most suitable for showing change over time?

A. Line chart

B. Pie chart

C. Scatter plot without a time axis

D. Nominal frequency table only

Correct Answer: A. Line chart

Question 5

Why should category codes such as 1 = red and 2 = blue not normally be averaged?

A. They are always continuous variables

B. The numbers are labels rather than quantities

C. They have a meaningful absolute zero

D. They are necessarily interval measurements

Correct Answer: B. The numbers are labels rather than quantities

Question 6

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0. fifty

C. 0.80

D. 1.00

Correct Answer: C. 0.80

Question 7

Which question is most clearly prescriptive?

A. How many orders were received yesterday?

B. Why did delivery times increase?

C. What delivery volume is likely next week?

D. Which staffing allocation should minimise expected delays?

Correct Answer: D. Which staffing allocation should minimise expected delays?

Question 8

What is the main danger of an incorrect join between transaction and customer tables?

A. It always removes all missing values

B. It can duplicate records and inflate totals

C. It converts ratio data into nominal data

D. It guarantees causal conclusions

Correct Answer: B. It can duplicate records and inflate totals

Question 9

Which statement about an outlier is most appropriate?

A. It must always be deleted

B. It is always evidence of fraud

C. It should be investigated before deciding how to treat it

D. It can never be a valid observation

Correct Answer: C. It should be investigated before deciding how to treat it

Question 10

Which is an appropriate success criterion for a predictive appointment-risk system?

A. The dashboard uses many colours

B. The model has an acceptable recall and manageable outreach workload

C. The dataset contains at least ten columns

D. The model produces a prediction without evaluation

Correct Answer: B. The model has an acceptable recall and manageable outreach workload