Foundations and Landscape of Data Science

Introduction

Data science is the systematic use of data, computing, statistical reasoning, domain knowledge and communication to produce evidence that supports understanding and action. It is not simply the use of software, nor is it identical to artificial intelligence or machine learning. A data scientist may design a measurement system, clean unreliable records, evaluate uncertainty, build a predictive model, explain its limitations and help an organisation decide what to do. The discipline therefore combines technical and human capabilities.

Data science has become important because organisations now collect data from transactions, sensors, websites, mobile devices, scientific instruments and social platforms. However, data alone does not create value. Value arises when data is relevant, trustworthy, interpreted correctly and connected to a decision or service. Poorly defined questions, biased samples, weak governance and inappropriate models can produce confident but harmful conclusions.

This chapter establishes the conceptual foundation for later study of data management, statistics, programming, machine learning and analytics.

Learning objectives

After studying this chapter, you should be able to:

1. Define data science and distinguish data, information, knowledge and insight.
2. Explain the historical development and interdisciplinary character of data science.
3. Compare data science with statistics, analytics, artificial intelligence, machine learning and business intelligence.
4. Describe common data-science roles, workflows, use cases and mechanisms of value creation.
5. Evaluate limitations, ethical risks and governance requirements.
6. Select an appropriate analytical approach for a practical organisational problem.

1. Fundamental concepts and terminology

Data, information, knowledge and insight

Data consists of recorded observations, measurements, symbols or representations. A temperature reading, a customer identifier and a text review are all data. Data may be structured, semi-structured or unstructured. Structured data fits a defined schema, such as rows and columns. Semi-structured data contains tags or variable fields, such as JSON documents. Unstructured data includes free text, images, audio and video.

Information is data organised and interpreted in context. “27” is data; “27 °C was recorded in Room B at 14:00” is information. Knowledge is a justified understanding of relationships, mechanisms or procedures developed from information and experience. Insight is a practically significant interpretation that can guide action. These categories are related, but the progression is not automatic: more data does not necessarily produce more knowledge.

A dataset is a collection of related observations. An observation, record or instance represents one unit being studied. A variable or feature records an attribute of an observation. A label, target or outcome is the variable a supervised model attempts to predict. A population is the complete set of units of interest, whereas a sample is the subset actually observed.

Descriptive, diagnostic, predictive and prescriptive questions

Data work often progresses through four kinds of question:

These categories are useful but not strict stages. A predictive model may be used descriptively, and causal analysis is needed before claiming that an intervention will produce an outcome.

Data quality

Important dimensions of data quality include accuracy, completeness, consistency, timeliness, validity, uniqueness and fitness for purpose. A dataset can be technically complete but unsuitable if it measures the wrong concept. For example, online clicks may be a poor proxy for learning. Data quality is therefore a socio-technical issue involving collection procedures, definitions, people, systems and incentives.

2. The historical development of data science

Data science has deep roots rather than a single invention. Probability theory developed through work on games of chance and later supported inference under uncertainty. Statistical methods became central to government, science, agriculture, medicine and industrial quality control. The growth of electronic computing enabled large-scale numerical calculation, databases and simulation.

During the second half of the twentieth century, researchers in statistics, pattern recognition, databases, operations research and artificial intelligence developed methods for extracting structure from data. The term “data mining” became associated with discovering patterns in large datasets, while “knowledge discovery in databases” emphasised the broader process of selecting, cleaning, transforming and interpreting data.

The modern use of “data science” reflects the convergence of several developments: inexpensive storage, networked systems, open-source programming tools, cloud computing, high-volume digital traces and advances in machine learning. The expansion of deep learning after the 2010s increased performance in image, language and speech tasks, but it did not eliminate the need for statistical reasoning, domain expertise or governance.

A useful historical lesson is that tools change faster than principles. Measurement validity, uncertainty, reproducibility, responsible use and communication remain essential regardless of whether analysis is performed with a spreadsheet, a cluster or a generative model.

3. The interdisciplinary landscape

Data science draws on several fields, each contributing a different perspective.

The overlap among these fields is substantial, but their objectives differ. A technically accurate model can still be operationally useless if it predicts an outcome that cannot be acted upon. Conversely, a simple descriptive report may create substantial value when it supports a timely decision.

4. Relationships among related disciplines

Data science and statistics

Statistics provides theories and methods for collecting data, quantifying variation, estimating unknown quantities and reasoning from samples. Classical statistical analysis often emphasises interpretability, assumptions and inference about a population or data-generating process. Data science includes these concerns but also places greater emphasis on software engineering, large-scale computation, unstructured data, deployment and product use.

A confidence interval, for example, communicates uncertainty in an estimated parameter under a specified sampling model. It should not be interpreted as a guarantee that a particular future observation lies inside the interval. Statistical thinking also distinguishes association from causation. If two variables move together, this may reflect confounding, reverse causality, selection or coincidence.

Data science and analytics

Analytics is the use of data and methods to generate evidence for decisions. Business, marketing, operational, financial and public-sector analytics are application domains. Data science is often broader in its concern with data products, algorithms, experimentation, engineering and automated systems. In practice the terms overlap, and organisations define job titles differently.

Data science and artificial intelligence

Artificial intelligence is a broad field concerned with machines performing tasks commonly associated with human intelligence, including perception, language processing, planning and decision-making. Some AI systems are rule-based and do not learn from data. Data science can support AI by preparing data and evaluating models, but much data science is not AI—for example, a carefully designed survey analysis.

Data science and machine learning

Machine learning is a set of computational methods in which model parameters are learned from data. In supervised learning, examples include an outcome label. In unsupervised learning, the system seeks structure without a supplied label, as in clustering. In reinforcement learning, an agent learns through interactions and rewards.

A generic supervised-learning objective can be written as:

Here, θ represents model parameters, fθ is the model, xi is the input for observation i, yi is its observed outcome, L is a loss function, n is the number of training observations, and θ̂ is the parameter value that minimises average training loss. The formula describes an optimisation objective, not a guarantee of good generalisation. A separate validation or test procedure is required, and the observations should be appropriate for the intended population.

Data science and business intelligence

Business intelligence (BI) traditionally focuses on reporting, dashboards, standard metrics and interactive exploration of historical or near-real-time organisational data. Data science may extend BI through prediction, experimentation, optimisation and automated decisions. BI answers questions such as “What were sales by region?” Data science may ask “Which factors predict demand, and how should inventory be allocated?” Mature organisations integrate both: reliable BI establishes common definitions, while data science develops advanced analyses.

5. The data-science lifecycle

A useful lifecycle is iterative rather than linear:

1. Frame the problem. Define the decision, users, constraints, success criteria and possible harms.
2. Acquire and understand data. Identify sources, ownership, population, sampling process, permissions and measurement definitions.
3. Prepare data. Integrate sources, resolve types and units, document transformations, handle missingness and detect errors.
4. Explore. Examine distributions, relationships, outliers, subgroup differences and possible leakage.
5. Model or analyse. Select a method appropriate to the question, data structure and decision cost.
6. Evaluate. Assess predictive performance, uncertainty, robustness, fairness, interpretability and operational usefulness.
7. Communicate. Present assumptions, evidence, limitations and recommended action to relevant audiences.
8. Deploy and monitor. Integrate the result into a workflow and monitor drift, data quality, performance and unintended effects.
9. Review or retire. Reassess whether the system remains valid, lawful, useful and proportionate.

The lifecycle resembles the CRISP-DM process—business understanding, data understanding, data preparation, modelling, evaluation and deployment—but modern practice adds explicit governance, monitoring and responsible innovation.

6. Roles and professional capabilities

Roles vary by organisation. A data analyst prepares data, produces reports and investigates trends. A data scientist develops statistical or machine-learning analyses and translates them into decisions or products. A data engineer builds reliable pipelines, storage and processing systems. A machine-learning engineer operationalises models, including serving, testing and monitoring. A data architect designs data structures and integration principles. A BI developer builds semantic models, dashboards and reporting systems. A domain expert supplies contextual knowledge and validates whether outputs make sense. A data steward supports definitions, quality, access and governance.

Effective professionals need more than tool familiarity. Core capabilities include problem formulation, SQL and programming, data visualisation, statistical reasoning, version control, documentation, communication, project management, privacy awareness and ethical judgement. Collaboration is essential because no single role normally owns the whole lifecycle.

7. Value creation and use cases

Data creates value through several mechanisms:

• Efficiency: automating repetitive classification, scheduling or document processing.
• Effectiveness: improving diagnosis, targeting, forecasting or resource allocation.
• Risk reduction: identifying fraud, equipment failure, cyber threats or unsafe conditions.
• Revenue and service innovation: personalisation, recommendation and new digital products.
• Learning: experimentation and feedback that improve processes over time.
• Public and social benefit: epidemiological surveillance, transport planning and environmental monitoring.

A use case should connect an analytical output to an action. For example, a churn score has little value if no intervention is available, if intervention costs exceed expected benefit or if the score unfairly excludes particular customers. Value should therefore be assessed using technical measures and decision measures such as cost, benefit, time, safety, equity and user adoption.

8. Illustrative case: reducing missed hospital appointments

Background. A regional clinic experiences missed appointments, reducing access and wasting clinical capacity. It has appointment records, lead time, clinic type, reminder history and attendance outcomes.

Problem. The clinic asks whether data can identify appointments at elevated risk of non-attendance without unfairly penalising patients.

Analysis. The team first defines the unit of analysis as an appointment and the prediction time as 48 hours before the appointment. It checks whether variables available after that time have accidentally leaked the outcome. It examines missing values, duplicate records and subgroup attendance rates. It also consults staff and patient representatives to identify relevant constraints.

Approach. A transparent baseline, such as the historical attendance rate by clinic and lead-time group, is compared with a logistic-regression model and a tree-based model. The team evaluates discrimination, calibration and operational impact rather than accuracy alone. A high-risk flag triggers an additional reminder or an offer to reschedule; it does not cancel an appointment or deny care. Thresholds are chosen using the relative costs of unnecessary reminders and missed appointments. Privacy controls limit access, and model performance is monitored as booking practices change.

Outcome. In this illustrative case, the model is valuable only if the intervention reduces missed appointments without increasing inequity. A seemingly accurate model could be rejected if it relies on protected or proxy variables, performs poorly for a small subgroup or encourages harmful administrative action.

Learning points. Problem framing precedes modelling; a baseline is necessary; prediction does not establish causation; interventions must be evaluated; and governance is part of technical quality.

9. Limitations, uncertainty and common failure modes

Data is not a neutral mirror of reality. It reflects what was measured, who was included, how categories were defined and which behaviours were recorded. Common problems include:

• Selection bias: the observed sample differs systematically from the target population.
• Measurement bias: a variable is recorded inaccurately or represents a poor proxy.
• Confounding: a third factor influences both an apparent cause and outcome.
• Missing-data bias: missingness is related to variables of interest.
• Data leakage: information unavailable at prediction time enters model training.
• Overfitting: a model captures noise or idiosyncrasies in training data.
• Distribution shift: deployment conditions differ from development conditions.
• Automation bias: users accept system outputs without appropriate scrutiny.
• Metric fixation: a convenient metric is optimised while the real objective is neglected.
• Reproducibility failure: code, data definitions or transformations are not documented.

A model’s output is conditional on its data, design and assumptions. Uncertainty should be communicated, not hidden behind excessive decimal places. Validation must reflect how the system will actually be used, including time order, geographic transfer and relevant subgroups.

10. Ethics, governance and regulation

Responsible data science requires lawful, fair and accountable practice. Key principles include purpose limitation, data minimisation, transparency, security, retention control, human oversight and the ability to challenge consequential decisions. Applicable obligations vary by jurisdiction and sector; privacy and data-protection law may impose requirements concerning consent or another lawful basis, access rights, profiling, security and breach response.

Ethical analysis also asks questions beyond legal compliance: Who benefits? Who bears the risk? Can people reasonably refuse? Is the use proportionate? Could the system reproduce historical discrimination? Are explanations meaningful to affected people? Governance should assign responsibility, maintain documentation, record model versions, test subgroup performance and provide incident procedures.

Generative AI introduces additional concerns, including fabricated outputs, confidential-data exposure, intellectual-property issues, prompt injection and unclear provenance. Human review is especially important when outputs influence health, employment, education, credit, safety or legal status. “Human in the loop” is not sufficient if the human lacks time, authority or understanding to challenge the system.

11. Contemporary developments and best practices

Current practice increasingly includes cloud data platforms, real-time streaming, foundation models, synthetic data, privacy-enhancing computation, automated machine learning and responsible-AI tooling. These developments can improve access and productivity, but they also increase scale and the speed at which errors propagate.

Good practice includes:

1. Start with a decision and stakeholder need, not an attractive dataset.
2. Establish a data dictionary, provenance record and quality checks.
3. Build a simple baseline before using a complex model.
4. Separate training, validation and testing in a way that respects time and dependence.
5. Use measures aligned with the decision, including cost and equity.
6. Document assumptions, exclusions, transformations and limitations.
7. Protect sensitive data through access control, minimisation and secure processing.
8. Involve domain experts and affected communities.
9. Make analysis reproducible through versioned code, environments and data definitions.
10. Monitor deployed systems and define conditions for retraining or withdrawal.

Chapter Summary

Data science integrates statistical reasoning, computing, domain knowledge, communication and governance to turn data into evidence for action. It encompasses descriptive, diagnostic, predictive and prescriptive work, but prediction alone does not establish causation or guarantee value. Statistics supplies inference and uncertainty; analytics connects evidence to decisions; AI is a broad field of intelligent systems; machine learning learns patterns from data; and BI provides reporting and organisational visibility. A professional lifecycle moves from problem framing through data understanding, preparation, analysis, evaluation, communication, deployment and monitoring. Value depends on actionability, quality and context. Bias, leakage, overfitting, distribution shift, privacy risks and automation bias must be addressed through sound methods and responsible governance.

Key Terms

Data: Recorded observations, measurements or representations.

Dataset: A collection of related observations.

Variable: An attribute recorded for an observation.

Feature: An input variable used by a model.

Label: The observed target outcome in supervised learning.

Population: The complete set of units of interest.

Sample: A subset observed from a population.

Data quality: The fitness of data for its intended use.

Analytics: Use of data and methods to support decisions.

Statistics: The study of data collection, variation, estimation and inference.

Artificial intelligence: Systems designed to perform tasks associated with intelligent behaviour.

Machine learning: Methods that learn patterns or parameters from data.

Business intelligence: Reporting, metrics and exploration of organisational data.

Data leakage: Use of information unavailable at the intended prediction time.

Overfitting: Learning noise or peculiarities rather than general patterns.

Confounding: Distortion caused by a variable related to both compared factors.

Distribution shift: A change between development and deployment data conditions.

Data governance: Policies, roles and controls for responsible data management.

Model monitoring: Ongoing assessment of data, performance and operational effects.

Insight: An interpretation that is significant and useful for action.

Review Questions

Short Answer Questions

1. Distinguish data, information, knowledge and insight.
2. What is the difference between descriptive and predictive analytics?
3. State two contributions that statistics makes to data science.
4. Define data leakage and explain why it is harmful.
5. Name four dimensions of data quality.

Long Answer Questions

1. Compare data science, machine learning, artificial intelligence and business intelligence.
2. Explain the stages of a data-science lifecycle and why the lifecycle is iterative.
3. Discuss how data science can create organisational value, including conditions required for value.
4. Explain selection bias, confounding, overfitting and distribution shift, using examples.
5. Discuss ethical and governance requirements for a system used in consequential decisions.

Analytical / Application Questions

1. A retailer wants to predict customer churn. Propose a problem definition, suitable data, evaluation approach and ethical safeguards.
2. A hospital model has high overall accuracy but substantially lower recall for a minority group. Analyse possible causes and recommend actions before deployment.
3. A dashboard shows that sales rose after a marketing campaign. Explain why this does not by itself establish that the campaign caused the increase, and design a stronger evaluation.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes information from raw data?

A. Information is always numerical, whereas data is always textual.

B. Information is data organised and interpreted in context.

C. Information is data that has been stored in the cloud.

D. Information is data collected only from experiments.

Correct Answer: B. Information is data organised and interpreted in context.

Question 2

Which question is primarily predictive?

A. How many orders were late last quarter?

B. Why did delivery delays increase?

C. Which future orders are likely to be late?

D. What policy should management adopt?

Correct Answer: C. Which future orders are likely to be late?

Question 3

What is the principal purpose of a test dataset?

A. To replace the need for data cleaning

B. To estimate performance on data not used to fit the model

C. To increase the number of model parameters

D. To guarantee causal validity

Correct Answer: B. To estimate performance on data not used to fit the model

Question 4

Which discipline primarily contributes sampling, estimation and uncertainty quantification?

A. Statistics

B. Graphic design

C. Database administration

D. Project management

Correct Answer: A. Statistics

Question 5

A model uses the final outcome status as an input when making predictions before that outcome occurs. This is an example of:

A. Clustering

B. Regularisation

C. Data leakage

D. Stratified sampling

Correct Answer: C. Data leakage

Question 6

Which statement about machine learning is most accurate?

A. It always establishes causation.

B. It learns patterns or parameters from data.

C. It is identical to business intelligence.

D. It requires only unstructured data.

Correct Answer: B. It learns patterns or parameters from data.

Question 7

Which is the best first step in a data-science project?

A. Choose the most complex available algorithm.

B. Deploy a dashboard immediately.

C. Frame the decision, users and success criteria.

D. Delete all missing records.

Correct Answer: C. Frame the decision, users and success criteria.

Question 8

A model performs well during development but poorly after a major change in customer behaviour. This most directly illustrates:

A. Distribution shift

B. Data minimisation

C. Data visualisation

D. Randomisation

Correct Answer: A. Distribution shift

Question 9

Why is a baseline useful in model development?

A. It provides a simple reference for judging whether added complexity helps.

B. It ensures that the model is unbiased.

C. It eliminates the need for validation.

D. It converts unstructured data into labels automatically.

Correct Answer: A. It provides a simple reference for judging whether added complexity helps.

Question 10

Which action best reflects responsible deployment of a high-impact model?

A. Hide limitations so users trust it.

B. Monitor performance and subgroup effects after deployment.

C. Use as many personal variables as possible.

D. Allow the model to make irreversible decisions without review.

Correct Answer: B. Monitor performance and subgroup effects after deployment.