Foundations of Data Science

Introduction

Data science is the systematic use of data, computational methods, statistical reasoning, and domain knowledge to generate useful evidence for decisions and action. It is not simply the use of programming, databases, or machine learning. A successful data science project connects a real-world question with appropriate data, valid methods, interpretable results, responsible governance, and an operational decision.

Data science has become important because organizations now collect data from transactions, sensors, mobile devices, websites, scientific instruments, social platforms, and enterprise systems. The availability of data does not automatically create value. Data may be incomplete, biased, poorly documented, duplicated, insecure, or irrelevant to the decision being considered. Data science therefore combines technical competence with critical thinking and communication.

This chapter introduces the vocabulary, intellectual foundations, analytical categories, problem-formulation process, CRISP-DM framework, and end-to-end lifecycle of data science.

Learning Objectives

After studying this chapter, you should be able to:

1. Define data science and distinguish it from related disciplines.
2. Explain the roles of statistics, computing, mathematics, and domain knowledge.
3. Classify business and research questions as descriptive, diagnostic, predictive, or prescriptive.
4. Formulate a data science problem in terms of decisions, outcomes, populations, and constraints.
5. Describe and apply the CRISP-DM framework.
6. Explain the complete lifecycle from problem discovery to deployment and monitoring.
7. Identify ethical, legal, organizational, and technical risks in data science projects.

1. What Is Data Science?

A useful definition is:

Data science is an interdisciplinary practice that uses data, scientific reasoning, computation, and communication to produce reliable knowledge and support decisions.

The definition contains several important ideas.

Data are recorded representations of observations, events, measurements, or claims. Data may be structured, such as rows in a relational table, or unstructured, such as text, images, audio, and video.

Science implies disciplined inquiry. A data scientist should state a question, identify evidence, make assumptions explicit, test alternatives, and communicate uncertainty. A visually impressive chart is not necessarily scientific evidence.

Computation includes data storage, programming, algorithms, databases, visualization, and deployment systems. Computational methods make it possible to process data at a scale or speed that manual analysis cannot achieve.

Decision support means that the result is connected to an action or judgment. A model may estimate the probability that a customer will leave, but the organization must still decide whether and how to intervene.

Data science should be distinguished from several related fields:

These areas overlap. A modern data science team often includes data engineers, analysts, data scientists, machine learning engineers, visualization specialists, subject-matter experts, product managers, and governance professionals.

2. The Interdisciplinary Foundations

2.1 Statistics and probability

Statistics provides methods for describing data, estimating unknown quantities, assessing uncertainty, testing claims, and designing studies. Probability provides a language for uncertain events and forms the basis of many predictive models.

Important statistical ideas include population, sample, variable, parameter, statistic, distribution, bias, variance, correlation, confidence interval, and hypothesis test. A data scientist must understand that an observed association does not automatically establish causation. For example, ice-cream sales and drowning incidents may rise together because both are influenced by hot weather. Temperature is a confounding factor.

A simple summary measure is the arithmetic mean:

Here, xᵢ is the i-th observation, n is the number of observations, and x̄ is the sample mean. The formula is appropriate for quantitative observations when a mean is meaningful; extreme values can make it unrepresentative. The median may be more suitable for heavily skewed data.

For a binary classification problem, common evaluation measures include accuracy, precision, recall, and F1 score. If TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives:

These formulas assume that the classification outcomes have been clearly defined. Accuracy can be misleading for imbalanced classes. In a disease-screening system, recall may be especially important because missing a genuine case can have serious consequences, while precision matters when follow-up examinations are costly.

2.2 Mathematics and optimization

Mathematics supports the representation of relationships and the optimization of models. Linear algebra is used for vectors, matrices, and transformations; calculus supports continuous optimization; discrete mathematics supports algorithms and structures; and optimization identifies parameter values that minimize loss or maximize utility.

A generic supervised-learning objective can be expressed as:

Here, xᵢ is an input, yᵢ is the observed target, fθ is a model with parameters θ, L is a loss function, Ω is a complexity penalty, λ ≥ 0 controls regularization, and n is the number of training observations. The first term measures fit to the data, while the second discourages excessive complexity. This formulation is general; the choice of loss and penalty depends on the task and model.

2.3 Computing and information systems

Computing enables the collection, storage, transformation, analysis, visualization, and deployment of data products. Core competencies include programming, algorithmic thinking, databases, version control, testing, cloud platforms, and application programming interfaces.

Data systems may be batch-oriented, where data are processed periodically, or streaming-oriented, where events are processed continuously or near real time. The appropriate architecture depends on latency requirements, cost, reliability, and the consequences of delayed information.

2.4 Domain knowledge

Domain knowledge explains what variables mean, how processes operate, what outcomes matter, and which constraints are realistic. A model trained without domain understanding may use a technically available but conceptually invalid target. For example, predicting whether a patient will receive a particular treatment may reproduce historical access patterns rather than predict clinical need.

2.5 Communication and visualization

Data science produces value only when findings can be understood and acted upon. Visualization helps reveal patterns, outliers, trends, and comparisons. Communication requires explaining assumptions, uncertainty, limitations, and implications to both technical and non-technical audiences.

3. Important Data Science Terminology

A dataset is a collection of related observations. A record or instance is one observational unit. A feature or predictor is an input variable used to describe or predict an outcome. A target, label, or response is the outcome of interest.

Structured data fit a predefined schema, such as a table. Semi-structured data contain organizational markers but not a rigid tabular structure, such as JSON or XML. Unstructured data include free text, images, and audio.

Training data are used to fit a model, validation data support model selection and tuning, and test data provide an estimate of performance on previously unseen data. Test data should not repeatedly influence model choices, or the reported performance may be overly optimistic.

Data quality includes accuracy, completeness, consistency, timeliness, validity, uniqueness, and relevance. Data leakage occurs when information unavailable at prediction time enters model development, producing unrealistically strong results.

A model is a formal representation used to describe, estimate, classify, predict, or optimize. A parameter is learned or estimated within a model; a hyperparameter is set before or around training. Overfitting occurs when a model captures noise or peculiarities of the training data and performs poorly on new data. Underfitting occurs when the model is too simple to represent important structure.

A data product is an analytical output integrated into a practical workflow, such as a fraud alert, recommendation system, dashboard, or demand forecast. A data pipeline is an automated sequence that moves and transforms data. Model drift refers to changes in data patterns or relationships that reduce model performance after deployment.

4. Types of Analytics

Analytics can be classified by the kind of question being answered. The categories overlap, but they provide a useful progression from understanding the past to supporting action.

4.1 Descriptive analytics

Descriptive analytics summarizes what has happened or what is currently observed. Examples include monthly sales reports, customer counts, average delivery times, and dashboards. It commonly uses aggregation, distributions, cross-tabulation, and visualization.

A descriptive result should identify the population, time period, units, and definitions. “Revenue increased by 12%” is incomplete unless the comparison period, currency, inclusion rules, and adjustment for returns are known.

4.2 Diagnostic analytics

Diagnostic analytics investigates why a result occurred. It may involve segmentation, drill-down analysis, comparison groups, process analysis, or investigation of anomalies. Diagnostic analysis can generate hypotheses, but observational evidence alone may not prove causation.

4.3 Predictive analytics

Predictive analytics estimates an unknown, future, or unobserved outcome. Examples include credit-risk estimation, demand forecasting, equipment-failure prediction, and customer-churn prediction. A predictive model should be evaluated on data representative of the deployment context.

Prediction is not the same as explanation. A variable can improve predictive accuracy without being a causal driver. This distinction is essential when a model is used to select interventions.

4.4 Prescriptive analytics

Prescriptive analytics recommends actions under objectives and constraints. It may combine predictions with optimization, simulation, rules, or human judgment. For example, a delivery system may predict demand and then recommend vehicle assignments subject to capacity, time, and cost constraints.

Prescriptive systems must make trade-offs visible. Maximizing profit may conflict with fairness, service quality, safety, sustainability, or legal obligations.

5. Formulating a Data Science Problem

Many projects fail before modeling begins because the problem is vague. “Use artificial intelligence to improve customer service” is an aspiration, not a data science problem. A stronger formulation identifies the decision, decision-maker, affected population, outcome, time horizon, constraints, and success criteria.

A practical problem statement can follow this structure:

For [decision-maker], determine whether or how to [decision] for [population] using information available at [time], in order to improve [measurable outcome], subject to [constraints and safeguards].

For example:

For the university advising office, estimate which currently enrolled students may need additional academic support by the end of the semester, using information available at the midterm point, so that advisors can prioritize outreach while protecting student privacy and avoiding discriminatory decisions.

This statement is stronger because it identifies the user, prediction time, target population, action, outcome, and constraints.

From organizational objective to analytical task

The formulation process usually includes the following steps:

1. Clarify the decision. What action might change as a result of the analysis?
2. Identify stakeholders. Who uses the result, and who may be affected?
3. Define the unit of analysis. Is it a customer, transaction, patient, machine, household, or time interval?
4. Define the target. Specify exactly what is measured, when it is measured, and how it is labeled.
5. Set the prediction or observation horizon. A target must not use information from the future relative to the decision.
6. Identify constraints. Consider cost, latency, privacy, fairness, interpretability, regulation, and available resources.
7. Choose evaluation criteria. Metrics should reflect the decision and the consequences of errors.
8. Establish a baseline. Compare the proposed method with a simple rule, existing process, or current performance.

A useful success criterion is measurable and operational. “The model must be accurate” is insufficient. A better criterion might require a specified recall at an acceptable false-alert rate, or a reduction in delivery delays without increasing operating cost beyond a stated limit.

6. CRISP-DM

CRISP-DM, or Cross-Industry Standard Process for Data Mining, is a widely used process model. It consists of six phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. The phases are iterative rather than strictly linear.

Figure: CRISP-DM phases as an iterative process rather than a one-way sequence.

6.1 Business understanding

The team defines objectives, stakeholders, resources, risks, assumptions, and success criteria. The output may include a problem statement, project charter, decision analysis, and preliminary ethical assessment. This phase prevents technically successful but strategically irrelevant work.

6.2 Data understanding

The team identifies data sources, collects initial data, describes variables, explores distributions, checks quality, and detects possible leakage. Questions include: How were the data generated? Which cases are missing? What population is represented? Are labels reliable? Does the data reflect the deployment environment?

6.3 Data preparation

Data preparation may include selection, integration, cleaning, transformation, feature construction, labeling, and partitioning. It is often the most time-consuming phase. Every transformation should be documented and reproducible. When data are split into training and test sets, operations that learn from the data—such as imputation or scaling—should generally be fitted using training data and then applied to other partitions to avoid leakage.

6.4 Modeling

The team selects techniques, constructs models, tunes settings, and records experiments. A simple interpretable baseline is valuable even when advanced algorithms are being considered. Model choice should reflect not only performance but also latency, maintainability, explainability, robustness, and governance requirements.

6.5 Evaluation

Evaluation asks whether the model solves the original problem and whether it is safe and appropriate for use. This includes statistical performance, subgroup performance, calibration where probabilities are used, error analysis, robustness, and operational simulation. A model should not be approved merely because it has the best single metric.

6.6 Deployment

Deployment may involve a report, dashboard, batch scoring process, application interface, or embedded decision system. Deployment requires documentation, access control, monitoring, retraining policies, incident response, and ownership. A model that cannot be maintained is not a successful data product.

7. The End-to-End Data Science Lifecycle

CRISP-DM is a project framework; the broader lifecycle emphasizes continuing operation. A typical lifecycle includes:

1. Discover: identify a meaningful opportunity or problem.
2. Define: formulate the decision, target, constraints, and success criteria.
3. Acquire: obtain data legally and ethically from suitable sources.
4. Prepare: clean, integrate, label, transform, and document data.
5. Explore: examine distributions, relationships, missingness, and anomalies.
6. Model: develop and compare appropriate analytical or machine-learning methods.
7. Validate: test generalization, fairness, robustness, and practical utility.
8. Communicate: present findings with uncertainty and limitations.
9. Deploy: integrate the result into a workflow or product.
10. Monitor: track data quality, performance, drift, user behavior, and harms.
11. Retire or revise: update, replace, or discontinue the system when conditions change.

The lifecycle is governed by feedback. Monitoring may reveal that the target definition is wrong, data quality has deteriorated, or user behavior has changed. The team may then return to problem definition or data understanding rather than simply retraining the model.

8. Illustrative Case: Predicting Student Support Needs

Background

A university wants to improve academic advising. Advisors have limited time and cannot meet every student frequently. The university considers a system that identifies students who may benefit from early outreach.

Problem

The initial request is “build a model to identify at-risk students.” This is too vague. The team reformulates it: estimate, at midterm, which enrolled students are likely to fail at least one course by the end of the semester, so advisors can offer voluntary support.

Analysis

The team identifies the unit of analysis as a student-semester. Candidate data include attendance records, learning-platform activity, assessment submissions, prior academic performance, and course load. The team excludes sensitive information unless there is a lawful, justified, and carefully governed reason to use it. It also examines whether historical labels reflect unequal access to teaching and support.

The outcome is defined as failure in at least one enrolled course by semester end. The prediction point is midterm. Information recorded after midterm must not be used. A baseline such as “refer students with no submitted assessment by midterm” is compared with statistical and machine-learning models.

Approach and solution

The team partitions historical student-semesters by time so that evaluation resembles future use. It measures recall, precision, calibration, and subgroup error rates. Because an alert may lead to supportive outreach rather than punishment, the system is designed as an advisor-assistance tool, not an automatic decision-maker. Students are informed about available support, and advisors receive explanations and data-quality warnings.

Outcome

In an illustrative deployment, the system helps advisors prioritize outreach, but it does not determine eligibility for progression or funding. Monitoring shows that course participation data are missing more often in some departments. The team therefore treats missingness as a governance and service issue rather than assuming that missing data indicate disengagement.

Learning points

The case demonstrates that a data science project involves more than model selection. The target, timing, baseline, action, evaluation metrics, privacy safeguards, human oversight, and monitoring plan are equally important. A technically accurate prediction can still produce harm if it is used for an inappropriate decision.

9. Ethics, Governance, and Responsible Practice

Data science can affect employment, education, health, credit, insurance, policing, and access to services. Ethical practice requires attention to both direct and indirect effects.

Privacy requires collecting and using personal data for legitimate purposes, limiting access, protecting data, and retaining it only as necessary. Consent may be important, but consent alone does not remove all obligations.

Fairness concerns whether individuals or groups experience unjustified differences in errors, opportunities, or outcomes. Fairness definitions can conflict mathematically and operationally, so the relevant standard must be connected to the context and law rather than chosen mechanically.

Transparency means documenting data sources, definitions, assumptions, model purpose, limitations, and responsible owners. Explainability concerns whether users can understand how an output was generated. Accountability requires that a person or institution remains responsible for decisions, even when automated tools are involved.

Good practice includes data minimization, access control, encryption, audit logs, reproducible workflows, versioned datasets and models, human review for high-impact decisions, pre-deployment testing, and post-deployment monitoring. Security threats include unauthorized access, data poisoning, model extraction, and adversarial manipulation.

10. Challenges and Limitations

Data science projects face recurring limitations:

• Poor problem definition: A model may optimize a proxy rather than the real objective.
• Sampling bias: The collected data may not represent the target population.
• Measurement error: Variables may be inaccurate or inconsistently recorded.
• Missing data: Missingness may be systematic and informative.
• Confounding: Observational relationships may not support causal conclusions.
• Data leakage: Future or outcome-derived information may inflate evaluation.
• Distribution shift: Deployment conditions may differ from development conditions.
• Overfitting: A complex model may perform well only on historical data.
• Metric mismatch: A convenient metric may not reflect social or operational costs.
• Organizational failure: Users may distrust, misuse, or ignore the output.
• Maintenance burden: Data pipelines and models require continuing ownership.

The solution is not always more data or a more complex algorithm. Better definitions, experiments, documentation, domain consultation, and process redesign may produce greater value.

11. Professional Best Practices

A competent data science practitioner should:

1. Begin with the decision and affected people, not with a fashionable algorithm.
2. Define variables, labels, time windows, and units precisely.
3. Establish a simple baseline before developing complex models.
4. Separate training, validation, and final testing appropriately.
5. Treat data preparation as an auditable analytical activity.
6. Report uncertainty, limitations, and subgroup performance.
7. Use visualizations that are accurate, accessible, and suited to the audience.
8. Maintain reproducible code, data dictionaries, experiment records, and model documentation.
9. Include domain experts, affected stakeholders, security specialists, and governance personnel.
10. Monitor deployed systems and provide a process for correction, appeal, and retirement.

Contemporary practice increasingly includes automated machine-learning tools, cloud analytics, real-time pipelines, generative AI, and foundation models. These technologies can accelerate development, but they do not eliminate the need for problem formulation, data governance, evaluation, human judgment, and accountability. Generated code or text must be tested, and generated outputs must not be treated as evidence without verification.

Chapter Summary

Data science is an interdisciplinary practice that combines data, statistics, computing, mathematics, domain knowledge, and communication to support reliable decisions. It differs from, but overlaps with, statistics, computer science, data engineering, machine learning, and business intelligence.

Descriptive analytics summarizes what happened; diagnostic analytics investigates why; predictive analytics estimates what may happen; and prescriptive analytics recommends actions under objectives and constraints. A well-formulated problem identifies the decision, stakeholders, unit of analysis, target, time horizon, success criteria, and safeguards.

CRISP-DM provides six iterative phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. The broader lifecycle adds discovery, validation, communication, monitoring, and retirement or revision. Successful projects attend to data quality, leakage, bias, uncertainty, privacy, fairness, security, maintainability, and organizational adoption. The central professional lesson is that a useful data product is not merely an accurate model; it is a trustworthy, well-governed solution to a meaningful problem.

Key Terms

Analytics: Systematic examination of data to produce insight or support decisions.

Baseline: A simple reference method used for comparison.

CRISP-DM: An iterative framework for data-mining and data-science projects.

Data leakage: Use of information during development that would not be available at decision time.

Data product: An analytical output integrated into a practical workflow or service.

Data quality: The degree to which data are accurate, complete, consistent, timely, valid, unique, and relevant.

Deployment: Integration of an analytical result into operational use.

Feature: An input variable used to describe or predict an outcome.

Model drift: Decline in model suitability caused by changes in data or relationships.

Overfitting: Fitting noise or peculiarities in development data rather than general patterns.

Prescriptive analytics: Analysis that recommends actions under objectives and constraints.

Predictive analytics: Analysis that estimates future, unknown, or unobserved outcomes.

Target: The outcome a model or analysis seeks to explain, classify, or predict.

Training data: Data used to fit model parameters.

Validation data: Data used to select models or tune settings.

Test data: Data reserved for estimating performance on unseen cases.

Unstructured data: Data such as text, images, audio, or video without a fixed tabular schema.

Visualization: Graphical representation of data designed to support understanding.

Review Questions

Short Answer Questions

1. Define data science and identify three disciplines that contribute to it.
2. Distinguish descriptive analytics from predictive analytics.
3. What is data leakage, and why does it produce misleading evaluation results?
4. Name the six phases of CRISP-DM.
5. Why is domain knowledge important in a data science project?

Long Answer Questions

1. Explain the interdisciplinary foundations of data science and discuss the contribution of statistics, computing, mathematics, and domain knowledge.
2. Describe the four major types of analytics and provide a suitable organizational example for each.
3. Explain how a vague organizational request can be transformed into a well-formulated data science problem.
4. Discuss each phase of CRISP-DM and explain why the phases should be treated as iterative.
5. Analyze the ethical, governance, and operational requirements for deploying a predictive model in education or healthcare.

Analytical / Application Questions

1. A retailer wants to reduce late deliveries. Formulate a data science problem by specifying the decision-maker, unit of analysis, target, prediction horizon, success metrics, and constraints.
2. A disease-screening model has high accuracy because only 2% of the tested population has the disease. Explain why accuracy may be inadequate and recommend more appropriate evaluation considerations.
3. A deployed customer-churn model performs well initially but poorly six months later. Using the data science lifecycle, identify possible causes and propose a monitoring and response plan.

Multiple Choice Questions with Answers

Question 1

Which statement best describes data science?

A. The storage of all organizational data in a database

B. The use of data, computation, reasoning, and domain knowledge to support reliable decisions

C. The creation of charts without statistical analysis

D. The replacement of all human decisions with algorithms

Correct Answer: B. The use of data, computation, reasoning, and domain knowledge to support reliable decisions

Question 2

Which type of analytics investigates why an observed outcome occurred?

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: B. Diagnostic analytics

Question 3

In a supervised-learning project, what is the target?

A. The software used to train a model

B. The input variable selected by the analyst

C. The outcome the model seeks to explain or predict

D. The database containing raw records

Correct Answer: C. The outcome the model seeks to explain or predict

Question 4

Which CRISP-DM phase focuses on checking data sources, distributions, missingness, and initial quality?

A. Business understanding

B. Data understanding

C. Modeling

D. Deployment

Correct Answer: B. Data understanding

Question 5

Why should a test dataset generally be protected from repeated model-selection decisions?

A. To increase the number of training records

B. To prevent the final performance estimate from becoming overly optimistic

C. To guarantee that the model is unbiased

D. To eliminate the need for validation data

Correct Answer: B. To prevent the final performance estimate from becoming overly optimistic

Question 6

A classifier has TP = 80 and FN = 20. What is its recall?

A. 20%

B. 50%

C. 80%

D. 100%

Correct Answer: C. 80%

Question 7

Which situation is an example of data leakage?

A. Splitting data by time before model evaluation

B. Using information recorded after the prediction time as a model feature

C. Removing duplicate records before analysis

D. Comparing a model with a simple baseline

Correct Answer: B. Using information recorded after the prediction time as a model feature

Question 8

Which statement about predictive and causal analysis is correct?

A. A variable that improves prediction is always a causal driver

B. Prediction and explanation are identical objectives

C. Predictive association does not by itself establish causation

D. Causal conclusions require no domain knowledge

Correct Answer: C. Predictive association does not by itself establish causation

Question 9

What is the primary purpose of a baseline?

A. To provide a simple reference for judging whether a complex method adds value

B. To guarantee perfect model accuracy

C. To replace data validation

D. To remove the need for deployment monitoring

Correct Answer: A. To provide a simple reference for judging whether a complex method adds value

Question 10

Which activity belongs most directly to the post-deployment lifecycle?

A. Monitoring drift and data quality

B. Defining the initial business objective only

C. Ignoring changes after launch

D. Deleting all validation records

Correct Answer: A. Monitoring drift and data quality