Data Science Foundations and Professional Practice

Introduction

Data science is the disciplined use of data, statistical reasoning, computational methods, domain knowledge, and communication to produce reliable insight and support decisions. It is not simply programming, dashboard construction, or the application of machine-learning algorithms. A data-science project connects a real-world question to data, transforms observations into evidence, evaluates uncertainty and risk, and communicates an action that can be justified.

The field has developed from several traditions: statistics, scientific computing, operations research, databases, business intelligence, and artificial intelligence. Its modern form is enabled by inexpensive storage, cloud computing, distributed systems, open-source software, and widespread data collection. Yet technical capability does not guarantee useful or responsible results. A model may be accurate but operationally unusable; a dashboard may be attractive but misleading; and a large dataset may contain systematic bias rather than objective truth.

This chapter establishes the conceptual and professional foundations required for postgraduate study in artificial intelligence and machine learning. It explains the relationships among data science, statistics, analytics, AI, machine learning, business intelligence, and data engineering, and introduces a reproducible workflow using Python or R, notebooks, version control, and documented computational environments.

Learning Objectives

By the end of this chapter, you should be able to:

1. Define data science and distinguish it from related disciplines.
2. Explain descriptive, diagnostic, predictive, and prescriptive analytics.
3. Relate statistics, artificial intelligence, machine learning, business intelligence, and data engineering.
4. Identify common professional roles and organize work across the data lifecycle.
5. Design a reproducible workflow using Python or R, notebooks, Git, tests, and documentation.
6. Evaluate data-science projects in terms of validity, usefulness, fairness, privacy, and operational risk.

1. What Is Data Science?

A useful definition of data science is: an interdisciplinary practice for collecting, preparing, analysing, modelling, evaluating, and communicating data-derived knowledge for a specified purpose. The phrase “specified purpose” is important. Data analysis is not automatically valuable; value depends on the decision, user, constraints, and consequences associated with the analysis.

Data science usually combines five forms of competence:

• Substantive knowledge: understanding the domain, such as finance, health, manufacturing, or public policy.
• Statistical reasoning: distinguishing signal from noise and quantifying uncertainty.
• Computational practice: writing reliable code and working with scalable systems.
• Data management: acquiring, integrating, documenting, securing, and governing data.
• Communication and decision analysis: translating evidence into an understandable and actionable recommendation.

A data product may be a report, a forecasting service, a recommendation system, a fraud-alert workflow, an experiment, or a decision-support dashboard. The output is not necessarily a model. In many projects, improved data quality, a well-designed metric, or a carefully conducted experiment creates more value than a sophisticated algorithm.

Data, information, evidence, and knowledge

These terms should not be treated as synonyms. Data are recorded observations or representations, such as a transaction amount, image, sensor reading, or survey response. Information is data organized in a meaningful context. Evidence is information assessed for relevance, credibility, and strength in relation to a claim. Knowledge includes justified understanding that supports explanation, prediction, or action.

For example, a table of customer cancellations is data. A monthly cancellation rate by product is information. Evidence that cancellations increased after a price change requires appropriate comparison, measurement validity, and consideration of alternative explanations. Knowledge may then support a decision about retention policy.

A common error is to assume that more data automatically produces stronger evidence. Measurement error, selection bias, missingness, duplicated records, and changing definitions can make a large dataset misleading.

2. The Data-Science Lifecycle

A lifecycle is a framework rather than a rigid sequence. Projects often move backward when analysis reveals that the original question is poorly defined or that the available data cannot support it.

1. Problem formulation: identify the decision, stakeholders, constraints, and success criteria.
2. Data acquisition: locate, collect, or generate relevant observations.
3. Data understanding: inspect structure, provenance, quality, missingness, and potential bias.
4. Preparation: clean, join, transform, label, and partition data without contaminating evaluation sets.
5. Exploration: examine distributions, relationships, anomalies, and plausible mechanisms.
6. Modelling or analysis: apply statistical, computational, or machine-learning methods.
7. Evaluation: assess technical performance, uncertainty, robustness, fairness, and business or social utility.
8. Communication: explain assumptions, limitations, findings, and recommended actions.
9. Deployment and monitoring: integrate the result into practice and detect data drift, performance degradation, and unintended effects.
10. Retirement or revision: remove, retrain, or redesign systems when their purpose or performance changes.

The lifecycle resembles a feedback loop. Deployment creates new observations, user responses, and failure cases, which can inform subsequent analysis. This also creates a risk: a model may influence the behaviour it is intended to measure. For example, a fraud model changes which transactions are investigated, so observed fraud rates after deployment may no longer represent the original population.

3. Related Disciplines and Their Relationships

Statistics

Statistics provides principles for describing populations and samples, estimating quantities, testing claims, designing experiments, and quantifying uncertainty. A sample mean is commonly written as:

Here, xᵢ is observation i, n is the number of observations, and x̄ is the sample mean. The formula assumes that the observations are numeric and that the arithmetic mean is meaningful for the measurement scale. It does not, by itself, establish causation or representativeness.

The sample variance is:

The denominator n − 1 is used for the usual unbiased estimator of population variance under independent random sampling from a population with finite variance. Other estimators and denominators may be appropriate in different settings. Standard deviation is s = √s², expressed in the same units as the observations.

Statistics is concerned not only with computation but also with study design. Randomization, control groups, sampling strategies, confidence intervals, and assumptions determine whether a conclusion is defensible.

Analytics

Analytics is the systematic use of data to understand situations and support decisions. A practical classification is:

These categories overlap. A predictive model can support a prescriptive decision, but prediction alone does not determine the best action. Action also depends on costs, constraints, preferences, and ethical considerations.

Artificial intelligence and machine learning

Artificial intelligence is a broad term for computational systems designed to perform tasks associated with capabilities such as perception, reasoning, planning, language use, or decision-making. Machine learning is a family of methods in which a system infers patterns or functions from data rather than being programmed entirely through explicit rules.

Machine learning includes supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and representation learning. In supervised learning, a model is fitted using examples with target values. A generic prediction function can be written as:

where x is an input vector, ŷ is the prediction, fθ is a function parameterized by θ, and the function is learned from training data. The notation does not imply that the prediction is causal or certain.

A loss function measures disagreement between predictions and observed targets. For a regression problem, mean squared error is:

where yᵢ is the observed target and ŷᵢ is the prediction. MSE has squared target units, is sensitive to large errors, and is appropriate only when squared-error penalties match the application. Evaluation must be performed on data not used to fit the model, and the split must reflect the intended deployment setting.

AI and machine learning are therefore related to data science but are not equivalent to it. Data science may use machine learning, but it also includes data quality assessment, experimentation, statistical inference, governance, and communication. Conversely, machine learning can be studied as an algorithmic discipline even when no broader decision process is considered.

Business intelligence

Business intelligence (BI) traditionally emphasizes structured organizational data, standard reports, dashboards, key performance indicators, and controlled access to information. BI often answers recurring descriptive and diagnostic questions. Data science tends to address more exploratory, predictive, experimental, or automated problems, although modern platforms increasingly combine both.

A metric must have a clear definition, numerator, denominator, time window, population, and data source. For example, a conversion rate may be written as:

The denominator must be defined carefully. Dividing conversions by all website visits is not equivalent to dividing by qualified leads. A metric can be mathematically correct yet conceptually invalid if its population is poorly specified.

Data engineering

Data engineering develops and operates the infrastructure through which data is collected, stored, transformed, accessed, and monitored. Responsibilities include database design, extraction and loading, batch and streaming pipelines, data warehouses, data lakes, metadata, orchestration, reliability, and access control.

A data scientist may create a transformation in a notebook, but production use requires repeatable execution, dependency management, logging, testing, monitoring, and clear ownership. Data engineering therefore provides the dependable foundations on which analytics and machine learning rely.

4. Professional Roles and Collaboration

Organizations use different titles, but common roles include:

These boundaries are porous. Effective teams make responsibilities explicit through project charters, data dictionaries, model cards, review procedures, and service ownership. A technically excellent model can fail if no one is responsible for responding to its output or maintaining its data pipeline.

5. Python, R, and Computational Notebooks

Python and R are both suitable for postgraduate data-science work. Python has a broad software-engineering ecosystem and is widely used in automation, machine learning, and production services. R has exceptional strengths in statistical modelling, visualization, and research communication. The choice should reflect the task, team capability, available libraries, deployment environment, and reproducibility requirements rather than fashion.

A notebook combines executable code, output, narrative text, equations, and visualizations. This makes it valuable for exploration and teaching, but notebooks also introduce hazards:

• cells can be executed out of order;
• hidden state can make results difficult to reproduce;
• large outputs can obscure important findings;
• credentials may be accidentally stored in the file;
• unreviewed exploratory code may be mistaken for production code.

A professional notebook should state its purpose, inputs, assumptions, software environment, execution order, random seeds where appropriate, and final conclusions. It should be possible for another person to run it from a clean environment. Notebooks should generally be complemented by modules, tests, configuration files, and automated pipelines for production systems.

6. Git and Reproducible Workflows

Git is a distributed version-control system. It records changes to code and text files, supports branching and merging, and provides a history of authorship and revisions. It does not automatically version large datasets, guarantee correct analysis, or protect sensitive information.

A basic workflow is:

1. Create a repository with a clear README.
2. Work on a branch or a small, focused change.
3. Review the difference between the working files and the previous commit.
4. Commit a coherent change with an informative message.
5. Merge through review when collaboration or risk warrants it.
6. Tag releases and record the corresponding data and environment versions.

Sensitive data, passwords, access tokens, and personally identifiable information should not be committed. A .gitignore file reduces accidental inclusion but is not a substitute for security controls.

Reproducibility has several levels:

• Computational reproducibility: the same code and inputs produce the same result.
• Statistical reproducibility: results remain stable under reasonable resampling, modelling choices, or new samples.
• Conceptual reproducibility: an independent study using a suitable design reaches compatible conclusions.
• Operational reproducibility: the method continues to work in the deployment environment over time.

A useful project structure may contain README.md, data/ or data-access instructions, src/, notebooks/, tests/, reports/, environment specifications, and configuration files. Raw data should be preserved as read-only when permitted, while derived data should have documented transformations. A data dictionary should define field meanings, units, allowed values, missing-value conventions, and provenance.

Reproducibility also requires deterministic decisions where possible. Random algorithms should use recorded seeds for debugging and comparison, but a seed does not eliminate uncertainty or guarantee identical results across hardware, libraries, or versions. Record package versions and environment specifications, and distinguish exploratory results from formally evaluated results.

7. Data Quality, Validity, and Uncertainty

Important dimensions of data quality include accuracy, completeness, consistency, timeliness, uniqueness, validity, and fitness for purpose. Quality is contextual: a sensor reading may be timely but noisy; a historical record may be accurate but unsuitable for current prediction because the process has changed.

Missing data should be investigated rather than automatically replaced. Missingness may be completely at random, related to observed variables, or related to the unobserved value itself. Imputation can be useful, but it introduces assumptions and should be performed within the training procedure when evaluating predictive models. Otherwise, information from the evaluation set may leak into model development.

Data leakage occurs when information unavailable at prediction time enters the features or preprocessing procedure. Examples include using a future outcome, calculating a normalization value from the entire dataset before splitting, or including a post-treatment variable in a causal analysis. Leakage often produces impressive validation results and disappointing deployment performance.

Uncertainty should be communicated. A point estimate, such as a forecast of 10,000 units, may conceal sampling uncertainty, model uncertainty, and future variability. A prediction interval or scenario range may be more useful, provided its interpretation and coverage assumptions are explained.

8. Ethics, Governance, and Responsible Practice

Data science can affect employment, credit, healthcare, education, safety, and access to public services. Professional practice therefore requires more than compliance with technical specifications.

Key principles include:

• Purpose limitation: collect and use data for a defined, legitimate purpose.
• Data minimization: avoid collecting information that is unnecessary for that purpose.
• Transparency: document sources, methods, assumptions, and significant limitations.
• Privacy and security: control access, protect sensitive data, and manage retention.
• Fairness: evaluate whether errors and benefits are distributed unjustly across groups.
• Accountability: identify owners, review mechanisms, appeal channels, and responsibilities.
• Human oversight: ensure that consequential decisions are not treated as unquestionable outputs.

Fairness cannot be reduced to a single universal metric. For a binary decision, one may compare group-specific false-positive rates, false-negative rates, selection rates, or calibration. These criteria can conflict, especially when groups have different base rates. The appropriate evaluation depends on the decision context, legal requirements, stakeholder values, and potential harms.

Privacy-preserving methods include access controls, aggregation, pseudonymization, differential privacy, and secure computation. Pseudonymization is not the same as anonymization: auxiliary information may permit re-identification. Governance should address the entire lifecycle, including data acquisition, model development, deployment, monitoring, incident response, and retirement.

9. Illustrative Case: Predicting Hospital Readmission

Background

This fictional case illustrates a realistic data-science workflow. A hospital wants to identify patients who may need additional support after discharge. The goal is not to deny care but to prioritize follow-up resources.

Problem

A team initially proposes a machine-learning model predicting readmission within 30 days. The clinical director clarifies that the operational decision is whether a patient should receive a follow-up call within 48 hours. The relevant success criteria therefore include useful sensitivity, manageable workload, clinical acceptability, and equitable performance—not merely an abstract accuracy score.

Analysis

The team audits three years of admission data. They find inconsistent diagnosis codes, missing socioeconomic variables, changes in discharge policy, and a feature recording whether a post-discharge call was completed. The last feature is excluded because it is unavailable at the time of discharge and may represent an intervention rather than a predictor. The team also discovers that readmission records from another hospital are not consistently captured, so the outcome is incomplete.

A time-based split is selected: earlier admissions are used for development and later admissions for evaluation. This better represents deployment than a random split if clinical practice changes over time. The team reports calibration, sensitivity, specificity, positive predictive value, workload at different thresholds, and performance across relevant patient groups.

Approach and solution

The team begins with a transparent baseline: a regularized logistic-regression model and a rule-based clinical score. More complex models are evaluated only if they produce meaningful improvement. Features are created using information available before discharge, and preprocessing is fitted only on the training period. A model card documents intended use, excluded uses, data limitations, evaluation population, and known risks.

The hospital does not automate the final decision. Instead, the model produces a risk category and a short explanation for a trained coordinator, who can override it and record a reason. Monitoring checks calibration, missingness, population change, override rates, and follow-up outcomes.

Outcome

The model improves the identification of patients who benefit from follow-up compared with the existing process, but its performance declines after a major change in discharge policy. Monitoring detects the shift, and the model is retrained after clinical review. The project is judged successful because it improves a defined care process while preserving human accountability—not because it achieves the highest benchmark score.

Learning points

1. Define the operational decision before choosing an algorithm.
2. Exclude features unavailable at decision time.
3. Treat outcome quality and missingness as modelling issues.
4. Use time-aware evaluation when processes evolve.
5. Evaluate utility, calibration, fairness, and workload alongside predictive metrics.
6. Design monitoring and accountability before deployment.

10. Contemporary Developments and Best Practices

Modern data science increasingly involves cloud platforms, data contracts, feature stores, large language models, automated machine learning, and responsible-AI documentation. These tools can increase productivity but also increase dependency, cost, attack surface, and governance complexity. Generative AI systems may produce fluent but unsupported content; their outputs require verification, provenance controls, privacy safeguards, and domain review.

A mature professional practice follows several principles:

• Begin with a decision and measurable success criterion.
• Establish data provenance and a shared vocabulary.
• Build a simple, interpretable baseline before increasing complexity.
• Separate training, validation, and final evaluation data.
• Test transformations and critical calculations.
• Review code, assumptions, and visualizations.
• Record environments, versions, seeds, and configuration.
• Report uncertainty and limitations honestly.
• Evaluate consequences for different groups and failure modes.
• Deploy incrementally with monitoring, rollback, and ownership.
• Prefer the simplest system that meets the requirement.

The most valuable data scientist is not the person who applies the most advanced method, but the professional who can connect a sound method to a meaningful problem and explain what the evidence does and does not justify.

Chapter Summary

Data science is an interdisciplinary practice that turns data into justified insight and action. It draws on statistics, computation, domain knowledge, data engineering, communication, and decision analysis. Statistics supplies principles for measurement, inference, uncertainty, and study design; analytics organizes questions about what happened, why, what may happen, and what should be done; AI is a broad field concerned with intelligent computational behaviour; machine learning learns patterns from data; BI emphasizes governed reporting and performance information; and data engineering builds reliable data infrastructure.

A data-science lifecycle includes problem formulation, acquisition, understanding, preparation, exploration, modelling, evaluation, communication, deployment, monitoring, and revision. Python or R, notebooks, Git, tests, environment specifications, documentation, and data provenance support reproducibility. Professional practice must also address leakage, missingness, uncertainty, privacy, fairness, security, accountability, and changing deployment conditions. The central lesson is that technical performance is only one component of a successful and responsible data product.

Key Terms

Data science: Interdisciplinary practice for producing and using data-derived knowledge.

Statistics: Discipline concerned with data description, inference, uncertainty, and study design.

Analytics: Use of data to understand situations and support decisions.

Artificial intelligence: Broad field of computational systems performing tasks associated with intelligent behaviour.

Machine learning: Methods that infer patterns or functions from data.

Business intelligence: Governed reporting, dashboards, metrics, and analysis for organizational decisions.

Data engineering: Design and operation of systems that collect, store, transform, and serve data.

Data provenance: Record of where data came from and how it was transformed.

Data leakage: Use of information in development that would be unavailable at the intended decision time.

Reproducibility: Ability to obtain dependable results through recorded methods, inputs, and environments.

Notebook: Interactive document combining code, narrative, outputs, and visualizations.

Version control: System for tracking and managing changes to files and code.

Supervised learning: Learning from examples with known target values.

Uncertainty: Lack of complete certainty about measurements, estimates, predictions, or conclusions.

Calibration: Agreement between predicted probabilities and observed frequencies.

Fairness: Evaluation and management of whether outcomes or errors are distributed unjustly.

Data governance: Policies and responsibilities for data quality, access, security, privacy, and use.

Concept drift: Change over time in the relationship between inputs, outcomes, or the data-generating process.

Model card: Document describing a model’s purpose, performance, limitations, and risks.

Data dictionary: Reference defining fields, meanings, units, values, and missingness conventions.

Review Questions

5 Short Answer Questions

1. Define data science and identify four capabilities it combines.
2. Distinguish descriptive, diagnostic, predictive, and prescriptive analytics.
3. Explain one important difference between machine learning and data science.
4. Why can a notebook be both useful and difficult to reproduce?
5. What is data leakage, and why does it produce misleading evaluation results?

5 Long Answer Questions

1. Compare statistics, analytics, AI, machine learning, BI, and data engineering, showing how they contribute to one project.
2. Explain the data-science lifecycle and discuss why projects may move backward between stages.
3. Develop a professional reproducibility plan for a Python- or R-based analytical project.
4. Discuss how data quality, missingness, uncertainty, and changing environments affect model validity.
5. Explain the ethical and governance responsibilities involved in deploying a model for consequential decisions.

3 Analytical / Application Questions

1. A retailer reports a rising conversion rate after changing the denominator from all visitors to qualified leads. Analyse why the metric increased and propose a valid reporting specification.
2. Design an evaluation strategy for a demand-forecasting model when historical data span several years and pricing policy changed midway through the period.
3. A hospital model performs well overall but has substantially higher false-negative rates for one patient group. Propose a structured investigation and recommend what evidence should be considered before deployment.

Multiple Choice Questions with Answers

Question 1

Which statement best describes data science?

A. The exclusive use of machine-learning algorithms to automate decisions

B. The interdisciplinary production and use of data-derived knowledge for a specified purpose

C. The preparation of financial statements from organizational databases

D. The storage of all organizational data in a cloud platform

Correct Answer: B. The interdisciplinary production and use of data-derived knowledge for a specified purpose

Question 2

Which type of analytics primarily asks what is likely to happen?

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: C. Predictive analytics

Question 3

What is the primary responsibility of data engineering?

A. Designing advertisements from customer segments

B. Operating systems that collect, transform, store, and serve data

C. Replacing all statistical analysis with neural networks

D. Writing only narrative reports for executives

Correct Answer: B. Operating systems that collect, transform, store, and serve data

Question 4

Why is a time-based split often appropriate when evaluating a clinical prediction model?

A. It guarantees perfect predictions

B. It prevents the need for missing-data analysis

C. It better represents deployment when clinical practice changes over time

D. It makes the model causal by definition

Correct Answer: C. It better represents deployment when clinical practice changes over time

Question 5

In the formula x̄ = (1/n)∑xᵢ, what does n represent?

A. The sample size

B. The population mean

C. The largest observation

D. The measurement unit

Correct Answer: A. The sample size

Question 6

Which practice most directly reduces accidental exposure of passwords in a Git repository?

A. Committing credentials in a README file

B. Using access controls and excluding secrets from committed files

C. Renaming the password variable after committing it

D. Increasing notebook output size

Correct Answer: B. Using access controls and excluding secrets from committed files

Question 7

What is data leakage?

A. A permanent loss of all records from a database

B. Use of information during development that would be unavailable at decision time

C. A disagreement between two software developers

D. A method for anonymizing every dataset

Correct Answer: B. Use of information during development that would be unavailable at decision time

Question 8

Why is a model’s accuracy alone insufficient for evaluating a consequential application?

A. Accuracy cannot be calculated for any dataset

B. Accuracy does not address uncertainty, subgroup effects, costs, or operational usefulness

C. Accuracy is always lower than the sample mean

D. Accuracy applies only to descriptive dashboards

Correct Answer: B. Accuracy does not address uncertainty, subgroup effects, costs, or operational usefulness

Question 9

Which document commonly describes a model’s intended use, performance, limitations, and risks?

A. Data dictionary

B. Model card

C. Git ignore file

D. Database index

Correct Answer: B. Model card

Question 10

A conversion rate is calculated as 240 conversions divided by 3,000 eligible opportunities. What is the rate?

A. 0.8%

B. 8%

C. 12.5%

D. 80%

Correct Answer: B. 8%