Differential and Integral Calculus

Introduction

Calculus provides the mathematical language for change, approximation, and accumulation. In data science, these ideas appear in optimisation, probability, machine learning, numerical methods, simulation, sensitivity analysis, and the interpretation of time-dependent data. A model may describe how a loss changes when a parameter changes, how a quantity accumulates over time, or how a smooth function can be approximated by a polynomial. Differential calculus studies local change; integral calculus studies accumulation. The two subjects are connected by the Fundamental Theorem of Calculus.

This chapter develops single-variable calculus sufficiently rigorously for postgraduate data-science work, while emphasising interpretation, assumptions, numerical implementation, and limitations.

Learning objectives

By the end of this chapter, you should be able to:

• explain limits and continuity and identify when a function is differentiable;
• calculate and interpret first- and higher-order derivatives;
• use the chain, product, and quotient rules;
• construct Taylor approximations and assess their error;
• evaluate definite and indefinite integrals;
• apply the Fundamental Theorem of Calculus;
• use derivatives and integrals in optimisation, probability, and model analysis;
• recognise numerical, modelling, and interpretive limitations of calculus-based methods.

1. Functions, limits, and continuity

A function assigns each permitted input x an output f(x). The set of permitted inputs is the domain, and the set of possible outputs is the range. In data science, x may represent time, a feature, a model parameter, or a latent quantity. A function may be deterministic even when it is used to describe a system containing random observations.

1.1 The idea of a limit

The limit of f(x) as x approaches a is the value approached by f(x), whether or not f(a) is defined. It is written as lim_{x → a} f(x) = L. More formally, this means that for every ε > 0 there is a δ > 0 such that 0 < |x − a| < δ implies |f(x) − L| < ε.

The formal definition is important because it distinguishes approaching a point from evaluating at the point. For example, if

then factorisation gives f(x) = x + 1 for x ≠ 1. Therefore lim_{x → 1} f(x) = 2, although f(1) is not defined.

A two-sided limit exists only when the left-hand and right-hand limits agree:

Limits may be finite or infinite. A function can satisfy lim_{x → a} f(x) = ∞ when its values grow without bound near a; this describes asymptotic behaviour rather than an ordinary finite function value.

1.2 Limit laws

Provided the relevant limits exist and c is a constant,

For continuous elementary functions, direct substitution is usually valid. Indeterminate forms such as 0/0 require algebraic simplification, factorisation, rationalisation, or—when its conditions hold—L’Hôpital’s rule.

1.3 Continuity

A function f is continuous at a if all three conditions hold:

1. f(a) is defined;
2. lim_{x → a} f(x) exists;
3. lim_{x → a} f(x) = f(a).

A function is continuous on an interval if it is continuous at every point in that interval. Polynomials, exponential functions, and trigonometric functions are continuous on their natural domains. Rational functions are continuous wherever their denominators are non-zero.

Continuity is a modelling assumption. If a small change in an input is expected to cause only a small change in an output, a continuous model may be appropriate. Discontinuities can nevertheless be meaningful: a threshold policy, a system failure, or a classification rule may produce abrupt changes.

The Intermediate Value Theorem states that if f is continuous on [a,b], then every value between f(a) and f(b) is attained at some point in [a,b]. This theorem supports bracketing methods for numerical root finding.

2. Derivatives and local change

The derivative measures instantaneous rate of change. It is defined by

provided the limit exists. The quotient for h ≠ 0 is the average rate of change over an interval of width h; the limit is the slope of the tangent line.

If x has units of seconds and f(x) has units of metres, then f′(x) has units of metres per second. Dimensional interpretation is a useful check on both formulas and conclusions.

A function is differentiable at a if its derivative exists there. Differentiability implies continuity, but continuity does not imply differentiability. For example, f(x) = |x| is continuous at 0 but has different left and right slopes, so it is not differentiable at 0.

2.1 Basic differentiation rules

For differentiable functions f and g and constant c:

Important special cases include

The logarithm formula assumes the ordinary natural logarithm and x > 0. For ln|x on x ≠ 0, the derivative is 1/x.

2.2 The chain rule

If y = f(g(x)), where g is differentiable at x and f is differentiable at g(x), then

The chain rule is central to neural networks and computational graphs. It explains how a change in an early parameter propagates through successive transformations.

Worked example. Let y = ln(1 + x²). The outer function is ln(u), and the inner function is u = 1 + x². Hence

At x = 1, the instantaneous rate is 1. The derivative is dimensionless if x and y are dimensionless; otherwise its units are output units divided by input units.

2.3 Higher-order derivatives

The second derivative f″(x) is the derivative of f′(x). It describes the change in slope, or curvature. In a physical model, if position is s(t), velocity is s′(t), and acceleration is s″(t), subject to consistent time units.

For a twice-differentiable function, f″(x) > 0 indicates local convexity and f″(x) < 0 indicates local concavity. In optimisation, curvature affects both interpretation and algorithmic step selection.

2.4 Critical points and optimisation

A critical point occurs where f′(x) = 0 or where f′(x) does not exist, provided x lies in the domain. For an interior point c:

• if f′ changes from positive to negative, c is a local maximum;
• if f′ changes from negative to positive, c is a local minimum;
• if the sign does not change, the point may be stationary but not an extremum.

The second-derivative test states that if f′(c) = 0 and f″ is continuous near c, then f″(c) > 0 implies a local minimum and f″(c) < 0 implies a local maximum. If f″(c) = 0, the test is inconclusive.

For a closed interval [a,b], a continuous function attains a global maximum and minimum. To find them, compare function values at critical points and endpoints. This endpoint check is essential and is often omitted in informal optimisation.

2.5 Gradient descent

For a differentiable loss function J(θ), where θ is a scalar parameter, gradient descent updates

where η > 0 is the learning rate and k indexes iterations. For a vector parameter θ ∈ ℝᵖ, the derivative becomes the gradient:

This is an iterative numerical method, not a guarantee of global optimisation. Convergence depends on the learning rate, smoothness, curvature, initial value, numerical precision, and the geometry of J. For convex functions, suitable conditions can provide stronger guarantees; for non-convex losses, the method may converge to a local minimum, saddle point, or stationary region.

3. Taylor series and approximation

A Taylor polynomial approximates a sufficiently smooth function near a point a. The degree-n Taylor polynomial is

where f⁽ᵏ⁾ denotes the kth derivative and 0! = 1. The infinite Taylor series is

only where the series converges to f(x). Smoothness alone does not guarantee equality with the Taylor series everywhere.

The remainder in Lagrange form is

for some ξ between a and x, assuming f has n + 1 derivatives on the relevant interval. If |f⁽ⁿ⁺¹⁾(t)| ≤ M there, then

Worked example. For f(x) = eˣ about a = 0,

using degree 2. At x = 0.1, the approximation is 1.105. Since the third derivative of eˣ is eˣ and eˣ ≤ e⁰·¹ on [0,0.1],

The true value is approximately 1.105170, so the observed error is about 0.000170, consistent with the bound.

Taylor methods are used in local sensitivity analysis, approximation of activation and link functions, uncertainty propagation, and derivation of optimisation algorithms. They can be poor approximations far from a and may suffer from cancellation or overflow in finite-precision computation.

4. Integral calculus and accumulation

An indefinite integral represents a family of antiderivatives:

The constant C is necessary because all functions F(x) + C have the same derivative.

A definite integral is a limit of signed sums:

where the interval [a,b] is partitioned into subintervals of widths Δx and xᵢ* is a sample point in subinterval i. If f is non-negative, the integral equals area under the curve. If f takes negative values, the integral is net signed area.

Units multiply: if f is measured in litres per minute and x in minutes, then ∫ f(x) dx is measured in litres.

4.1 Fundamental Theorem of Calculus

The first part states that if f is continuous and

then F is differentiable and F′(x) = f(x). The variable t is a dummy integration variable; using it prevents confusion with the upper limit x.

The second part states that if F is an antiderivative of continuous f on [a,b], then

Together, these results connect accumulation and rate of change.

Worked example. If a data pipeline receives records at rate r(t) = 100 + 20t records per minute for 0 ≤ t ≤ 5, the accumulated records are

The units are records per minute multiplied by minutes, giving records.

4.2 Substitution and integration by parts

For a differentiable substitution u = g(x),

For differentiable functions u and v,

These formulas follow from the chain rule and product rule, respectively. Their use requires that the relevant functions be integrable and that domains be respected.

4.3 Numerical integration

When an antiderivative is unavailable or data are observed only at discrete points, numerical integration is required. The trapezoidal rule with n equal subintervals of width h = (b − a)/n is

For a sufficiently smooth function, its composite error is proportional to h² and depends on the maximum magnitude of f″. Smaller h generally improves accuracy, but increases computation and can amplify measurement noise. Simpson’s rule can achieve higher accuracy under stronger smoothness assumptions and an even number of subintervals.

5. Calculus in probability and data science

If a continuous random variable X has probability density p(x), then

with p(x) ≥ 0 and ∫_−∞^∞ p(x) dx = 1. The cumulative distribution function is

When differentiable, F′(x) = p(x). The expected value of a function g(X), when the integral exists, is

For a joint density p(x,y), marginalisation uses integration:

These formulas assume continuous distributions and proper integrability. Discrete variables require sums rather than integrals; mixed distributions require both components.

In maximum likelihood, a differentiable log-likelihood ℓ(θ) is often maximised by solving ℓ′(θ) = 0 or by iterative optimisation. The Hessian, the matrix of second partial derivatives, describes local curvature in multiple dimensions. A negative-definite Hessian at a stationary point indicates a strict local maximum under standard differentiability assumptions.

6. Multivariable perspective

Although this chapter focuses on single-variable calculus, data-science models typically involve many parameters. For f: ℝᵖ → ℝ, the gradient is

The directional derivative in unit direction u is

The gradient points in the direction of greatest local increase, and −∇f points in the direction of greatest local decrease. The first-order approximation is

The second-order approximation is

where H(θ) is the Hessian. These approximations are local and become unreliable when Δθ is large or the function is highly nonlinear.

7. Illustrative case: optimising a regularised prediction model

Background. Consider a model with one parameter w and loss

The first term measures deviation from a target-like value 3; the second is an illustrative L₂ regularisation penalty that discourages large parameters.

Problem. Find the parameter minimising J and interpret the effect of regularisation.

Analysis. Differentiate:

Set the derivative to zero:

The second derivative is J″(w) = 2.4 > 0, so J is strictly convex and w = 2.5 is the unique global minimiser. Without regularisation, minimising (w − 3)² gives w = 3. Thus the penalty shrinks the estimate toward zero.

The minimum loss is

Gradient descent with learning rate η = 0.1 and initial w₀ = 0 gives

The iterates move toward 2.5. Because the objective is quadratic and strictly convex, a sufficiently small constant learning rate produces convergence; excessively large rates can oscillate or diverge.

Outcome and learning points. Derivatives identify the optimum, curvature establishes its nature, and an iterative method approaches it numerically. Regularisation changes the objective, not merely the computational procedure. In a real prediction system, the regularisation coefficient would be selected using validation data, and feature scaling would be important because penalty effects depend on parameterisation.

8. Limitations, responsible use, and best practice

Calculus-based analysis depends on assumptions. A derivative may not exist at a discontinuity, cusp, or noisy empirical curve. Numerical derivatives amplify measurement noise, particularly when finite differences subtract nearly equal values. Smoothing can help but may erase genuine sharp changes. Integration of sampled data inherits errors from sampling, interpolation, missingness, and boundary choices.

In machine learning, differentiability is not equivalent to fairness, validity, or causal meaning. An optimiser can efficiently minimise a biased objective. Automated hyperparameter tuning may optimise a proxy that conflicts with social or operational goals. Practitioners should document the objective, data provenance, units, scaling, differentiability assumptions, numerical tolerances, stopping criteria, and validation procedure.

Best practice includes:

• check domains and units before differentiating or integrating;
• distinguish local from global conclusions;
• inspect endpoints and constraints in optimisation;
• report approximation error or numerical tolerance;
• use analytical derivatives when reliable and automatic differentiation when appropriate;
• verify gradients with finite-difference checks on small test cases;
• assess sensitivity to step size, regularisation, and initialisation;
• preserve uncertainty and avoid presenting a smooth model as unquestionable reality.

Chapter Summary

Limits formalise approach and support the definitions of continuity and derivatives. Continuity describes stable variation, while differentiability provides an instantaneous rate of change and implies continuity. Product, quotient, and chain rules permit systematic differentiation, and higher derivatives describe curvature. Critical-point analysis and gradient descent connect derivatives to optimisation, although numerical convergence and global optimality require conditions.

Taylor polynomials provide local approximations, with remainder bounds that quantify error under smoothness assumptions. Integrals represent accumulation and signed area; substitution and integration by parts transform integrals, while numerical rules approximate them when analytic antiderivatives or complete functions are unavailable. The Fundamental Theorem of Calculus establishes the deep relationship between differentiation and integration. In data science, these ideas support probability densities, expectations, likelihood optimisation, regularisation, sensitivity analysis, and numerical computation. Responsible use requires attention to domains, units, smoothness, noise, approximation error, and the difference between mathematical optimisation and meaningful decision-making.

Key Terms

Limit: The value a function approaches as its input approaches a specified point.

Continuity: The property that a function’s limit at a point equals its defined value there.

Derivative: The instantaneous rate of change or tangent slope of a function.

Differentiability: Existence of a finite derivative at a point or throughout an interval.

Chain rule: Rule for differentiating a composition of functions.

Gradient: Vector of first partial derivatives of a scalar-valued multivariable function.

Hessian: Matrix of second partial derivatives.

Critical point: Interior point where the derivative is zero or does not exist.

Convexity: A shape property associated with nonnegative second derivative in one dimension.

Taylor polynomial: A finite polynomial formed from derivatives at an expansion point.

Remainder: The error left after truncating a Taylor series.

Antiderivative: A function whose derivative equals a specified integrand.

Definite integral: The limiting signed accumulation over an interval.

Numerical integration: Approximation of an integral using sampled function values.

Fundamental Theorem of Calculus: Results connecting derivatives and definite integrals.

Probability density: Nonnegative function whose integral over a region gives probability.

Regularisation: Addition of a penalty to an objective to control model complexity.

Learning rate: Step-size parameter controlling an iterative optimisation update.

Sensitivity analysis: Study of how outputs respond to changes in inputs or parameters.

Review Questions

Short Answer Questions

1. What is the difference between a function’s value at a point and its limit at that point?
2. Why does differentiability imply continuity, but continuity not imply differentiability?
3. State the chain rule and identify its two factors.
4. What does the second derivative indicate about local curvature?
5. Distinguish an indefinite integral from a definite integral.

Long Answer Questions

1. Explain the derivative definition from first principles and interpret the units of a derivative in an applied model.
2. Derive and explain the product rule, and illustrate its use with a suitable example.
3. Explain Taylor approximation, including the role of the expansion point and remainder term.
4. State both parts of the Fundamental Theorem of Calculus and explain their relationship.
5. Discuss how gradients, Hessians, and regularisation influence optimisation in machine learning.

Analytical / Application Questions

1. A continuous cost function is defined on a closed interval. Describe a complete procedure for finding its global minimum and explain why endpoint checks are necessary.
2. A sensor records a rate in units per minute at equally spaced times. Design a numerical integration procedure to estimate the accumulated quantity and identify at least three sources of error.
3. For a differentiable loss function, compare one gradient-descent update with a second-order Taylor-based update. Discuss the benefits, computational costs, and risks of each approach.

Multiple Choice Questions with Answers

Question 1

Which condition is required for f to be continuous at a?

A. f′(a) must be zero

B. f(a) must be undefined

C. lim_{x → a} f(x) = f(a)

D. f must be a polynomial

Correct Answer: C. lim_{x → a} f(x) = f(a)

Question 2

What is the derivative of y = ln(1 + x²)?

A. 2x

B. 1/(1 + x²)

C. 2x/(1 + x²)

D. ln(2x)

Correct Answer: C. 2x/(1 + x²)

Question 3

If f′(c) = 0 and f″(c) > 0 under the second-derivative test, what does c represent?

A. A local maximum

B. A local minimum

C. A discontinuity

D. An inflection point necessarily

Correct Answer: B. A local minimum

Question 4

What is the degree-2 Taylor approximation of eˣ about 0?

A. 1 + x

B. x + x²

C. 1 + x + x²/2

D. e + x²/2

Correct Answer: C. 1 + x + x²/2

Question 5

A rate is r(t) = 100 + 20t records per minute for 0 ≤ t ≤ 5. How many records accumulate?

A. 500

B. 600

C. 700

D. 750

Correct Answer: D. 750

Question 6

Which statement is a form of the Fundamental Theorem of Calculus?

A. The derivative of an accumulated integral recovers the integrand

B. Every continuous function is linear

C. Every integral is zero at an endpoint

D. Every derivative is constant

Correct Answer: A. The derivative of an accumulated integral recovers the integrand

Question 7

For gradient descent on J(θ), which update is correct?

A. θ_{k+1} = θ_k + η∇J(θ_k)

B. θ_{k+1} = θ_k − η∇J(θ_k)

C. θ_{k+1} = η/J(θ_k)

D. θ_{k+1} = θ_k − J(θ_k)

Correct Answer: B. θ_{k+1} = θ_k − η∇J(θ_k)

Question 8

If X has density p(x), which expression gives P(a ≤ X ≤ b)?

A. p(a) + p(b)

B. ∫ₐᵇ p(x) dx

C. p(b) − p(a)

D. ∫₋∞ᵃ p(x) dx

Correct Answer: B. ∫ₐᵇ p(x) dx

Question 9

For J(w) = (w − 3)² + 0.2w², what is the minimising value of w?

A. 0

B. 2

C. 2.5

D. 3

Correct Answer: C. 2.5

Question 10

What is the principal limitation of a Taylor polynomial?

A. It cannot use derivatives

B. It is always exact globally

C. It is generally a local approximation whose accuracy depends on distance and smoothness

D. It applies only to discontinuous functions

Correct Answer: C. It is generally a local approximation whose accuracy depends on distance and smoothness