Exceptions, Debugging, Testing and Logging

Introduction

Reliable Python programs do more than produce correct results under ideal conditions. They must respond sensibly to invalid input, missing files, network failures, incorrect assumptions, and defects introduced during maintenance. In data science, this requirement is especially important because programs often process large datasets, depend on external services, and support decisions that may have financial, scientific, or social consequences.

This chapter presents a systematic approach to reliability. Exceptions provide a controlled mechanism for responding to run-time problems. Assertions express assumptions that should hold during development. Debuggers help locate defects interactively, while logging creates a persistent account of program behaviour. Testing provides repeatable evidence that software behaves as intended. Validation and defensive programming reduce preventable failures, and code-quality conventions make defects easier to detect and correct.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish syntax errors, exceptions, and logical errors;
• raise, handle, chain, and design appropriate exceptions;
• use assertions appropriately and explain their limitations;
• diagnose programs with tracing, breakpoints, and a debugger;
• configure useful and safe logging;
• write unit, integration, and regression tests with unittest or pytest;
• apply validation, defensive programming, and code-quality conventions;
• evaluate reliability, maintainability, and ethical risks in data-processing software.

1. The Nature of Program Failure

A syntax error violates Python's grammar. It is normally detected before execution begins:

for item in values
    print(item)

An exception is an event raised while a syntactically valid program is running. Examples include ValueError when a string cannot be converted to a number, KeyError when a dictionary key is absent, and FileNotFoundError when a file does not exist.

A logical error produces an incorrect result without necessarily raising an exception. For example, using total / (n - 1) when the intended denominator is n may execute successfully but calculate the wrong statistic.

The distinction matters because each failure requires a different response:

A traceback is not merely an error message. It records the exception type, message, and sequence of calls leading to the failure. Read it from the final line upward: the final line identifies the immediate exception, while earlier lines identify the execution path.

2. Exception Handling

Python uses try, except, else, and finally blocks:

try:
    amount = float(text)
except ValueError:
    print("Please enter a valid number.")
else:
    print(f"Accepted amount: {amount}")
finally:
    print("Input attempt completed.")

The try block contains operations that may fail. An except block handles a matching exception. The else block runs only when no exception occurs, and finally runs whether an exception occurs or not. The latter is useful for cleanup, although context managers are usually preferable for files and similar resources.

2.1 Catch specific exceptions

Catch the narrowest meaningful exception. This is preferable:

try:
    age = int(value)
except ValueError:
    age = None

This is dangerous:

try:
    age = int(value)
except Exception:
    age = None

A broad handler may conceal programming defects such as NameError or AttributeError. Catching BaseException is generally inappropriate because it also includes system-exiting events such as KeyboardInterrupt.

An exception can be given a name:

try:
    result = 10 / denominator
except ZeroDivisionError as error:
    print(f"Invalid denominator: {error}")

Multiple exception types may be handled together:

try:
    record = records[index]
except (IndexError, TypeError):
    record = None

2.2 Raising exceptions

A function should raise an exception when it cannot fulfil its contract:

def percentage(part, whole):
    if whole == 0:
        raise ValueError("whole must not be zero")
    return 100 * part / whole

ValueError is appropriate when the type is valid but the value is unacceptable. TypeError is appropriate when an operation receives an inappropriate type. FileNotFoundError communicates a specific file problem, while RuntimeError is a more general choice when no more precise built-in exception applies.

Custom exceptions clarify domain-specific failures:

class InvalidTransactionError(ValueError):
    """Raised when a transaction violates business rules."""


def validate_transaction(transaction):
    if transaction["amount"] < 0:
        raise InvalidTransactionError("amount cannot be negative")

Custom exceptions should be few, meaningful, and documented. They allow callers to distinguish expected domain failures from unrelated programming defects.

2.3 Exception propagation and chaining

If a function does not handle an exception, it propagates to its caller. This is useful when the caller has better information about recovery or user communication.

When translating a low-level error into a domain-level error, preserve the cause:

def load_config(path):
    try:
        with open(path, encoding="utf-8") as file:
            return file.read()
    except OSError as error:
        raise RuntimeError(f"Could not load configuration: {path}") from error

The from error clause creates an explicit exception chain. It preserves diagnostic information while presenting a more meaningful interface to higher-level code.

2.4 Resource management

Use a context manager whenever an object must be closed or released:

with open("data.csv", encoding="utf-8") as file:
    contents = file.read()

The file is closed even if reading raises an exception. The same principle applies to database connections, locks, and other resources that implement the context-manager protocol.

3. Assertions and Defensive Programming

An assertion expresses a condition that the programmer believes must be true:

def mean(values):
    assert values is not None, "values must not be None"
    if not values:
        raise ValueError("values must not be empty")
    return sum(values) / len(values)

Assertions are valuable for internal invariants, impossible states, and development-time checks. They are not a substitute for validation of user input or external data. Python can disable assertions when run with optimisation, so essential checks must use explicit exceptions and conditional statements.

if password == "":
    raise ValueError("password must not be empty")

Defensive programming assumes that inputs, services, and environmental conditions may be imperfect. Its practices include checking types and ranges, using timeouts, limiting resource consumption, validating file paths, and returning clear errors. Defensive programming should not mean silently accepting every value. It should define an explicit policy: reject, transform, default, retry, or escalate.

A useful validation sequence is:

1. Check presence: is the required value available?
2. Check type and structure: is it a string, number, mapping, or expected record?
3. Check range and domain: is it non-negative, within bounds, or a permitted category?
4. Normalize deliberately: for example, remove surrounding whitespace.
5. Report the problem with actionable information.

Validation of external data must not rely solely on client-side checks. A web form may be bypassed, a CSV may be edited, and an API may return malformed data. Important validation belongs at trust boundaries and should be repeated where necessary.

4. Debugging as a Systematic Process

Debugging is the disciplined process of locating and correcting the cause of incorrect behaviour. It is not random alteration of code. A practical cycle is:

1. Reproduce the failure reliably.
2. Record the input, environment, and expected result.
3. Form a specific hypothesis.
4. Collect evidence with tests, logs, or a debugger.
5. Make the smallest justified change.
6. Run focused and then broader tests.
7. Record the cause and prevention strategy.

4.1 Tracing and instrumentation

Temporary print() statements can reveal control flow, but structured logging is more useful for maintained software. A debugger permits execution to pause at a breakpoint, inspect variables, evaluate expressions, and step through lines or function calls.

The Python debugger can be started from code:

import pdb

pdb.set_trace()

At the prompt, common commands include n (next line), s (step into a function), c (continue), p expression (print an expression), and q (quit). Integrated development environments provide equivalent features through graphical controls.

A breakpoint should be placed near the first point where observed state differs from expected state. Inspecting only the final failure may identify a symptom rather than the cause.

4.2 Debugging data-processing programs

For data-science code, inspect shape, schema, data types, missing-value counts, ranges, and representative records. A model failure may originate in a previous transformation, such as a date parsed with the wrong locale or a target column accidentally included among predictors. Preserve a small failing dataset so that the defect can be reproduced quickly.

5. Logging

Logging provides a time-stamped, severity-labelled record of events. Unlike ad hoc output, it can be filtered, redirected, formatted, and centrally collected.

Python defines standard levels:

Basic configuration:

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s"
)
logger = logging.getLogger(__name__)

logger.info("Starting import")
logger.warning("Skipped %d invalid rows", skipped)

Use lazy formatting, as above, rather than constructing a message unnecessarily. In reusable modules, create a module logger and allow the application to configure handlers. Avoid calling basicConfig() inside a library.

Record facts that support diagnosis: operation, identifier, duration, counts, and relevant status. Do not log passwords, access tokens, full payment-card numbers, or unnecessary personal data. Logs can become sensitive records and should have access controls, retention limits, integrity protection, and appropriate deletion procedures. Time synchronization and consistent correlation identifiers are important in distributed systems.

To include a traceback, use:

try:
    process_file(path)
except OSError:
    logger.exception("File processing failed: %s", path)
    raise

logger.exception() should be called inside an exception handler; it automatically includes traceback information.

6. Testing for Confidence

Testing compares actual behaviour with expected behaviour under defined conditions. Tests do not prove that a program has no defects, but they provide evidence and prevent known failures from returning.

6.1 Unit tests

A unit test focuses on a small component, often a function, and isolates external dependencies. A well-designed test follows Arrange–Act–Assert:

from mymodule import percentage


def test_percentage_returns_expected_value():
    # Arrange
    part, whole = 25, 100

    # Act
    result = percentage(part, whole)

    # Assert
    assert result == 25

The same test using unittest is:

import unittest
from mymodule import percentage


class TestPercentage(unittest.TestCase):
    def test_valid_values(self):
        self.assertEqual(percentage(25, 100), 25)

    def test_zero_whole_is_rejected(self):
        with self.assertRaises(ValueError):
            percentage(1, 0)

Test normal cases, boundary cases, invalid values, and important failure paths. A boundary test examines values at, just below, and just above a rule's limit. Tests should be independent, deterministic, readable, and fast.

Floating-point results require care. Binary floating-point cannot represent every decimal fraction exactly. Use a tolerance or an appropriate approximate assertion rather than assuming exact equality for calculations involving fractions:

assert abs(actual - expected) < 1e-9

The tolerance must be justified by the application's numerical requirements; it is not universally correct.

6.2 Integration and regression tests

An integration test checks interactions among components, such as a parser, database, and service layer. It is usually slower and may require controlled resources. A regression test is added after a defect is discovered to ensure that the same defect does not recur.

External services should be replaced with fakes, stubs, or mocks in unit tests. Integration tests should use isolated test databases or containers and must not alter production data. Test fixtures should be small, representative, and version-controlled.

6.3 Test coverage and test design

Coverage tools measure which lines or branches executed, but high coverage does not guarantee meaningful tests. A test that executes a line without checking its result provides weak evidence. Prioritize business rules, security boundaries, transformations, error handling, and high-risk integrations.

Property-based testing can generate many inputs to test general properties, such as “sorting preserves length and produces a nondecreasing sequence.” This complements, rather than replaces, carefully chosen example-based tests.

7. Code Quality and Maintainability

Reliable code is easier to test and debug. Apply the following conventions:

• follow PEP 8 formatting and use meaningful names;
• keep functions focused and reasonably short;
• document public interfaces, assumptions, and exceptions;
• use type hints where they improve communication;
• avoid duplicated validation and duplicated business rules;
• separate input/output from computation;
• use version control and small, reviewable commits;
• run formatters, linters, and tests automatically in continuous integration;
• treat warnings as information requiring a deliberate decision.

Static type checkers and linters detect some defects before execution. They do not replace run-time validation, tests, or domain review. Code review should ask whether errors are handled at the correct layer, whether logs expose sensitive information, and whether tests represent realistic failure conditions.

8. Applied Case: Reliable Sales Import

The following is an illustrative case. A retail analytics team imports daily CSV sales files. A previous script converted amounts directly to float, ignored malformed rows, and logged only “import complete.” A corrupted file consequently produced incomplete dashboards without immediate warning.

Background and problem

The team needs to reject records with missing identifiers, non-numeric amounts, or negative quantities while retaining valid records. Operations staff need a count of rejected rows and a useful explanation without exposing customer information.

Approach

The import function validates each row, raises a domain exception for malformed structure, logs aggregate results, and returns valid records. A unit test covers valid rows and each rejection rule. An integration test uses a temporary file and verifies that the complete import pipeline produces the expected count.

import logging

logger = logging.getLogger(__name__)


class InvalidRowError(ValueError):
    pass


def parse_sale(row):
    if not row.get("sale_id"):
        raise InvalidRowError("missing sale_id")
    try:
        quantity = int(row["quantity"])
        amount = float(row["amount"])
    except (KeyError, TypeError, ValueError) as error:
        raise InvalidRowError("quantity and amount must be numeric") from error
    if quantity < 0 or amount < 0:
        raise InvalidRowError("quantity and amount must be non-negative")
    return {"sale_id": row["sale_id"], "quantity": quantity, "amount": amount}


def import_sales(rows):
    valid, rejected = [], 0
    for row_number, row in enumerate(rows, start=1):
        try:
            valid.append(parse_sale(row))
        except InvalidRowError as error:
            rejected += 1
            logger.warning("Rejected row %d: %s", row_number, error)
    logger.info("Sales import completed: accepted=%d rejected=%d", len(valid), rejected)
    return valid

Outcome and learning points

The revised design makes invalid data visible, prevents negative values from entering analysis, and creates testable units. In a production system, row-level logging would need review for privacy and volume; aggregate metrics may be preferable. The team should also define whether rejecting a row is acceptable or whether a failed file should stop the pipeline. That decision belongs to the data-quality contract, not merely to the programmer.

9. Limitations, Ethics, and Contemporary Practice

No testing strategy covers every possible input. Tests may encode incorrect requirements, mocks may differ from real services, and logs may be unavailable during an outage. Reliability therefore requires layered controls: validation, tests, monitoring, code review, backups, and incident response.

In data science, exceptions and logs can expose personal, health, financial, or behavioural information. Apply data minimisation, access control, encryption, retention schedules, and organisational privacy policies. Error messages shown to users should be helpful without revealing internal paths, SQL statements, credentials, or security-sensitive details.

Contemporary development commonly combines automated testing with continuous integration, static analysis, dependency scanning, observability platforms, structured logs, metrics, and distributed tracing. AI-assisted coding can generate tests or suggest fixes, but generated code must be reviewed, tested, licensed appropriately, and checked for insecure exception handling or fabricated assumptions. Responsibility for deployed software remains with the development team.

Chapter Summary

Exceptions provide structured responses to run-time problems, while specific exception types and meaningful custom exceptions improve program interfaces. else, finally, context managers, propagation, and exception chaining support clear control flow and reliable cleanup. Assertions express internal assumptions but must not replace validation of external input.

Debugging is an evidence-based process involving reproduction, hypotheses, inspection, and regression testing. Debuggers reveal execution state, while logging provides a durable operational record. Good logs use suitable severity levels, structured context, and privacy safeguards.

Unit tests examine focused components; integration tests examine interactions; regression tests preserve fixes. Effective tests are independent, deterministic, and designed around normal, boundary, invalid, and failure conditions. Defensive programming, static analysis, code review, formatting, type hints, and continuous integration reinforce reliability. In professional data-science systems, technical correctness must be combined with security, privacy, governance, and clear data-quality policies.

Key Terms

Exception: An event raised during program execution.

Traceback: A record of the call sequence leading to an exception.

Exception handler: Code that responds to a selected exception.

Exception propagation: Movement of an unhandled exception to a calling function.

Exception chaining: Preserving a lower-level cause when raising a new exception.

Assertion: A development-time check of a condition believed to be true.

Defensive programming: Designing software to handle invalid or unexpected conditions safely.

Validation: Checking that input meets defined structural and domain rules.

Debugger: A tool for pausing execution and inspecting program state.

Breakpoint: A location at which a debugger pauses execution.

Logging: Recording program events for diagnosis and operation.

Traceback logging: Recording the call history associated with an exception.

Unit test: A test of a small, isolated program component.

Integration test: A test of interactions among components or systems.

Regression test: A test that prevents a previously corrected defect from returning.

Fixture: Controlled data or setup used by a test.

Mock: A test object that imitates a dependency and can verify interactions.

Code coverage: A measure of which program elements tests execute.

Invariant: A condition that should remain true during a computation.

Continuous integration: Automated building and checking of changes as they are submitted.

Review Questions

5 Short Answer Questions

1. What is the difference between a syntax error, an exception, and a logical error?
2. Why should programs usually catch specific exceptions rather than Exception?
3. Why are assertions unsuitable for essential validation of user input?
4. State two purposes of logging in a production data-processing system.
5. What is the Arrange–Act–Assert structure of a unit test?

5 Long Answer Questions

1. Explain the roles of try, except, else, and finally, using a suitable example.
2. Compare unit, integration, and regression testing, including the situations in which each is most useful.
3. Describe a systematic debugging process from reproducing a failure to preventing its recurrence.
4. Explain how defensive programming and context managers improve reliability in Python.
5. Discuss privacy, security, and governance risks associated with application logs and error messages.

3 Analytical / Application Questions

1. A CSV import silently skips malformed rows and produces an incomplete report. Design a validation, exception-handling, logging, and testing strategy to make the failure visible and recoverable.
2. A function works for ordinary values but fails when given an empty list, a missing dictionary key, and a floating-point result. Identify suitable tests and explain which failures should be handled by validation, exceptions, or approximate comparison.
3. A web application catches every exception and displays “Invalid input” to users. Analyse the technical risks of this design and propose a layered alternative that protects users while preserving diagnostic information for developers.

Multiple Choice Questions with Answers

Question 1

Which exception is most appropriate when a numeric value has the correct general type but violates a permitted range?

A. ValueError

B. SyntaxError

C. ImportError

D. KeyboardInterrupt

Correct Answer: A. ValueError

Question 2

What is the primary purpose of a finally block?

A. To run only when an exception occurs

B. To run cleanup code whether or not an exception occurs

C. To suppress every exception automatically

D. To retry the failed operation

Correct Answer: B. To run cleanup code whether or not an exception occurs

Question 3

Why should assertions not be used for essential validation of external input?

A. Assertions cannot compare values

B. Assertions are disabled when Python runs with optimisation

C. Assertions work only with strings

D. Assertions always terminate the operating system

Correct Answer: B. Assertions are disabled when Python runs with optimisation

Question 4

Which logging method normally includes the current exception traceback when called inside an exception handler?

A. logger.debug()

B. logger.info()

C. logger.exception()

D. logger.disable()

Correct Answer: C. logger.exception()

Question 5

Which test type primarily examines one small, isolated function?

A. Unit test

B. Integration test

C. Load test

D. Acceptance deployment

Correct Answer: A. Unit test

Question 6

What does exception chaining with raise NewError(...) from error preserve?

A. The original error as the cause of the new error

B. The user's password

C. The program's source-code formatting

D. A successful return value

Correct Answer: A. The original error as the cause of the new error

Question 7

Which practice best protects a file resource when reading it?

A. Use a context manager with with open(...)

B. Ignore all OSError exceptions

C. Open the file repeatedly without closing it

D. Use an assertion instead of closing the file

Correct Answer: A. Use a context manager with with open(...)

Question 8

A test compares two calculated decimal values that may contain floating-point representation error. What is generally most appropriate?

A. Compare them using a justified tolerance

B. Convert both values to unrelated strings

C. Skip the test

D. Catch all exceptions

Correct Answer: A. Compare them using a justified tolerance

Question 9

Which item should normally not be written to application logs?

A. A processing duration

B. The number of rejected rows

C. An access token

D. A correlation identifier

Correct Answer: C. An access token

Question 10

What is the main purpose of a regression test?

A. To ensure a previously corrected defect does not return

B. To replace all integration tests

C. To format source code automatically

D. To disable exception handling

Correct Answer: A. To ensure a previously corrected defect does not return