Packages, Environments, and Dependency Management

Introduction

A small Python program may begin as one file, but professional software is usually composed of multiple modules, third-party libraries, configuration files, tests, documentation, and build metadata. As a project grows, developers must answer several practical questions: How should code be organised? Which files belong to a package? How can two projects use different library versions on the same computer? How can another developer install the project reliably? How can an organisation reduce the risk of malicious or legally unsuitable dependencies?

Python provides an integrated set of mechanisms for addressing these questions. Packages organise modules; virtual environments isolate project installations; pip installs distributions from package indexes; dependency specifications communicate compatibility requirements; semantic versioning provides a shared vocabulary for releases; and pyproject.toml describes modern build and project metadata. These mechanisms are related but not identical. A package is a source-code structure, whereas a distribution is an installable artefact. A virtual environment isolates installed packages, whereas a lock file records precise versions. Understanding these distinctions is essential for dependable Python development.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish modules, regular packages, namespace packages, and distributions;
• design a maintainable package structure and use __init__.py appropriately;
• create, activate, inspect, and remove virtual environments;
• install and inspect dependencies with pip;
• specify compatible dependency versions using requirement files and project metadata;
• interpret semantic versioning and understand its limitations;
• create a basic pyproject.toml configuration;
• evaluate supply-chain, licensing, security, and reproducibility risks; and
• apply professional dependency-management practices to a Python project.

1. Modules, Packages, and Distributions

A module is normally a Python file containing definitions, such as functions, classes, and variables. For example, validators.py is a module that can be imported with import validators when it is available on Python’s import path.

A package is a way of organising related modules under a common import namespace. A conventional package has a directory containing an __init__.py file:

inventory_project/
├── pyproject.toml
├── src/
│   └── inventory/
│       ├── __init__.py
│       ├── models.py
│       ├── pricing.py
│       └── reports/
│           ├── __init__.py
│           └── csv_report.py
└── tests/
    └── test_pricing.py

The package inventory contains the modules models and pricing, while inventory.reports is a subpackage. Imports may therefore be written as:

from inventory.pricing import calculate_total
from inventory.reports.csv_report import write_report

A distribution is a project prepared for installation and distribution. It may contain one or more import packages, metadata, documentation, tests, and compiled or generated components. A distribution installed with pip install requests provides importable modules, but the distribution name and import name are not necessarily identical. This distinction explains why the command may be pip install beautifulsoup4 while the Python statement is from bs4 import BeautifulSoup.

The role of __init__.py

For a regular package, __init__.py performs several possible functions:

1. It identifies the directory as a conventional Python package.
2. It runs package-initialisation code when the package is imported.
3. It may expose a carefully selected public interface.
4. It may define package metadata such as __version__.

A minimal file is often best:

# inventory/__init__.py
"""Tools for inventory management."""

__version__ = "1.0.0"

Names can be re-exported to make an interface convenient:

# inventory/__init__.py
from .pricing import calculate_total

__all__ = ["calculate_total"]

However, excessive work in __init__.py is undesirable. Importing a package should not unexpectedly connect to a database, read large files, or perform network operations. Such side effects make testing, command-line use, and dependency analysis more difficult.

Python also supports namespace packages, which can span multiple directories and do not require an __init__.py file. They are useful for large ecosystems in which separately distributed projects contribute to one namespace. For ordinary applications and most diploma-level projects, regular packages are easier to understand and maintain.

Absolute and relative imports

An absolute import states the package path from its top-level name:

from inventory.models import Product

A relative import begins with dots and is interpreted relative to the current package:

from .models import Product
from .reports.csv_report import write_report

Absolute imports are often clearer across a large project. Relative imports can make internal package relationships explicit and avoid collisions with unrelated top-level modules. A module intended to be executed directly should generally use a small entry-point module rather than relying on fragile relative-import behaviour.

2. Designing a Package Structure

A good package structure separates responsibilities. For example, a web service might contain:

service_project/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── service/
│       ├── __init__.py
│       ├── api.py
│       ├── config.py
│       ├── domain/
│       │   ├── __init__.py
│       │   └── orders.py
│       └── persistence/
│           ├── __init__.py
│           └── repository.py
└── tests/
    ├── test_orders.py
    └── test_repository.py

The src layout is widely used because it prevents accidental imports from the project’s working directory. Without it, tests may import the local source tree even though the package has not been installed, hiding packaging errors. In a src layout, the project is normally installed into the active environment, making the test process more closely resemble how users will consume the distribution.

Package boundaries should reflect stable responsibilities rather than arbitrary file sizes. A module should have a coherent purpose, and public interfaces should be deliberately chosen. Internal implementation details may be marked by a leading underscore, although this is a convention rather than a security mechanism.

Circular imports are a common design problem. If module a imports module b while b imports a, initialisation may occur with incomplete definitions. Refactoring shared abstractions into a third module, importing locally where justified, or improving dependency direction is usually preferable to relying on import-order accidents.

3. Virtual Environments

A virtual environment is an isolated directory containing a Python interpreter and an environment-specific installation location. It allows projects on one computer to use different versions of the same dependency. It does not provide a complete operating-system container: it shares the underlying operating system and usually the base Python installation, and it does not isolate external services or system libraries.

Create an environment with:

python -m venv .venv

On systems where Python 3 is invoked as python3, use:

python3 -m venv .venv

Activate it as follows:

# Linux or macOS
source .venv/bin/activate

# Windows PowerShell
.venv\Scripts\Activate.ps1

The command prompt commonly displays the environment name. Verify the interpreter and installer:

python --version
python -m pip --version

Using python -m pip is safer than invoking a bare pip, because it associates pip with the selected Python interpreter. Deactivate the environment with:

deactivate

A virtual environment is disposable. It should normally not be committed to version control. Add .venv/ to .gitignore, and recreate the environment from project metadata when necessary.

4. Installing and Inspecting Dependencies with pip

pip is a package installer and resolver for Python distributions. Typical commands include:

python -m pip install requests
python -m pip uninstall requests
python -m pip list
python -m pip show requests
python -m pip freeze
python -m pip check

pip list gives a readable list of installed projects. pip freeze produces a requirements-style snapshot, often useful for reproducing an environment but not always ideal as the primary declaration of direct project requirements. pip check reports incompatible installed requirements.

A dependency can be installed from a requirement file:

python -m pip install -r requirements.txt

Example:

requests>=2.31,<3
pydantic>=2.0,<3
pytest>=8,<9

A professional project usually distinguishes direct dependencies from transitive dependencies. A direct dependency is required by the project itself. A transitive dependency is installed because a direct dependency requires it. Recording every transitive package in a manually maintained file can become difficult because the complete dependency graph changes across platforms and releases.

Modern workflows often use project metadata for direct requirements and a resolver or lock mechanism for exact, reproducible installations. The precise tool used for locking varies, but the underlying principle is consistent: declare broad, intentional compatibility and record an exact tested environment separately when reproducibility requires it.

5. Dependency Specifications and Version Constraints

A version constraint describes which releases a project accepts. Common forms include:

Exact pins can improve repeatability, but they can also prevent security and bug-fix updates. Unbounded requirements such as library permit the resolver to choose a current version, but can allow a future breaking release. A bounded range such as library>=2.4,<3 often expresses the intended compatibility policy more effectively.

Environment markers allow conditional requirements:

colorama>=0.4; sys_platform == "win32"
importlib-metadata>=6; python_version < "3.10"

Optional features, sometimes called extras, allow users to install additional functionality:

python -m pip install inventory[postgresql]

The project metadata defines which additional dependencies belong to the postgresql extra.

Dependencies should be tested together, not merely installed. A valid version range is not evidence that every version in that range behaves correctly. Continuous integration can test supported Python versions and important dependency combinations.

6. Semantic Versioning

Semantic Versioning, commonly written as SemVer, uses a three-part form:

MAJOR.MINOR.PATCH

Under the conventional rule:

• MAJOR increases for incompatible public API changes;
• MINOR increases for backward-compatible functionality; and
• PATCH increases for backward-compatible bug fixes.

Thus, changing 1.4.2 to 1.4.3 suggests a compatible correction, while changing it to 1.5.0 suggests an additional backward-compatible feature. Changing it to 2.0.0 signals possible incompatibility.

SemVer is a communication convention, not a guarantee. “Breaking” depends on the public interface and documented behaviour. An undocumented behaviour may nevertheless be relied upon by users. Conversely, a major release may preserve most practical use cases. Python package version comparison also follows the Python Packaging Authority’s version-specification rules, including pre-release and local-version semantics; these rules should not be confused with a simplistic string comparison.

Pre-release identifiers such as 2.0.0rc1 indicate a release candidate. Development releases and post-releases have specialised meanings. Projects should document their release policy and test compatibility rather than assuming that version numbers alone establish safety.

7. pyproject.toml and Modern Project Metadata

pyproject.toml is a standard configuration file for Python projects. It can identify the build system and contain project metadata. A basic example is:

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "inventory-tools"
version = "1.0.0"
description = "Utilities for inventory processing"
readme = "README.md"
requires-python = ">=3.10"
license = {file = "LICENSE"}
authors = [
  {name = "Example Development Team"}
]
dependencies = [
  "pydantic>=2.0,<3",
  "requests>=2.31,<3"
]

[project.optional-dependencies]
test = ["pytest>=8,<9"]
postgresql = ["psycopg>=3.1,<4"]

[project.scripts]
inventory-report = "inventory.cli:main"

The [build-system] table tells a frontend such as pip which backend to use to build a wheel or source distribution. The [project] table contains standard metadata such as the distribution name, supported Python versions, dependencies, and entry points.

A wheel is a built distribution designed for installation. A source distribution, often called an sdist, contains source and build information from which a wheel may be built. Building and installing a project tests whether its packaging metadata is complete. A project that works only when executed from its source directory is not necessarily correctly packaged.

The license field and related metadata have evolved across packaging standards and tooling versions. Projects should follow current official specifications and verify that their chosen build backend accepts the syntax used. Metadata is not merely administrative: it affects installation, discoverability, legal communication, and automated tooling.

8. Reproducibility and Environment Records

A reproducible environment is one that can be rebuilt with sufficiently similar results from documented inputs. Perfect bit-for-bit reproducibility may require more than Python package versions. Relevant factors include:

• Python interpreter version and implementation;
• operating system and processor architecture;
• direct and transitive package versions;
• package hashes and repository sources;
• system libraries and compilers;
• environment variables and configuration; and
• external services or data sets.

A requirements snapshot may be generated with:

python -m pip freeze > requirements-lock.txt

This records installed distributions, but it is platform-specific in some projects and may include packages that are not direct dependencies. A stronger workflow uses a resolver or lock-file tool that records the complete dependency graph, supported markers, and sometimes hashes. Hash checking helps ensure that the downloaded artefact is the expected file, although it does not prove that the source code is benign.

Reproducibility should be balanced with maintenance. A team should regularly update dependencies, run tests, review changes, and regenerate its lock record. An old frozen environment may be repeatable but insecure.

9. Supply-Chain Security and Licensing

Installing a dependency executes or makes available code written by another party. Dependency management is therefore a supply-chain governance issue.

Important risks include typosquatting, dependency confusion, compromised maintainer accounts, malicious updates, abandoned projects, vulnerable transitive dependencies, and packages that execute build-time code. Controls include:

1. use trusted package indexes and configured repository policies;
2. verify package names before installation;
3. review release history, maintainers, and project health;
4. scan dependencies for known vulnerabilities;
5. pin or constrain versions according to risk;
6. use hashes or trusted publishing controls where appropriate;
7. limit installation and build permissions in CI systems;
8. generate software bills of materials when required; and
9. monitor and promptly remediate security advisories.

No scanner detects every vulnerability. Human review, tests, least privilege, and secure build infrastructure remain necessary.

Licensing is also part of technical responsibility. A dependency may use an MIT, BSD, Apache-2.0, GPL, LGPL, proprietary, or other licence. The obligations differ. Some licences require notices or attribution; others impose conditions when derivative works are distributed. An organisation should maintain a dependency and licence inventory, retain required notices, and obtain legal advice when a project’s distribution model creates uncertainty. “Open source” does not mean “without obligations.”

10. Applied Case: Packaging an Inventory Service

Background

This illustrative case concerns a small company developing an inventory service. The prototype consists of inventory.py, imports requests, and works on the developer’s laptop. A second developer cannot reproduce the environment, and a deployment machine uses a different version of a validation library.

Problem

The project has no package boundary, no declared Python version, no documented dependencies, and no separation between development and production tools. Tests pass locally but fail after deployment.

Analysis

The team identifies three direct runtime dependencies and two development dependencies. It chooses Python 3.11 as the supported interpreter, uses a src layout, and decides that validation-library major version 2 is supported but version 3 is not yet tested. It also discovers that a transitive HTTP library has a security advisory.

Approach and Solution

The team creates:

inventory-service/
├── pyproject.toml
├── src/inventory/__init__.py
├── src/inventory/client.py
├── src/inventory/models.py
├── tests/
└── README.md

The project metadata declares requires-python = ">=3.11,<3.13" and uses bounded requirements such as validation-lib>=2.2,<3. A virtual environment is created for development. Test dependencies are placed in an optional test group, and a supported lock record is generated after successful testing. Continuous integration builds the package, installs it into a clean environment, runs tests, checks dependency vulnerabilities, and verifies the supported Python versions.

The team reviews the licences of direct and transitive dependencies and records notices in the release documentation. It updates the vulnerable transitive package through a compatible dependency update, then repeats the test and security review.

Outcome

A new developer can create an environment from documented instructions and install the project consistently. Packaging errors are detected before release, version policy is explicit, and the organisation can identify which third-party code is present.

Learning Points

The central lesson is that dependency management is an engineering process rather than a single pip install command. Package structure, metadata, testing, security review, licensing, and release procedures must support one another.

11. Common Errors and Best Practices

Common errors include committing .venv, naming a local file requests.py, using a bare pip associated with another interpreter, importing only from an uninstalled source directory, placing all packages into a single unstructured module, and copying an old pip freeze file without reviewing it.

Recommended practices are:

• use a clear package name that does not shadow standard or third-party modules;
• keep public APIs small and document them;
• avoid import-time side effects;
• declare direct dependencies in project metadata;
• separate runtime, test, documentation, and optional dependencies;
• use python -m pip with the intended interpreter;
• build and install the package in a clean environment during CI;
• test supported Python versions and important dependency combinations;
• review updates rather than accepting them blindly;
• record reproducibility information appropriate to the deployment; and
• treat security and licence review as part of ordinary development.

Chapter Summary

Python modules and packages provide structure for reusable code, while distributions are installable project artefacts. A regular package commonly contains __init__.py, which may define metadata and a public interface but should avoid expensive side effects. The src layout reduces accidental imports and exposes packaging mistakes early.

Virtual environments isolate project installations but do not replace operating-system or container isolation. pip installs and inspects distributions, while dependency specifications express acceptable versions. Semantic versioning communicates intended compatibility but cannot guarantee it. pyproject.toml centralises build configuration and project metadata, including Python requirements, dependencies, optional features, and command-line entry points.

Professional dependency management also requires reproducibility, security, and legal awareness. Teams should distinguish direct from transitive dependencies, test clean installations, monitor vulnerabilities, evaluate licences, and document the environment sufficiently for its operational purpose. These practices turn a collection of Python files into a maintainable and governable software project.

Key Terms

Module: A Python file containing importable definitions.

Package: A namespace that organises related Python modules.

Regular package: A package conventionally containing __init__.py.

Namespace package: A package namespace that may span directories without a traditional __init__.py.

Distribution: An installable Python project artefact and its metadata.

__init__.py: A package-initialisation file that may define exports and metadata.

Import path: The locations Python searches for importable modules.

Virtual environment: An isolated Python installation context for a project.

pip: A tool for installing and managing Python distributions.

Direct dependency: A package explicitly required by a project.

Transitive dependency: A package required by another dependency.

Semantic Versioning: A versioning convention using major, minor, and patch components.

Dependency constraint: A rule describing acceptable package versions.

Extra: An optional dependency group providing additional project functionality.

pyproject.toml: Standard project configuration and metadata file.

Build backend: Software that builds Python distributions.

Wheel: A built Python distribution format.

Reproducibility: The ability to rebuild an environment with sufficiently consistent results.

Software supply chain: The people, tools, artefacts, and dependencies involved in producing software.

Software licence: Legal terms governing use, modification, and distribution of software.

Review Questions

5 Short Answer Questions

1. What is the difference between a Python package and a distribution?
2. State two appropriate uses of __init__.py.
3. Why should a project normally exclude .venv from version control?
4. What is the purpose of the [build-system] table in pyproject.toml?
5. Distinguish a direct dependency from a transitive dependency.

5 Long Answer Questions

1. Explain how a src layout can reveal packaging errors that a flat project layout may hide.
2. Describe the process of creating, using, checking, and removing a virtual environment.
3. Compare exact version pins, lower bounds, and bounded version ranges, including advantages and disadvantages.
4. Explain Semantic Versioning and discuss why version numbers cannot guarantee compatibility.
5. Describe how pyproject.toml, a build backend, a wheel, and pip interact during installation.

3 Analytical / Application Questions

1. A team has a project that works only when run from its repository root. Design a package layout and testing procedure that would identify and correct the problem.
2. A dependency update fixes a security vulnerability but introduces a major-version change. Analyse the evidence the team should review before adopting it.
3. Design a dependency-management policy for a small organisation that addresses version constraints, lock records, vulnerability monitoring, licence review, and continuous integration.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a distribution from a package?

A. A distribution is always a single Python file, while a package is always compiled.

B. A distribution is an installable project artefact, while a package organises importable Python modules.

C. A distribution can contain only standard-library modules, while a package can contain third-party modules.

D. A package is installed by the operating system, while a distribution is imported by Python.

Correct Answer: B. A distribution is an installable project artefact, while a package organises importable Python modules.

Question 2

What is a principal benefit of using a virtual environment?

A. It isolates project-specific Python packages from other project installations.

B. It encrypts source code before it is executed.

C. It guarantees that all operating-system libraries are identical.

D. It automatically removes vulnerable dependencies.

Correct Answer: A. It isolates project-specific Python packages from other project installations.

Question 3

Which command most directly associates pip with the selected Python interpreter?

A. pip install package

B. python -m pip install package

C. python install package

D. python --pip package

Correct Answer: B. python -m pip install package

Question 4

What does the constraint library>=2.4,<3 express?

A. Only version 2.4.0 is permitted.

B. Versions below 2.4 and version 3 are required.

C. Versions at least 2.4 but below 3 are acceptable.

D. Any version except 2.4 is acceptable.

Correct Answer: C. Versions at least 2.4 but below 3 are acceptable.

Question 5

Under conventional Semantic Versioning, which change normally indicates a backward-compatible bug fix?

A. 1.4.2 to 1.4.3

B. 1.4.2 to 1.5.0

C. 1.4.2 to 2.0.0

D. 1.4.2 to 0.4.2

Correct Answer: A. 1.4.2 to 1.4.3

Question 6

Which practice is generally inappropriate in a package’s __init__.py file?

A. Defining a small package docstring

B. Re-exporting a deliberately selected public function

C. Defining package metadata

D. Opening a database connection whenever the package is imported

Correct Answer: D. Opening a database connection whenever the package is imported

Question 7

What is the main purpose of the dependencies field in the [project] table?

A. To declare packages required by the project at runtime

B. To list every file tracked by Git

C. To configure the operating-system kernel

D. To define Python’s built-in modules

Correct Answer: A. To declare packages required by the project at runtime

Question 8

Which item is a supply-chain security concern?

A. A package uses four-space indentation

B. A package name is confused with a similarly named malicious project

C. A module contains a class rather than a function

D. A project uses a src directory

Correct Answer: B. A package name is confused with a similarly named malicious project

Question 9

Why might a team build and install its project in a clean environment during continuous integration?

A. To ensure tests rely on undeclared packages already installed on a developer’s computer

B. To detect incomplete packaging metadata and undeclared dependencies

C. To avoid running the project’s test suite

D. To make version constraints unnecessary

Correct Answer: B. To detect incomplete packaging metadata and undeclared dependencies

Question 10

Which statement about open-source software licences is most accurate?

A. Open-source software has no legal conditions.

B. All open-source licences impose exactly the same obligations.

C. Licence obligations can differ, so dependencies should be reviewed and required notices preserved.

D. Licence review is needed only for code written in C or C++.

Correct Answer: C. Licence obligations can differ, so dependencies should be reviewed and required notices preserved.