Dependency management

Warehouse’s approach to dependency management can be summarized as follows:

  • Separate requirements files for different environments (deploy, development, docs, linting, testing, etc.);
  • All dependencies are pinned to precise versions, and include artifact hashes;
  • Pinned requirements and subdependencies are compiled from .in files.

We install all dependencies with pip, and we use pip-tools to compile dependencies.

In practice, developers need to interact with our dependencies in three ways:

Upgrading existing dependencies

Dependencies are automatically upgraded via Dependabot pull requests, and occasionally merged by maintainers.

Adding new dependencies

Deciding to add a new dependency should be made carefully. Generally, we are not opposed to adding more dependencies, however some effort should be made to ensure that a given dependency:

  • Is reasonably stable;
  • Is currently maintained;
  • Doesn’t introduce a large amount of sub-dependencies.

All top-level dependencies are included in one or more .in files, which are then compiled into .txt files with precise versions and artifact hashes.

When adding a new dependency, it’s important to add it to the correct .in file:

File Purpose Required only to run in production For our documentation For linting our docs and codebase Every dependency of our web service Required to run our tests

Dependencies that are either private or aren’t deployed to production aren’t compiled:

File Purpose
dev.txt Various development dependencies
ipython.txt Specific to using IPython as your shell
theme.txt Private dependencies for our logos and theme

To add a new dependency:

  1. Add the project name to the appropriate .in file

  2. Recompile the dependencies for each modified .in file:

    $ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in
  3. Commit the changes

Removing existing dependencies

Only top-level dependencies should be removed. The process is similar to the process for adding new dependencies:

  1. Remove the project name from the appropriate .in file

  2. Recompile the dependencies for each modified .in file:

    $ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in
  3. Commit the changes

Returning vs Raising HTTP Exceptions

Pyramid allows the various HTTP Exceptions to be either returned or raised, and the difference between whether you return or raise them are subtle. The differences between returning and raising a response are:

  • Returning a response commits the transaction associated with the request, while raising rolls it back.
  • Returning a response does not invoke the exec_view handler, while raising does.

The follow table shows what the default method should be for each type of HTTP exception, this is only the default and judgement should be applied to each situation.

Class Method
HTTPSuccessful (2xx) Return
HTTPRedirection (3xx) Return
HTTPClientError (4xx) Raise, except for HTTPNotFound which should be return.
HTTPServerError (5xx) Raise