Django Dependency Management with pip-compile and pip-tools

Even a basic Django project has multiple dependencies. After installing Django itself, various third-party packages--and their dependencies!--must be installed and managed. How do you track version numbers in a reproducible way?

In the JavaScript ecosystem, the npm package manager makes it easy to keep track of requirements. Every time a package is installed via npm install <package-name>, an additional file, package.json, is created that is automatically updated with the package name and version number.

Unfortunately, the Python package manager, pip, does not do that, so other approaches are required. In this post, we'll run through several options, including pip freeze and requirements.txt, and then demonstrate why pip-compile is a popular approach for many Django developers.

Django Set Up

Let's quickly go through the standard setup for a new Django project. Navigate to a new directory on the command line, create a fresh virtual environment, and install Django.

# Windows
$ python -m venv .venv
$ .venv\Scripts\Activate.ps1
(.venv) $ python -m pip install django~=4.2.0

# macOS
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ python3 -m pip install django~=4.2.0

Even at this introductory phase, we've installed Django and some of its dependencies.

pip freeze and requirements.txt

The command pip freeze lists all installed packages within a virtual environment and their version numbers. Let's try it now.

(.venv) $ pip freeze
asgiref==3.7.2
Django==4.2.4
sqlparse==0.4.4

The latest version of Django, 4.2.4, relies on two other packages: asgiref and sqlparse.

You can output this to a physical file, typically called requirements.txt using the following command:

(.venv) $ pip freeze > requirements.txt

This file can be stored in source control using git. A fellow developer can download the code, create a fresh virtual environment, and run a command to install all the packages into their local environment.

(.venv) $ pip install -r requirements.txt

But this manual process requires the developer to regenerate a requirements.txt file every time a package is updated. It also doesn't directly address package dependencies.

pip-tools

A more sophisticated approach is using pip-tools, a command line tool for managing Python project dependencies. Let's install it and then run pip freeze to confirm it is installed correctly.

(.venv) $ python -m pip install pip-tools
(.venv) $ pip freeze
asgiref==3.7.2
build==0.10.0
click==8.1.7
Django==4.2.4
packaging==23.1
pip-tools==7.3.0
pyproject_hooks==1.0.0
sqlparse==0.4.4

Whoa, suddenly there are a lot more package dependencies to manage! Using our previous method, we would need to store these new packages manually in requirements.txt.

(.venv) $ pip freeze > requirements.txt

But at a high level, we only have two packages in our project: Django and pip-tools. All the other packages are dependencies of those two. With pip-tools, we can create a new root-level file called requirements.in to manage high-level dependencies. When compiled, it will automatically generate the full list of all dependencies into a requirements.txt file. Let's see this in action.

Since this tutorial is for demonstration purposes, use the deactivate command to exit the virtual environment. Delete the requirements.txt file using your text editor. Then delete the .venv directory. Note that you should always take great care when using the commands rm -rf together because you could inadvertently delete your entire computer if you're not careful. In this instance, we tell our computer to delete the entire .venv directory and its contents.

(.venv) $ deactivate
$ rm -rf .venv

Now, create a fresh virtual environment and activate it. Then install pip-tools.

# Windows
$ python -m venv .venv
$ .venv\Scripts\Activate.ps1
(.venv) $ python -m pip install pip-tools

# macOS
$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ python -m pip install pip-tools

If you run pip freeze, you can see that pip-tools and all its dependencies are now installed in our virtual environment.

(.venv) $ pip freeze 
build==0.10.0
click==8.1.7
packaging==23.1
pip-tools==7.3.0
pyproject_hooks==1.0.0

Rather than use pip to install Django, we will add it to the requirements.in file. Here is the current project layout.

.venv/
├── requirements.in

And we only need one line in the requirements.in file since we only install one high-level package.

# requirements.in
django

To install Django and its dependencies into our virtual environment as well as a new requirements.txt file, run the following command:

(.venv) $ pip-compile requirements.in --strip-extras
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --strip-extras requirements.in
#
asgiref==3.7.2
    # via django
django==4.2.4
    # via -r requirements.in
sqlparse==0.4.4
    # via django

The command outputs the contents of the virtual environment and automatically adds it to the requirements.txt file for us.

# requirements.txt
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --strip-extras requirements.in
#
asgiref==3.7.2
    # via django
django==4.2.4
    # via -r requirements.in
sqlparse==0.4.4
    # via django

Add another package: environs

To see how pip-compile shines, try adding another package. Let's say it's environs, a way to implement environment variables that also comes with handy Django defaults.

Add it to the requirements.in file.

# requirements.in
django
environs[django]

Then, run the command pip-compile requirements.in --strip-extras to install it into the virtual environment.

(.venv) $ pip-compile requirements.in --strip-extras
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --strip-extras requirements.in
#
asgiref==3.7.2
    # via django
dj-database-url==2.1.0
    # via environs
dj-email-url==1.0.6
    # via environs
django==4.2.4
    # via
    #   -r requirements.in
    #   dj-database-url
django-cache-url==3.4.4
    # via environs
environs==9.5.0
    # via -r requirements.in
marshmallow==3.20.1
    # via environs
packaging==23.1
    # via marshmallow
python-dotenv==1.0.0
    # via environs
sqlparse==0.4.4
    # via django
typing-extensions==4.7.1

The output is also now present in the requirements.txt file. And a good thing, too, because environs[django] has a bunch of dependencies: dj-database-url, dj-email-url, django-cache-url, marshmallow, and python-dotenv. Note that marshmallow has its own dependency, packaging, as does dj-database-url, which relies on typing-extensions.

# requirements.txt
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --strip-extras requirements.in
#
asgiref==3.7.2
    # via django
dj-database-url==2.1.0
    # via environs
dj-email-url==1.0.6
    # via environs
django==4.2.4
    # via
    #   -r requirements.in
    #   dj-database-url
django-cache-url==3.4.4
    # via environs
environs==9.5.0
    # via -r requirements.in
marshmallow==3.20.1
    # via environs
packaging==23.1
    # via marshmallow
python-dotenv==1.0.0
    # via environs
sqlparse==0.4.4
    # via django
typing-extensions==4.7.1
    # via dj-database-url

We've only installed two high-level packages at this point--Django and environs[django]--yet already managing our dependencies feels tricky. Thanks to pip-compile it doesn't have to be!

Updates

By default, pip-compile generates a requirements.txt file using the latest versions for the high-level packages in requirements.in. If an existing requirements.txt fulfills a dependency, no change will be made even if updates are available.

It is up to the developer to periodically update the packages. You can do this simultaneously with pip-compile --upgrade or update a specific package using --upgrade-package.

# only update the django package
(.venv) $ pip-compile --upgrade-package django

# update all packages
(.venv) $ pip-compile --upgrade

See the pip-tools docs for complete instructions.