pypi2nix - generate Nix expressions for Python packages

.. contents::

See https://pypi2nix.readthedocs.io/en/latest/ for documentation.

.. inclusion-marker

pypi2nix is a command line tool that generates Nix expressions_ from different requirements.txt. This is useful for:

• Building a Nix derivation for a program written in Python as part of packaging it.

• Setting up a development environment to hack on a program written in Python.

  The only way we can fix bugs with pypi2nix is if you report them. Please create an issue if you discover problems.

pypi2nix will (until further notice) only work with latest unstable channel. This is due to ongoing changes in Python infrastructure happening in Nixpkgs.

The Nixpkgs manual section about Python <https://nixos.org/nixpkgs/manual/#python>_ makes good reading if you haven't seen it already.

1. Installation

---

pypi2nix is part of nixpkgs. If you just want to use pypi2nix on your system, it is recommended that you install it via the regular means, e.g. nix-env -iA nixos.pypi2nix on NixOS or nix-env -iA nixpkgs.pypi2nix on other systems utilizing nix.

System Requirements ^^^^^^^^^^^^^^^^^^^

Make sure Nix is installed::

% curl https://nixos.org/nix/install | sh

Currently pypi2nix is only tested against linux systems. Supported nixpkgs channels are nixos-19.09 and nixos-unstable. Due to the nature of nixos-unstable the occasional breakage of pypi2nix is to be expected. We try to provide fixes in that regard in a timely manner.

Ad hoc Installation (Simple) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For just installing the package with a command, use nix-env_::

nix-env -if https://github.com/nix-community/pypi2nix/tarball/master

Declarative Installation (Advanced) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you prefer to explicitly declare each installed package in your Nix(OS) or project configuration, you can do the following:

First, import the package from its default.nix by fetching the whole git repository with pkgs.fetchgit. Afterwards you can just add the imported attribute the list of installed software.

Below you find an example for NixOS' configuration.nix. Other methods like home-manager_ work similar::

let
  pypi2nix = import (pkgs.fetchgit {
    url = "https://github.com/nix-community/pypi2nix";
    # adjust rev and sha256 to desired version
    rev = "v2.0.1";
    sha256 = "sha256:0mxh3x8bck3axdfi9vh9mz1m3zvmzqkcgy6gxp8f9hhs6qg5146y";
  }) {};
in
  environment.systemPackages = [
    # your other packages
    pypi2nix
  ];

2. Usage

The easiest way to generate a Nix expressions is to invoke::

% pypi2nix -e packageA -e packageB==0.1

If you also have a requirements.txt file for your Python project you can use the -r option.

::

% pypi2nix -e packageA -e packageB==0.1 \
    -r requirements.txt -r requirements-dev.txt

What is being generated ^^^^^^^^^^^^^^^^^^^^^^^

Option -V tells pypi2nix which python version to be used. To see which Python versions are available consult pypi2nix --help.

Once Nix expressions are generated you should be able to see 3 new files:

• requirements_frozen.txt - full frozen set for your for your pypi2nix call. This is the output you would expect from pip freeze.

• requirements.nix is a file which contains a nix expression to for the package set that was built.

• requirements_override.nix - this is an empty file which lets you override generated nix expressions.

Building generated packages ^^^^^^^^^^^^^^^^^^^^^^^^^^^

Build one package::

% nix build -f requirements.nix packages.empy

Build all packages::

% nix build -f requirements.nix packages

Build python interpreter with all packages loaded::

% nix build -f requirements.nix interpreter
% ./result/bin/python -c "import empy"

Enter development environment::

% nix run -f requirements.nix interpreter
[user@localhost:~/dev/nixos/pypi2nix) % python -c "import empy"

Using generated packages ^^^^^^^^^^^^^^^^^^^^^^^^

If you are working on a project where its dependencies are defined in requirements.txt then you can create a default.nix and add generated packages as buildInputs, as demonstrated here::

{}:
let
  python = import ./requirements.nix { inherit pkgs; };
in python.mkDerivation {
  name = "ProjectA-1.0.0";
  src = ./.;
  buildInputs = [
    python.packages."coverage"
    python.packages."flake8"
    python.packages."mock"
    python.packages."pytest"
    python.packages."pytest-asyncio"
    python.packages."pytest-cov"
    python.packages."pytest-mock"
    python.packages."pytest-xdist"
    python.packages."virtualenv"
  ];
  propagatedBuildInputs = [
    python.packages."aiohttp"
    python.packages."arrow"
    python.packages."defusedxml"
    python.packages."frozendict"
    python.packages."jsonschema"
    python.packages."taskcluster"
    python.packages."virtualenv"
  ];
  ...
}

As you can see you can access all packages via python.packages."<name>". If you want to depend on all packages you can even do::

propagatedBuildInputs = builtins.attrValues python.packages;

Command line options ^^^^^^^^^^^^^^^^^^^^

-v Increase amount and detail of information output to the user. Verbosity levels are ERROR, WARNING, INFO and DEBUG in that order. The default verbosity is INFO.

-q Reduce amount and detail of information output to the user. See -v for more information.

-I/--nix-path TEXT Add entries to the NIX_PATH environment variable similarly to how -I works with nix executables like nix-build. This can be useful for generating package sets based on a different nixpkgs version than the one used one the local system.

--nix-shell PATH Path to an alternative version of the nix-shell command. The default is the first executable that will be found in the current PATH of the system.

--version Show the current version of pypi2nix

--basename TEXT This option determins the name the produced files. So with --basename environment you would get the files environment.nix, environment_frozen.nix and environment_override.nix.

--extra-build-inputs/-E TEXT Extra build inputs that the required python packages need to run, e.g. libffi or libgl. In that case you would provide -E "libffi libgl". These nix packages will be available in the build environment for the wheels.

--emit-extra-build-inputs/--no-emit-extra-build-inputs These options let you control if external build dependencies specified via -E will end up in the generated nix package set. Please note that if you select this option, your overrides need to make sure that python packages find their respective external dependencies.

--extra-env/-N TEXT Extra environment variables that will be passed to the build environment. Note that you can use nix expressions in this string, e.g. -N 'BERKELEYDB_DIR=${pkgs.db.dev}'.

--enable-tests/-T Specify this flag if you want to enable the check phase of all packages in the generated nix expression. Please note that this feature is highly exprimental and will probably not work for your use case.

--python-version/-V Specify the python version you want the requirement set to be built with. The default is 3 which translates to the python3 derivation of nixpkgs.

--requirements/-r FILE Specify a requirements file, similar as you would with pip. pypi2nix tries to be fully compatible with the file format of pip.

--editable/-e TEXT This option allows you to specify individual requirements that get added to the requirement set, e.g. pypi2nix -e attrs, pypi2nix -e $HOME/src/myproject#egg=myproject or pypi2nix -e .#egg=myegg.

--setup-requires/-s TEXT Allows you to specify python packages that need to be present in the build environment of other packages, a good example of this would be setuptools-scm. Note that pypi2nix tries to detect these dependencies on its own. You only need to specify this flag in cases where a package author or maintainer forgot to mention build time dependencies in their setup or neither setup.cfg nor pyproject.toml is used.

--overrides/-O URL Allows you to specify additional overrides that conform to the general structure of requirements_override.nix. We support regular URLs with http and https scheme and also git. An example for using https would be pypi2nix -O https://myoverrides.test/overrides.nix. Reusing an overlay from a git repository would be done like so: pypi2nix -O git+https://github.com/nix-community/pypi2nix.git&path=requirement_override.nix. Please keep in mind that these overrides are incorporated in a nix expression with a precalculated hash value. So if the file changes upstream your generated package can not be built anymore.

--default-overrides/--no-default-overrides Pull in overrides from https://github.com/nix-community/pypi2nix-overrides. This feature is enabled by default.

--wheels-cache/-W TEXT A location where prebuilt wheels can be found. This option will ultimately be passed to pip --find-links. Only point to wheels that are built through pypi2nix on your own or a very similar system.

--build-directory TEXT Warning A bug in pypi2nix currently prevents some pack