Thoth Solver

Python dependency solver used in Thoth project <https://thoth-station.ninja>__.

As Python is a dynamic programming language, Thoth runs several types of solvers that differ in software environment (operating system, native packages present, system symbols and their versions and Python interpreter version). An example can be a solver which is running raw RHEL 8.0 with Python 3.6, another example can be a solver with Fedora 33 with Python 3.9 installed with different version of glibc and some of the ABI symbols of native libraries provided by operating system (see also Python manylinux standards and devtools for more info).

See the listing built solvers used in Thoth:

• quay.io/thoth-station/solver-rhel-8-py38 <https://quay.io/repository/thoth-station/solver-rhel-8-py38>__

• quay.io/thoth-station/solver-rhel-8-py36 <https://quay.io/repository/thoth-station/solver-rhel-8-py36>__

• quay.io/thoth-station/solver-fedora-34-py39 <https://quay.io/repository/thoth-station/solver-fedora-34-py39>__

For a detailed explanation see this blog post <https://dev.to/fridex/how-to-beat-python-s-pip-solving-python-dependencies-2d6e>__.

This solver is run in different environments (different operating systems with various native packages provided) to obtain dependency information about Python packages. An important information is also the fact whether the given Python package is installable into the environment (e.g. dependencies on native packages being present in the environment). An example could be UBI 8 specific solver running Python 3.8 or UBI 8 running Python 3.6 with gcc tooling for building native extensions.

The source code is based on the original solver code that was implemented by Pavel Odvody in project CuCoS.

Project Scope

The aim of this project is to answer a simple question - what packages will be installed (resolved by pip or any Python compliant dependency resolver) for the provided stack?

Imagine you have an application that has one dependency:

.. code-block:: console

tensorflow

Tool provided by this project will tell you what dependencies will be considered during resolution (the whole dependency graph):

.. code-block:: console

thoth-solver --verbose python -r tensorflow

The output of this solver is a dependency analysis for the given software stack

• in the example above, package tensorflow in any release with analysis of its all dependencies (direct and indirect ones) with additional information from Python ecosystem needed for a Python resolver to perform the actual tensorflow resolution.

The tool also allows specifying custom Python package indexes which conform to PEP-503 <https://www.python.org/dev/peps/pep-0503/>__ - see the --index option for analyzing your custom Python packages provided by your repositories.

It is also possible to restrict version using standard Python version range specifiers and/or limit the output just to direct dependencies.

Produced output

This tool (unless --no-transitive is specified) recursively analyzes all the dependencies of the desired package inside a specific environment. Dependencies to be analyzed can be defined in similar to requirements.txt file or as a string in a form of:

.. code-block:: console

<package-name><version-cmp><version-identifier>

Where <package-name> is the analyzed package name (as present on PyPI for example), part <version-cmp><version-identifier> is optional and creates version specifier for the given package (if not specified, all versions are considered).

An example output shown bellow can be reproduced by running the tool with the following arguments (with an example of produced log):

.. code-block:: console

$ thoth-solver python --requirements 'tensorflow==2.0.0' --index https://pypi.org/simple --no-transitive 2019-10-01 14:01:02,756 [31432] INFO root:128: Logging to a Sentry instance is turned off 2019-10-01 14:01:02,756 [31432] INFO root:150: Logging to rsyslog endpoint is turned off 2019-10-01 14:01:06,838 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'tensorflow==2.0.0' 2019-10-01 14:01:07,003 [31432] INFO thoth.solver.python.python:356: Using index 'https://pypi.org/simple' to discover package 'tensorflow' in version '2.0.0' 2019-10-01 14:01:40,568 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'absl-py' with range '>=0.7.0' from 'https://pypi.org/simple' 2019-10-01 14:01:40,568 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'absl-py>=0.7.0' 2019-10-01 14:01:40,689 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'astor' with range '>=0.6.0' from 'https://pypi.org/simple' 2019-10-01 14:01:40,689 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'astor>=0.6.0' 2019-10-01 14:01:40,841 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'gast' with range '==0.2.2' from 'https://pypi.org/simple' 2019-10-01 14:01:40,841 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'gast==0.2.2' 2019-10-01 14:01:40,984 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'google-pasta' with range '>=0.1.6' from 'https://pypi.org/simple' 2019-10-01 14:01:40,985 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'google-pasta>=0.1.6' 2019-10-01 14:01:41,104 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'keras-applications' with range '>=1.0.8' from 'https://pypi.org/simple' 2019-10-01 14:01:41,104 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'keras-applications>=1.0.8' 2019-10-01 14:01:41,273 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'keras-preprocessing' with range '>=1.0.5' from 'https://pypi.org/simple' 2019-10-01 14:01:41,274 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'keras-preprocessing>=1.0.5' 2019-10-01 14:01:41,443 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'numpy' with range '<2.0,>=1.16.0' from 'https://pypi.org/simple' 2019-10-01 14:01:41,443 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'numpy<2.0,>=1.16.0' 2019-10-01 14:01:41,723 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'opt-einsum' with range '>=2.3.2' from 'https://pypi.org/simple' 2019-10-01 14:01:41,723 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'opt-einsum>=2.3.2' 2019-10-01 14:01:41,828 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'six' with range '>=1.10.0' from 'https://pypi.org/simple' 2019-10-01 14:01:41,828 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'six>=1.10.0' 2019-10-01 14:01:41,942 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'protobuf' with range '>=3.6.1' from 'https://pypi.org/simple' 2019-10-01 14:01:41,943 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'protobuf>=3.6.1' 2019-10-01 14:01:42,095 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'tensorboard' with range '<2.1.0,>=2.0.0' from 'https://pypi.org/simple' 2019-10-01 14:01:42,095 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'tensorboard<2.1.0,>=2.0.0' 2019-10-01 14:01:42,286 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'tensorflow-estimator' with range '<2.1.0,>=2.0.0' from 'https://pypi.org/simple' 2019-10-01 14:01:42,287 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'tensorflow-estimator<2.1.0,>=2.0.0' 2019-10-01 14:01:42,411 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'termcolor' with range '>=1.1.0' from 'https://pypi.org/simple' 2019-10-01 14:01:42,411 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'termcolor>=1.1.0' 2019-10-01 14:01:42,580 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'wrapt' with range '>=1.11.1' from 'https://pypi.org/simple' 2019-10-01 14:01:42,581 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'wrapt>=1.11.1' 2019-10-01 14:01:42,693 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'grpcio' with range '>=1.8.6' from 'https://pypi.org/simple' 2019-10-01 14:01:42,693 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'grpcio>=1.8.6' 2019-10-01 14:01:43,007 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'wheel' with range '>=0.26' from 'https://pypi.org/simple' 2019-10-01 14:01:43,008 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'wheel>=0.26' 2019-10-01 14:01:43,116 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'backports-weakref' with range '>=1.0rc1' from 'https://pypi.org/simple' 2019-10-01 14:01:43,117 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'backports-weakref>=1.0rc1' 2019-10-01 14:01:43,262 [31432] INFO thoth.solver.python.python:405: Resolving dependency versions for 'enum34' with range '>=1.1.6' from 'https://pypi.org/simple' 2019-10-01 14:01:43,262 [31432] INFO thoth.solver.python.python_solver:113: Parsing dependency 'enum34>=1.1.6'

An the output can be pretty verbose, the following section describes some most interesting parts of the output using JSONPath:

• .metadata - metadata assigned to the solver run - these metadata are especially useful within project Thoth, where analyzer is run in a cluster, the purpose of metadata is to